Implement PostgreSQL + FAISS Hybrid Storage for Secure PII Management #38

[massoudsh]

Summary

Implements the PostgreSQL + FAISS hybrid described in #38: PII is stored encrypted in PostgreSQL, FAISS is used only for vector search (no PII). Backward compatible: without DATABASE_URL / USE_PG_STORAGE, behavior is unchanged (pickle only).

Closes #38

What’s included

• Postgres: docker-compose service, schema (pii_records, documents, query_logs), init script
• Encryption: Fernet (app/utils/pii_encryption.py) for passage text; key via ENCRYPTION_KEY
• DB layer: app/db/store.py — save/load passages (encrypted), audit query_logs
• RetrieverAgent: When USE_PG_STORAGE=true, meta load/save uses Postgres; FAISS unchanged; fallback to pickle if PG empty or unavailable
• Audit: Each /query logs to query_logs when PG is enabled
• Migration: scripts/migrate_pickle_to_pg.py to copy existing pickle data into Postgres (encrypted)
• Deps: psycopg2-binary, cryptography, sqlalchemy; .env.example for DATABASE_URL, ENCRYPTION_KEY

Question for maintainer

Is the hybrid (FAISS + Postgres) what you want, or would you prefer a full move to PostgreSQL?

• Hybrid (this PR): Keeps FAISS for vector search; Postgres only for encrypted PII and audit. No change to retrieval performance; minimal change to indexing path.
• Full Postgres: Use pgvector for embeddings in Postgres as well, and drop FAISS/pickle. Single store, simpler ops, but requires pgvector and a decision on indexing/migration.

If you’d rather go full Postgres (e.g. pgvector), we can treat this as a stepping stone and plan a follow-up that migrates vectors into Postgres and removes FAISS. Happy to adjust direction based on your preference.

Made with Cursor