[Discussion / Benchmarks] TurboVec vs. pgvector HNSW Evaluation & FAISS IndexPQ Cross-Comparison

[divyanshk007]

Context & Objective

We have been evaluating TurboVec for an internal large-scale Retrieval-Augmented Generation (RAG) project. As part of our architectural validation, we ran a comprehensive baseline benchmark comparing TurboVec against PostgreSQL's pgvector HNSW strategy over a multi-million document corpus.
Our results revealed extreme variations in memory footprint, metadata filtering efficiency, and bulk ingestion performance. We wanted to share these high-level insights with the community, ask if you have run similar verification profiles against pgvector, and gather thoughts on how these metrics align with your internal benchmarks against FAISS IndexPQ.

Benchmark Setup

• Dataset: BEIR msmarco dev split containing 8,843,823 passages and 68 queries.
• Embedding Model: /noi embed-text-v1.5 (pinned revision 966763023-670ca8431644204150261009).
• Dimensions Evaluated: 256D, 512D, and 768D.
• Environment: Target Ubuntu 24.04 x86_64, running Python 3.14.3 inside an isolated container space to prevent backend cross-interference.
• Core Packages: turbovec==0.7.1, psycopg==3.3.2, numpy==2.4.1.
• Index Tuning: pgvector HNSW configured with ef_construction up to 7680 and search sensitivity ef_search=100.

Key Benchmark Data Highlights (512D Primary Focus)

1. Storage & Memory Footprint
   TurboVec yields significant efficiency improvements over vanilla pgvector f32. At 512 dimensions, the memory savings are on the order of ~7x to ~11x:

Experiment	Disk Footprint (MB)	Working-Set RAM (MB)	Bytes / Passage
pgvector f32 d512 ef100	21,980.0	22,292.0	2,606.7
TurboVec 2-bit d512	1,696.0	1,983.0	198.2
TurboVec 4-bit d512	2,739.0	3,084.0	328.7

2. Filtered Search Results (Metadata Pre-filtering)
   When restricting the search domain using an allowed-ID list at varying levels of corpus selectivity ($K=10, C=1$), pgvector experiences a severe latency breakdown, whereas TurboVec scales predictably:

Backend Strategy	Selectivity %	Allowed IDs Count	p50 Latency (ms)	p95 Latency (ms)	QPS (C=16)
pgvector f32 d512	1.0%	88,418	350.00 ms	437.50 ms	6.0
TurboVec 4-bit d512	1.0%	88,418	3.41 ms	4.09 ms	321.7
pgvector f32 d512	20.0%	1,768,364	4,400.00 ms	5,500.00 ms	0.5
TurboVec 4-bit d512	20.0%	1,768,364	23.18 ms	27.82 ms	47.9

3. Ingestion & Bulk Update Anomalies

Testing a 10% append scenario (adding 884,182 vectors to a 7,957,641 base index) surfaced an inverse performance bottleneck:

• pgvector (COPY INSERT): Required 8,500.0 seconds to incrementally balance and re-index the HNSW graph.
• TurboVec (Full Rebuild): Rebuilding the binary index totally from scratch took only 1,095.0 seconds.

Questions & Discussion Points for Maintainers

1. TurboVec vs. pgvector HNSW Baseline: Have you internally run benchmarks directly comparing TurboVec against pgvector using its HNSW index strategy? Do our observed spikes in pgvector filtered latency (e.g., jumping past 4.4 seconds at 20% selectivity ) align with expected engine boundaries when mapped to the in-kernel filter execution pattern optimized in TurboVec?
2. FAISS IndexPQ Comparison Correlation: The TurboVec documentation explicitly benchmarks performance optimizations against FAISS IndexPQFastScan configurations. Given the significant runtime constraints we observed with pgvector during filtered queries and large-batch graph adjustments, have you executed comparative benchmarks directly scaling FAISS IndexPQ vs. TurboVec on high-selectivity filtering or index initialization limits?
3. Graph Rebuilding Insights: Given that a complete clean rebuild of TurboVec outperformed pgvector's native incremental indexing step by nearly 8x for large batches, are there any operational boundaries we should watch out for regarding index degradation over prolonged write updates?We appreciate any feedback or thoughts you have on this data profile!