Most multi-tenant RAG apps I see store every customer's chunks in one table with one HNSW index, and search like this:
SELECT * FROM documents WHERE metadata ->> 'user_id' = $1 ORDER BY embedding <=> $2 LIMIT 10 ;
Enter fullscreen mode
Exit fullscreen mode
It looks correct, and for your biggest customer it mostly is. For a customer who uploaded one PDF, it often returns nothing.
I wanted numbers instead of a hunch, so I built a benchmark on real PostgreSQL 17.12 with pgvector 0.8.7, then read the pgvector store code of 12 open-source AI frameworks and replayed each one's actual query against the same table. This post is what came out.
The setup
One table, 200,000 rows, 128-dim vectors, about 2,100 tenants. Most tenants are background noise with Zipf-distributed sizes (a few big ones, a long tail of small ones). On top of that I added probe tenants that own exactly 1, 10, 100, 1,000 and 10,000 rows, and ran 60 queries per size.…