Menu

Your RAG app forgets small customers: I measured filtered pgvector search in 12 framework stores
📰

Your RAG app forgets small customers: I measured filtered pgvector search in 12 framework stores

DEV Community: postgres·Sunnatulla·about 13 hours ago
#CfBaWLQt
#devtopostgresql#devto#dev#index#tenant
Reading 0:00
15s threshold

Most multi-tenant RAG apps I see store every customer's chunks in one table with one HNSW index, and search like this:

SELECT * FROM documents WHERE metadata ->> 'user_id' = $1 ORDER BY embedding <=> $2 LIMIT 10 ;

Enter fullscreen mode

Exit fullscreen mode

It looks correct, and for your biggest customer it mostly is. For a customer who uploaded one PDF, it often returns nothing.

I wanted numbers instead of a hunch, so I built a benchmark on real PostgreSQL 17.12 with pgvector 0.8.7, then read the pgvector store code of 12 open-source AI frameworks and replayed each one's actual query against the same table. This post is what came out.

The setup

One table, 200,000 rows, 128-dim vectors, about 2,100 tenants. Most tenants are background noise with Zipf-distributed sizes (a few big ones, a long tail of small ones). On top of that I added probe tenants that own exactly 1, 10, 100, 1,000 and 10,000 rows, and ran 60 queries per size.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.