Menu

Going from top-5 to top-20 retrieval makes every RAG answer 2.7x more expensive
📰

Going from top-5 to top-20 retrieval makes every RAG answer 2.7x more expensive

DEV Community: beginners·Andrey Altrouter·10 days ago
#WQN0wiUs
#devtobeginners#devto#dev#answer#chunks
Reading 0:00
15s threshold

Your support bot answers from your docs. It gets a wrong answer in testing, so you raise top_k from 5 to 20 in the retriever settings, the next answer comes out right, and you ship it. Nobody notices that each answer now costs 2.7 times as much, because the setting lives in the search step, far from the line that calls the model.

Where a RAG answer's money goes

RAG (retrieval-augmented generation) means that before asking the model, your code searches your own documents and pastes the best matches into the prompt. The documents are cut in advance into chunks , pieces of a few hundred words. The search gives each chunk a score , how close it is to the question, and top_k is how many of the best-scoring chunks you paste in.

Model APIs bill per token , a chunk of text about three-quarters of an English word. Input tokens are what you send, output tokens are what the model writes back, and output costs more per token.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.