Menu

#Top_k

2 posts

Feed·
2 of 2 posts
Going from top-5 to top-20 retrieval makes every RAG answer 2.7x more expensive
📰
0Read More

Going from top-5 to top-20 retrieval makes every RAG answer 2.7x more expensive

DEV Community: beginners·Andrey Altrouter·10 days ago
#WQN0wiUs
#devtobeginners#devto#dev#answer#chunks

In a RAG bot, the question is a fraction of a percent of the bill. Most of it is the chunks you paste in to answer it. At 500 tokens a chunk, top_k = 20 instead of 5 makes every answer 2.7 to 2.8 times more expensive on any of five popular models.…

15s