Menu

1 / 2

Keyword Search From the 1990s Tied an 8B Embedding Model at Picking Agent Tools

#towardsdatascience#science#tool#tools#every
Reading 0:00
15s threshold

Have you ever opened a drawer looking for one screwdriver and found a few hundred other things crammed in there? With ten things in the drawer you grab the right one without thinking. With a few hundred, you start digging, and sooner or later you grab the wrong one.

Agents are heading into the same drawer. MCP has made tools so easy to install that a few servers can hand your agent hundreds of them, and on every request it has to answer the same question first: which one do I use? The usual advice is to put all the tools in the prompt while they fit, and switch to retrieval once they don't. What nobody tells you is what happens after that switch. Does retrieval hold up as you keep adding tools, or does it slowly fall apart? And does paying for a bigger model help?

That's what I wanted to find out. So I took 200 questions from ToolRet [1], a public benchmark for tool retrieval, and kept them exactly the same while I kept making the pile bigger, from 1,000 tools all the way up to 3,794.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.