Menu

1 / 4

A Local LLM Got 0 of 15 New-Paper Questions Right, With the Passages in Front of It

Reading 0:00
15s threshold

I read a lot of technical material: machine learning textbooks, particle physics, and papers in my own field. I wanted an assistant that answers questions about my library, cites the exact page, shows me the passage, and runs on my own laptop. No uploads, no subscription, no guessing where an answer came from.

So I built one!

To be honest about how: I didn't type most of the code myself. Claude Code built it on my machine, phase by phase, while I designed it, wrote the test questions, read every report and decided at every gate. That way of working is a story on its own. This post is about what the measurements showed, because some of it surprised me.

The short version: on textbook questions, small local models do well. On a paper published this year, they collapse to almost zero, with the same retrieval, the same prompt and the same passages in front of them.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.