Menu

1 / 2

Leaderboards Are Not Evidence. Your Failure Corpus Is.

DEV Community·kongkong·8 days ago
#Fm86gmNM
#dev#json#corpus#case#model
Reading 0:00
15s threshold

A user opens a document, asks the assistant to rename one section, and gets back a patch that edits a heading that stopped existing two saves ago. The prose was fine; the patch was confident, well-formed, and wrong. Notice where it actually broke: not in the model, but in the seam between a stale snapshot, a schema your validator never enforced, and a retry that replayed the old state. That seam is the layer I keep losing afternoons to, and no leaderboard on earth would have flagged it.

So here is the position I want to argue, and I will argue it plainly: a public model ranking is not evidence about your application, and treating it as evidence is how a team migrates a working feature onto a worse model with a better average. Leaderboards measure a mean over someone else's distribution of tasks, prompts, and tolerances. Your product fails on a specific, embarrassingly small set of inputs — a heading that moved, a tool call that returned 409 , a JSON field that arrived as a string.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.