A user opens a document, asks the assistant to rename one section, and gets back a patch that edits a heading that stopped existing two saves ago. The prose was fine; the patch was confident, well-formed, and wrong. Notice where it actually broke: not in the model, but in the seam between a stale snapshot, a schema your validator never enforced, and a retry that replayed the old state. That seam is the layer I keep losing afternoons to, and no leaderboard on earth would have flagged it.
So here is the position I want to argue, and I will argue it plainly: a public model ranking is not evidence about your application, and treating it as evidence is how a team migrates a working feature onto a worse model with a better average. Leaderboards measure a mean over someone else's distribution of tasks, prompts, and tolerances. Your product fails on a specific, embarrassingly small set of inputs — a heading that moved, a tool call that returned 409 , a JSON field that arrived as a string.…