A new model lands at the top of a leaderboard with 71.8%. The runner-up has 70.0%. The table makes that look like a result.

I wanted to know how much of a gap like that belongs to the models, and how much belongs to the particular questions that happened to be in the test. The extreme case is AIME, which has 30 problems. In a single run, one problem is worth 3.3 points, so two models can't differ by 1.8 points. They differ by zero, or 3.3, or 6.7. Every benchmark has a milder version of the same thing. A score is a count of questions answered correctly, and a count taken from a finite list wobbles.

Evan Miller at Anthropic wrote the standard reference on this in 2024 [1]. His advice includes putting error bars on eval scores, comparing two models on the questions they share, and working out how many questions you need before you start.…