I run a platform where a couple of hundred independent models publish probability estimates on football fixtures. They aren't variants of one model with different seeds β each one has its own training corpus, its own feature set, its own retraining schedule, and its own life cycle. At the time of writing: 209 have existed, 27 are active, 102 are retired, and 25,963 predictions have been graded.
The first version of the admin panel had a leaderboard. Sort by accuracy, descending, take the top one. It took about three weeks to learn that a leaderboard over a population that large is not a ranking β it's a multiple-comparisons machine with a UI.
Here's what replaced it, and the four bugs that forced each piece.
The problem with sorting by accuracy
Two hundred models, dozens of fixtures a day. Somebody will hit a hot streak. It is not a hypothesis that one of them will look excellent over a 40-fixture window; it is arithmetic.β¦