Menu

You can't rank 200 models with one leaderboard
πŸ“°

You can't rank 200 models with one leaderboard

DEV Community: postgresΒ·SoccerDream AIΒ·8 days ago
#F1erNLFt
#devtopostgresql#devto#dev#model#models
Reading 0:00
15s threshold

I run a platform where a couple of hundred independent models publish probability estimates on football fixtures. They aren't variants of one model with different seeds β€” each one has its own training corpus, its own feature set, its own retraining schedule, and its own life cycle. At the time of writing: 209 have existed, 27 are active, 102 are retired, and 25,963 predictions have been graded.

The first version of the admin panel had a leaderboard. Sort by accuracy, descending, take the top one. It took about three weeks to learn that a leaderboard over a population that large is not a ranking β€” it's a multiple-comparisons machine with a UI.

Here's what replaced it, and the four bugs that forced each piece.

The problem with sorting by accuracy

Two hundred models, dozens of fixtures a day. Somebody will hit a hot streak. It is not a hypothesis that one of them will look excellent over a 40-fixture window; it is arithmetic.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.