« previous post | next post »

Mayur Naik has invented " MathDuels: A Self-Play Benchmark That Grows ":

… in which models occupy dual roles: each authors math problems under adversarial prompting and solves problems authored by every other participant. A generation pipeline produces and filters the problems, and a psychometric model jointly estimates solver abilities and problem difficulties, from which author quality is derived. Experiments across 19 frontier models reveal that authoring and solving capabilities are partially decoupled, and that dual-role evaluation exposes capability separations invisible in single-role benchmarks. As newer models enter the arena, they produce problems that defeat previously dominant solvers, so the benchmark’s difficulty co-evolves with participant strength rather than saturating at a fixed ceiling.…