Moritz Hardt, author of a new book on benchmarks, believes these “standardized tests” for artificial intelligence are fundamentally broken — but not unfixable.

The computer scientist Moritz Hardt calls benchmarks “the mistake that made machine learning.” He’s studied where they went wrong, and why they’ve managed to drive progress anyway.

Jerrit Peinelt for Quanta Magazine

Introduction

Benchmarks are the rulers of AI, in both senses of the word. As measuring sticks, they work according to an extremely simple logic: If you want to know what an AI model can do, devise a standardized test to gauge it. But AI benchmarks can also dominate the behavior of the entire field.

In 1987, the Defense Advanced Research Projects Agency’s first benchmark on automatic speech-to-text translation showed that state-of-the-art AI could not outperform so-called “dumb” statistical approaches. The poor performance helped to usher in an AI winter.…