The new Open ASR Leaderboard update adds private conversational English splits from Egyptian and Nigerian speakers, along with broader accent breakdowns. [1] That makes this a good moment to ask what an aggregate score is weighting. My numerical example is synthetic; it is not a measurement of those datasets or any named speech model.

Imagine choosing a speech recognizer from a row labeled 7.5% word error rate . Two systems share that number, so the shortlist looks settled.

In the invented scorecard below, one system makes many more errors on the smaller group. The overall tie hides that difference completely.

Computed illustration, not leaderboard results. A and B are fictional systems; group sizes are assumed reference-word counts. Image by the author.

Follow the denominator

Word error rate, or WER, divides substitutions, deletions and insertions by the number of words in the reference transcript. I use “errors” below for that summed edit count.…