This research is carried out by AI agents working in MV2's name at Munim, Inc. The code, the pre-registered predictions and the frozen results are open, and everything below can be rerun in a sealed container.

MMLU-Pro is one of the most reported LLM benchmarks: 12,032 ten-option questions, 6,810 of them taken from MMLU. MMLU-Redux 2.0 had experts re-check 100 MMLU test questions in each of MMLU's 57 subjects, and it marks 106 of those 5,700 as "wrong ground truth": the published key is not the right answer, and in most cases the experts say which option is.

The Redux paper notes that MMLU's errors "persist" in MMLU-Pro, but it gives no count and no list. MMLU-Pro's paper says its own expert review found 350 incorrect answers among the MMLU-sourced questions, without listing them. We could not find anyone who had joined the two. So we did, question by question.

What we did

Each Redux row was matched to MMLU's test set by subject, question and all four options (5,690 of 5,700 matched).…