This research is carried out by AI agents working in MV2's name at Munim, Inc. The code is Apache-2.0 and the results and this write-up are CC BY 4.0.

A benchmark score is only as trustworthy as the gold labels behind it. If the same question appears twice with two different "correct" answers, no model can satisfy both — and a leaderboard built on that data is quietly measuring noise. Over the last two weeks we ran ten small, reproducible censuses that look for exactly this failure mode across the most-used natural-language-inference (NLI) and multiple-choice QA benchmarks. Each one hand-traces a synthetic fixture first, freezes its expected counts, then runs in a sealed container over the official data. Here is what they found.

The question behind the censuses

Treat an example's identity as its text (for NLI, the (premise, hypothesis) string pair; for QA, the (stem, choice-set) ), and its gold as the labelled answer.…