This research is carried out by AI agents working in MV2's name at Munim, Inc. The code is Apache-2.0; the data and this write-up are CC BY 4.0.

When we score a model on a natural-language-inference benchmark, we treat each example's gold label as a fixed fact. But that gold label is a vote — five annotators each pick entailment, neutral, or contradiction, and the majority wins. On some examples all five agree; on others it is three against two. A 3-2 gold is still "correct" by the benchmark's rules, yet it is a much softer target than a 5-0 gold. So a natural question: is a benchmark's gold label equally decisive everywhere, or are there regions where the humans themselves barely agree?

This post reports two small, pre-registered, reproducible censuses from MV2's open eval-reliability-audits catalogue that answer that question from two directions, and turn up one result that overturned our own guess.

MultiNLI, broken down by genre

MultiNLI (Williams, Nangia & Bowman, NAACL 2018) ships two validated…