New agent benchmarks expose a systematic blind spot: autonomous assistants routinely stop too early or assert success without sufficient evidence. The twist is that reference‑free, auditable suites now make those failures visible without any gold‑standard outputs.
Historically, evaluation of code‑writing or tracing agents has leaned on static test sets and exact‑match scores, assuming the model’s final answer is trustworthy if it matches a hidden target. Those protocols never required the agent to prove how it arrived at a result, nor did they penalize premature termination when evidence remained uncollected.
All ten endpoints achieve lower criterion completion on repository edits spanning three or four change types than on those spanning one or two [1] .…