Count the Empty Cells Before You Quote a Pass Rate
A pass rate is an average over a grid you never drew. That's the whole problem. Ninety-four percent sounds like a verdict, but it's really a summary of whatever happened to be sitting in your case file that afternoon, and the file has holes in it you've never looked at.
I stopped trusting my own numbers the day I printed the grid instead of the total. Four of fifteen cells were empty. Not "thin" β empty. Nobody had ever sent the model an empty input with a tool-call contract, because writing that case is boring and it felt redundant. It wasn't redundant. It was the one place the schema silently returned a fabricated argument.
So here's the workflow I now run before any eval number leaves my laptop: draw the taxonomy, audit the grid, gate the commit. More cases is not the fix. Coverage first, volume second.
Draw the grid before you write the cases
The taxonomy is a small YAML file and it should live in git next to the cases it describes.β¦