Menu

Count the Empty Cells Before You Quote a Pass Rate
πŸ“°

Count the Empty Cells Before You Quote a Pass Rate

DEV Community: pythonΒ·Jordan LiuΒ·9 days ago
#Q1XvZgX4
#devtopython#devto#dev#empty#grid
Reading 0:00
15s threshold

Count the Empty Cells Before You Quote a Pass Rate

A pass rate is an average over a grid you never drew. That's the whole problem. Ninety-four percent sounds like a verdict, but it's really a summary of whatever happened to be sitting in your case file that afternoon, and the file has holes in it you've never looked at.

I stopped trusting my own numbers the day I printed the grid instead of the total. Four of fifteen cells were empty. Not "thin" β€” empty. Nobody had ever sent the model an empty input with a tool-call contract, because writing that case is boring and it felt redundant. It wasn't redundant. It was the one place the schema silently returned a fabricated argument.

So here's the workflow I now run before any eval number leaves my laptop: draw the taxonomy, audit the grid, gate the commit. More cases is not the fix. Coverage first, volume second.

Draw the grid before you write the cases

The taxonomy is a small YAML file and it should live in git next to the cases it describes.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.