A team running an AI patch pipeline once showed me their nightly eval: 214 golden cases, a clean green dashboard, and a production regression that the suite had walked straight past. The postmortem took an hour. Then someone asked the obvious question, and nobody had an answer: how many of those 214 cases had ever failed for a reason worth knowing?

We instrumented the suite instead of arguing about it. Sixty-one cases had never failed on any patch the pipeline had ever scored. Thirty-three failed on nearly every patch, which made them noise rather than signal. The suite looked thorough and was mostly ceremony.

This article describes the instrument we built: a kill matrix. It is a small, reproducible way to find out which golden cases actually earn their runtime, and which ones you are paying for out of habit.…