Two of twenty reruns passed before the patch. Three of twenty passed after it. That is not a fix — it is the same coin landing on a different face.

Most agent-patch reviews still treat a green retry as proof. It is not proof of anything until you know how many trials produced it and how wide the uncertainty is. This article builds a small, reproducible gate for that: a trial ledger, Wilson intervals, a decision table, and a retry budget that stops the suite from manufacturing false confidence.

The conclusion first

A flaky test can only license a patch verdict when the two arms of the experiment — baseline and patched — are statistically separable at the sample size you actually ran. In my sizing table below, 20 trials per arm cannot separate a 50% pass rate from 85%, and cannot separate 90% from 95% at all.…