A merge gate that reruns a failing test three times is not testing for flakes. It is testing whether the flake happens to be loud.
Run the arithmetic before you argue about it. At a 2% per-run failure rate, three reruns catch the flake about 5.9% of the time. The other 94% of the time the gate prints green, the agent patch merges, and the flake ships with a clean receipt stapled to it.
This post is about that receipt. It gives you a small script, a decision table, and a gate policy for sizing how many reruns a patch actually needs β and for admitting when reruns are the wrong instrument entirely.
What a rerun is actually measuring
A rerun is a sample. One execution of a test is a Bernoulli trial: pass or fail, with some unknown probability p of failing on any given run. Rerunning k times and reporting green means you observed k passes in a row.
The probability that the gate catches a flake at all is:
detection(k, p) = 1 - (1 - p)^k
Enter fullscreen mode
Exit fullscreen mode
That is the whole model.β¦