Menu

Three Green Reruns Prove Almost Nothing: Sizing a Flake-Detection Budget for Agent Patches
πŸ“°

Three Green Reruns Prove Almost Nothing: Sizing a Flake-Detection Budget for Agent Patches

DEV Community: cicdΒ·Finley ZhouΒ·11 days ago
#op8SlhWy
#devtocicd#devto#dev#reruns#gate
Reading 0:00
15s threshold

A merge gate that reruns a failing test three times is not testing for flakes. It is testing whether the flake happens to be loud.

Run the arithmetic before you argue about it. At a 2% per-run failure rate, three reruns catch the flake about 5.9% of the time. The other 94% of the time the gate prints green, the agent patch merges, and the flake ships with a clean receipt stapled to it.

This post is about that receipt. It gives you a small script, a decision table, and a gate policy for sizing how many reruns a patch actually needs β€” and for admitting when reruns are the wrong instrument entirely.

What a rerun is actually measuring

A rerun is a sample. One execution of a test is a Bernoulli trial: pass or fail, with some unknown probability p of failing on any given run. Rerunning k times and reporting green means you observed k passes in a row.

The probability that the gate catches a flake at all is:

detection(k, p) = 1 - (1 - p)^k

Enter fullscreen mode

Exit fullscreen mode

That is the whole model.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.