Ask a team about their worst flaky test and you will get an opinion: "it fails sometimes," "it's the checkout one, it's really bad," "just re-run it, it always passes the second time." None of those statements can support a decision. You cannot tell from "sometimes" whether the test deserves a day of root-cause work, a quarantine slot, or nothing at all — and teams routinely burn days fixing flakes that fail once a month while ignoring the one quietly taxing every pull request.
The fix is not a better opinion. It is a number: how often does this test fail, under real CI conditions? Measure that first, and the decisions mostly make themselves.
Why the measurement has to happen on a real runner
Flakiness is usually a product of the environment — CPU contention, parallelism, network behaviour, timing. A test that never fails on your fast local machine may fail regularly on a loaded CI runner, and a loop run locally measures the wrong machine.…