You rewrote the support prompt. Two days later the dashboard says the new version resolves four points more tickets, and the 95% interval doesn't touch zero. You ship it. A month later the resolution rate is exactly where it was.

The arithmetic was fine. What went wrong was when it was read.

This is a condensed version of a longer guide, with an interactive version of the demo below, sample sizes and a FAQ: A/B testing LLM prompts without false winners .

A 95% interval makes one promise

When there's no real difference, a 95% interval excludes zero only 5% of the time. That promise holds for one look, at a sample size you fixed in advance . A live dashboard recomputes the interval after every batch of results, and every look is a fresh chance for noise to cross the line.

How much does that cost? We simulated 4,000 A/A tests, where both versions are the same prompt at 30% success, and looked after every 100 outcomes per version.…