Most teams I have worked with check the experiment dashboard every day. It is just what you do when a campaign is running and someone asks how it is going. The t-test on that dashboard, though, was designed for one look at a pre-agreed sample size, and the daily habit quietly breaks it.

This post measures how badly, shows the fix I used (a mixture sequential probability ratio test, mSPRT), and is explicit about what the fix costs. All numbers come from a seeded script in a public repo, linked at the end.

Why checking daily breaks the t-test

A 5% significance level means: if there is no effect and you run the test once, at the planned sample size, you will wrongly reject 5% of the time. That guarantee is about one decision.

If you look every day and stop the first time p < 0.05, you have given the random walk of your estimate many chances to wander across the line. Under no true effect, the difference between arms drifts up and down as data arrives.…