When you look at data over time, it's fairly common to see weird blips, even in cumulative mean effect calculations, that look like this:

/\ | \ ________| \___________________

Or, while waiting for regression to the mean, it's very normal to see your cumulative mean effect, plotted over time, look like this:

\ \ /\ \ / \ /\/\___________ \ / \/ \/

In a recent role advising a company on how to run experiments, I showed them examples of this type of data to demonstrate how stopping a test early could lead to misleading results. We had a great example of data where a bizarre blip on a single day, if the test had concluded that day, would have led to both Frequentist and Bayesian methods calling the experiment with the wrong result with extreme statistical confidence.

We ended up creating a framework for measuring stability over time to make sure that we'd only call a test if the effect has continued over time .

But it was an approach I made up, and that always makes me very nervous.

To me, this feels like…