You tested 8 strategies and one looks brilliant. PBO asks the question most backtests never face: among everything you tried, how likely is the winner just the luckiest loser?
You ran 8 versions of a strategy. Seven were mediocre. One was brilliant. You ship the brilliant one. Question: did you find an edge, or did you run a lottery and crown the winning ticket?
That is exactly what PBO — the Probability of Backtest Overfitting — measures. And it is one of the least explained ideas in trading, because the people who understand it write papers, and the people who write blog posts mostly skip it. Here it is in plain English, with a worked example.
The core idea
PBO does not ask “is this strategy good?” It asks something sneakier: if you pick the best performer from a set of candidates, how likely is it to disappoint out of sample?…