A backtest ends with a number. A rule was run over some years of past prices, and it came out ahead. The natural reaction is to be pleased. The more useful reaction is to ask a dull question: ahead of what?
Most people compare against zero. If the rule finished above where it started, it is called good. But zero is the wrong mark, and a few coins show why.
A room full of coin flippers
Picture a hundred people in a room. Each flips a coin ten times and writes down the result. Nobody in the room has any skill at flipping coins. Even so, when the sheets are collected, a handful will almost certainly show eight heads or more. Sort the sheets from best to worst and the one on top looks like talent.
It is not talent. It is what the best of a hundred looks like when nothing is going on.
A search for a trading rule works the same way. You never test just one rule. You try a longer look-back and a shorter one, a tighter threshold and a looser one, this exit and that exit.…