Anyone who tests a trading rule on past prices learns the first lesson quickly. A rule tuned on a stretch of history looks good on that stretch, because it was shaped to fit it. So you keep some history aside, tune on the rest, and check the rule on the part it never saw. That part is usually called a hold-out, or out-of-sample data.
It is a sound idea. It also wears out faster than most people expect, and it wears out through a perfectly reasonable habit: looking at the result and then doing something about it.
How a hold-out gets used up
Take a made-up example. You search one stretch of history and come away with twenty candidate rules. You run all twenty on the hold-out. Most do poorly. Three do well. You keep the three and write down their hold-out scores as your evidence.
Those scores are no longer clean. The hold-out was supposed to judge the rules. Instead, it chose them. You kept those three because they scored well there, so of course they score well there.
Coins make it plain.…