Menu

Train, test, held, reserve: why one hold-out is not enough once you have looked at it
📰

Train, test, held, reserve: why one hold-out is not enough once you have looked at it

DEV Community: datascience·Ultimate Trading System·about 7 hours ago
#FE1ShXYT
#devtodatascience#science#devto#dev#history
Long article (832 words) — no AI summary yet.
Reading 0:00
15s threshold

Anyone who tests a trading rule on past prices learns the first lesson quickly. A rule tuned on a stretch of history looks good on that stretch, because it was shaped to fit it. So you keep some history aside, tune on the rest, and check the rule on the part it never saw. That part is usually called a hold-out, or out-of-sample data.

It is a sound idea. It also wears out faster than most people expect, and it wears out through a perfectly reasonable habit: looking at the result and then doing something about it.

How a hold-out gets used up

Take a made-up example. You search one stretch of history and come away with twenty candidate rules. You run all twenty on the hold-out. Most do poorly. Three do well. You keep the three and write down their hold-out scores as your evidence.

Those scores are no longer clean. The hold-out was supposed to judge the rules. Instead, it chose them. You kept those three because they scored well there, so of course they score well there.

Coins make it plain.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.