Out of sample

Out-of-sample testing means holding back part of the history, developing the strategy on the rest, and only then running it on the part you never looked at. It is the closest thing available to an honest test.

The logic is simple. If the rules were tuned on data you had already seen, good performance there proves only that the tuning worked. Performance on data that had no influence on any decision is the first genuine evidence that the strategy found something real.

The rule that makes it work, and the one that destroys it

You look once. The moment you check the out-of-sample result and go back to adjust the strategy, that period has become in-sample and its value as evidence is gone. Every subsequent look is worth less than the last. This is the discipline people find hardest, because the natural response to a disappointing result is to improve the strategy, and improving it against that result is precisely the thing that cannot be done.