In this lesson
- understand why to test on data you did not tune the strategy on
- see how the "winner" of the last test did on new data
- learn the walk-forward method
In the last lesson the 40-day average strategy looked great. But we chose that length precisely from that data – so the test really only showed that we can find the best number in the past. The real question is: how would it have done on data it has never seen?
Two piles of data#
The data is split in advance, before the very first test:
- in-sample – where you invent and tune the strategy (for us 2019–2021),
- out-of-sample – you must not touch it until the rules are finished (for us January 2022 to September 2025).
The out-of-sample test is done once. Anyone who looks at the result, adjusts the rules and tests again has turned "out-of-sample" into more tuning data.
How the winner did#
| 2022 – Sep 2025 | 40-day average (winner of 2019–2021) | 200-day average | Buy and hold |
|---|---|---|---|
| Total return | +166% | +234% | +147% |
| Maximum drawdown | −47% | −32% | −67% |
| Number of trades | 47 | 13 | 1 |
| Profit factor | 2.4 | 9.8 | – |
- 40-day average
- 200-day average
- Buy and hold
What happened:
- The "best" 40-day average slightly beat simply holding on the new data and had smaller drawdowns – but among the 30 tested lengths it came only 20th. Its brilliance in 2019–2021 was largely chance.
- The 200-day average did better. Does that prove it is better? No. It has only 13 trades, and we have just chosen it – based on the out-of-sample data. If we started using it, we would overfit again, just on different years.
Walk-forward#
The walk-forward method repeats the split step by step, the way a real trader would:
- Pick the best average length on the last three years.
- Trade with it for the following year.
- Move forward a year and repeat.
| Year | Chosen length | Strategy | Buy and hold |
|---|---|---|---|
| 2022 | 40 | −44% | −64% |
| 2023 | 40 | +136% | +156% |
| 2024 | 110 | +93% | +121% |
| 2025 (to Sep) | 170 | −1% | +22% |
In total +153% against +147% for simply holding. The strategy lost less in the 2022 crash and made less in the rallies. No miracle – and that is valuable information.
An honest result tends to be boring
When a strategy passes an out-of-sample and walk-forward test and comes out "roughly like simply holding, with smaller drawdowns", that is a realistic result. Miraculous numbers usually mean a mistake in the test.
Common myths#
- "It did not work out of sample, so I will tune it." That turns it into tuning data.
- "The 200-day average did best, I will use it." You chose it by its result – the same trap again.
- "Walk-forward only matched holding, the strategy is useless." Smaller drawdowns can be worth something.
Summary
- Split the data in advance: for tuning (in-sample) and for a single final test (out-of-sample).
- Our winner of 2019–2021 ended up in the middle of the field on new data.
- Walk-forward simulates real trading: it picks parameters only from the past.
- A realistic result tends to be modest; miracles usually mean a mistake.