Automated tradingUpdated 2 min readBy the FXNOD team
Overfitting in trading: why perfect backtests fail
Overfitting is adjusting a strategy so closely to past data that it captures the accidents of that period instead of anything lasting. An overfitted strategy looks excellent in a backtest and fails on new data. The defences are few settings, testing on data not used for tuning, and distrust of any result that looks too good.
How it happens
- Trying many settings and keeping the best one.
- Adding a rule to remove each losing trade in the test.
- Testing many ideas on the same data until one works.
- Choosing the market, timeframe and period after seeing the results.
Signs of an overfitted strategy
| Sign | Why it is suspicious |
|---|---|
| Many parameters | Each one is another way to fit noise |
| Very specific values, such as RSI 13.7 | Nothing real is that precise |
| Results collapse if a setting moves slightly | A real effect is robust to small changes |
| A profit factor far above 2 on few trades | Too good for the sample size |
| Works on one market and one period only | It has learned that period |
| A rule that exists to skip one bad week | It was added after the fact |
The tests that expose it
Out-of-sample test
Tune on one part of the history. Run once, unchanged, on another.
Neighbour test
Nudge each setting. Results should degrade gently, not fall off a cliff.
Other markets and periods
A sound idea usually works, less well, elsewhere.
Forward test
Live prices with virtual money. It cannot be fitted to the past.
Why testing many ideas is itself a trap
Test twenty random strategies and one will look good by chance at ordinary confidence levels. The more you search, the more convincing your best result becomes and the less it means. Count how many variations you tried, and demand more from the winner accordingly.
On random markets
A random series is all noise, so any rule that looks profitable on its past is overfitted by definition. Optimising a bot's entry settings on a synthetic index will always find a best setting. It will not carry forward.
Habits that prevent it
- Start from an idea with a reason behind it, then test. Not the reverse.
- Use as few settings as possible, at round, conventional values.
- Decide the test before you run it, and run it once.
- Keep a part of the data that you never look at until the end.
Questions people ask
- What is overfitting in simple terms?
- Making a strategy fit the past so exactly that it has memorised history instead of learning anything that repeats.
- How do I know if my strategy is overfitted?
- Run it unchanged on data it has never seen. If the results collapse, it was.
- Is optimisation always bad?
- No. Light tuning checked on unseen data is reasonable. Searching for the perfect setting is not.
Run a bot that keeps its limits
FXNOD's bots run on its servers and will not start without a stop loss, so your limits hold when your device is off. Build one in Bot Forge or start a ready-made one in Auto Hub, on your Deriv demo account first.
Get started