Death by Backtest
How to tell genuine diligence from procrastination in disguise
I asked a room full of traders last week whether anyone had a system that had been “almost ready” for more than a month.
Hands went up immediately.
One trader admitted he had not traded his own money for more than a year. Not because he had no system. He had several. There was just always one more thing to check first.
Then another trader described the mechanism better than I ever have.
“We get the acceptable ratio, we get the acceptable percentage, and then you sleep on it. And tomorrow morning, what was acceptable last night is not acceptable anymore.”
That is the ratchet. You set your numbers. You hit them. You sleep on it, and by morning what was acceptable has quietly become normal, so you go hunting for better. You find better, sleep on that, and it becomes normal too.
Nothing about the system changed. You did.
Run that loop long enough and you do not end up with a better system. You end up with four more filters, fifteen more optimisations, and something that fails every out-of-sample test you throw at it. Then you either bin it or spend weeks unpicking everything you did.
That is death by backtest, and it is one of the quietest killers of trading results I know. It does not feel like a mistake while it is happening. It feels like being careful.
The rabbit holes are all the same shape
Here is what the room actually named when I asked where they get stuck testing...
Minimum and maximum entry prices
Volatility levels
Parameter ranges
Regime filters
Allocation percentages
Position score models
Position sizes
Volume filters
Candlestick patterns
Scaling In/Out
Ideas from podcasts
Optimization methods
and more…
And the classics that dress themselves up as high standards:
“I want the return above 20% before I launch.” “I only trade systems with a MAR ratio over 1.5. This one is 1.4, so clearly there is more work to do.” “I will start when the market is calmer.” “I will go live once I understand every line of the code.”
Look at that list again and notice what every single item has in common.
They are all about you. Your preference, your comfort, your threshold. Not one of them is about what the market will actually give you, or what the system is capable of generating. You are trying to squeeze a number out of the market by wishing for it in a backtest, and the market has no idea you set a hurdle.
That is a dangerous place to work from.
What is actually underneath it
If the tinkering were really about the system, it would stop when the system was good. It does not. So it is worth asking what it is really about.
The fears that came up in the room, and the ones I recognise in myself:
Losing money. The obvious one. A new system is an unknown, and unknowns can cost you.
Being wrong where people can see it. If you tell someone close to you that you are building a trading system, and a month later they ask how it is going, you have to say “it is in drawdown” or “I over-fitted it and lost money”. Nobody wants that conversation.
Needing certainty. Most of us have some tolerance for uncertainty. It shrinks fast once money is involved.
Perfectionism, and where it comes from. This work attracts analytical people with an engineer’s brain. We did well at school. We were rewarded, for twenty years, for getting things right. Then we arrive in trading, where you will be wrong a great deal of the time and down on plenty of days. No wonder it feels uncomfortable.
The comfort of research. This is the big one, and the most honest. While an idea is still in testing, it is still a good idea. It has not failed yet. You have not been wrong yet. Nobody has to be told the system is in drawdown. Every fear on this list sits on the other side of going live, which makes the research phase a genuinely lovely place to stay.
And you are usually doing it alone, which means the whole psychological game is yours to manage with no one to check it against.
At some point the testing stops being diligence and starts being shelter. The trick is noticing when
The line between diligence and procrastination
Have you ever been certain you were being thorough, then realised months later that you were avoiding something? I have. Most systematic traders have. The difficulty is that from the inside, the two look identical. Both involve work. Both involve testing. Both feel responsible.
Here is how I tell them apart.
Genuine diligence looks like this:
You are following a documented, step-by-step process.
Each test answers a specific question you had written down before you ran it.
Results get validated out of sample.
The rules you are adding are broad, like a sledgehammer, rather than fine adjustments, like a scalpel.
You are working to “good enough” criteria you set before you started.
Procrastination looks like this:
Lots of tweaks, none of them part of a sequence.
“I will just try this.” “I wonder what happens if.”
Patching small problems by adding more rules.
Re-running the same data with no decision framework.
Moving the goalposts when the deadline arrives.
If those lists are still ambiguous in the moment, one question settles it:
Is this step getting me closer to a decision, or is it giving me more questions?
There is a time for questions. Right after the first backtest you should be asking plenty of them. What is wrong with this strategy? What do the good trades look like? What do the bad ones look like? Under what conditions does it fail? That is exactly the right moment for curiosity.
But after that, every step should narrow. If your work keeps generating new questions instead of closing them, you are circling the decision rather than closing in on it.
Why batching makes the work finite
The structural reason iteration never ends is that every test hands you a new idea.
You test something, it half works, and that gives you another idea. You code that, test it, and it suggests a third. Then you hear something on a podcast and try that too. There is no natural end point. You stop when you get tired, not when you are done.
So separate the steps and do them in batches.
Step one: generate hypotheses only. Look at the trade list, the equity curve and the results, and brainstorm every idea you have for improving the system. No coding. No testing. Just the list.
This is the part that matters. With no podcasts, no videos and no new inputs arriving mid-process, hypothesis generation is a finite activity. You will run out of ideas. And when you run out, you know the step is done. Iterating never gives you that signal.
Step two: code every hypothesis. Still no testing.
Step three: test them one at a time.
Step four: compare each result against the baseline that contains none of them, and keep only what genuinely helps.
Batching also produces better rules, and this is the part traders underestimate. Ideas generated up front, before you have seen any results, are broad by nature. Sledgehammers. Ideas generated off the back of the last test are fine adjustments to whatever that test just showed you. Scalpels. And scalpels are what fail out of sample.
One trader in the session put it plainly: when he predefines his hypotheses, his out-of-sample results are better. When he tests, reacts, tests, reacts, out of sample fails. That matches my experience exactly.
The “good enough to start” checklist
Think about how a pilot handles this problem.
If a pilot worried about every possible thing that could be wrong with the aircraft, checking and rechecking in a random order as each worry occurred to them, no flight would ever leave on time. That is not what they do. They sit down with a checklist and work through it item by item. At the end of the checklist, they take off.
Here is the equivalent for a trading system. Six items. When all six tick, it is good enough to start.
You understand every rule and what it actually does.
The backtest was done properly. One pass through a documented process, every step completed, nothing skipped and nothing repeated because you felt uneasy.
You understand the system’s catastrophic risk potential. What is the worst trade this can produce? What does it do if the market crashes next month? If you cannot answer that, fear of losing money will stop you going live no matter how good the numbers look.
It survived out-of-sample and robustness testing.
It earned a place in your portfolio through the capital allocation process.
Your paper trading process runs smoothly - the scans, the watchlists, the execution.
Now the honest caveat, because this matters more than the checklist itself: your system probably does not tick all six yet. Mine often do not. I have several in the pipeline right now that have not come out the other end.
That is completely fine. It means there is real work still to do, and knowing exactly which box is missing tells you precisely what that work is.
But if all six boxes tick and you are still running tests, the extra testing is stalling dressed up as diligence. The checklist is what separates the two.
The two reframes that change everything
Most of the endless polishing traces back to two definitions being wrong.
Redefine “done”
Done is not “the system hits my numbers”. Setting out to build something that returns 22% with a drawdown under 15% and a ratio above whatever is not a definition of done. It is a wish list with a spreadsheet attached, and it makes finishing structurally impossible, because there is always one more parameter between you and the number you wished for.
Done is: I followed every step of the process, and it passed out-of-sample testing. That is the whole definition. However ugly it looks when it comes out the other end, that is as good as it is going to get for now.
Redefine “good enough”
This one is bigger, and it took me years to see it.
Good enough is not “it meets my performance benchmarks”. Good enough is: does it earn a place in my portfolio when I run capital allocation?
Consider a system that makes 2% a year with a 30% drawdown. On its own, that is rubbish. Bin it.
Unless it is strongly negatively correlated to everything else you trade. Then it might be one of the most valuable things in your portfolio.
This is precisely why performance hurdles are dangerous. “I only trade systems with a MAR above 1.5, and this is 1.4, so back to work.” That hurdle just deleted a diversifier before it ever got tested where it counted.
Notice what this means about the backtesting process itself: there is no step in it that says “the system must clear a return hurdle”. That absence is deliberate. If the hurdle were there, you would throw away systems that would have earned capital and diversified your portfolio.
A bear market system judged alone looks unappealing. Volatile, high drawdown, unspectacular returns. Judged inside a long-only trend-following portfolio, it earns its place comfortably. I have run great-looking systems through capital allocation and watched them earn nothing at all. I have run distinctly average-looking systems through and watched them earn a real allocation, because of the market, the direction or the strategy type they covered that nothing else did.
There is even a timing dimension to it. A system that does not earn a place alongside five others at a $50,000 account might well earn one alongside twenty systems at half a million, where spreading capital more widely reduces position sizes and slippage.
When I had this realisation, my first thought was uncomfortable: how many good systems have I thrown away because the equity curve looked ugly? How much diversification did I lose over those years by judging systems one at a time? I will never know what I left on the table.
If you have never built a portfolio of systems and allocated capital across them, start here - it is the piece that makes the rest of this make sense.
The cost of waiting is real, and it compounds
We measure the cost of a bad trade obsessively. We almost never measure the cost of not starting.
It compounds on four levels:
Financial. Returns you never get back, and the compounding on those returns you also never get back.
Psychological. Confidence comes from trading a system and watching it behave. It does not come from testing one. Waiting builds nothing.
Skill. Slippage, fills, how the process fits your actual week - these only reveal themselves with real money live. Every month you wait is a month of learning you do not bank.
Momentum. Paralysis disguised as preparation is the hardest state to get out of, and it gets harder the longer it runs.
Here is a framing from the session that stuck with me. Imagine you traded for a boss or a client. Someone whose money is sitting in cash while you research.
Would they set you an objective of “spend at least six months backtesting each strategy and test at least 100 filter ideas from 30 different podcasts, to make sure you have been completely exhaustive”?
Of course not. No manager on earth writes that objective.
What they would actually ask at the end of the year is this: how many diversified, uncorrelated strategies did you add to the portfolio, and how did you improve the performance of the money we are actually trading? They would not care how many filters you tested. They would care what you deployed. And they would be distinctly unimpressed by six months spent on a project that turned out to be a dead end. Two hours spent finding that out? Perfectly fine.
One trader in the session builds this in deliberately. Before starting on a system he books the time like a work project. Two hours. Whatever happens in those two hours is what that system gets. It goes in the calendar: 11am this system, 2pm that system, no matter what. The pressure of a deliverable is the one thing independent traders remove from their process, and then wonder why nothing ever ships.
You do not have a boss. So you have to be one.
The pressure release nobody gives themselves
Why are you trying to build a perfect trading system?
It is worth answering honestly, because the answer usually exposes the problem. If you have been agonising over one system for months, you are treating it as though your results depend on it. As though this is the one.
But you would never allocate 100% of your capital to a single system. Systems break. Edges degrade. Markets shift underneath them. If the plan is 15 or 20 strategies across different markets, directions, timeframes and profit drivers, then any single system is going to carry a small slice of your capital.
That changes everything about how hard you should be polishing it. A system that will hold 5% of your portfolio does not need to be perfect. It needs to be sound, tested, and different enough from the others to earn its slice.
Nearly all of the agony during backtesting comes from looking at a system in isolation, as if that one system has to carry you. The portfolio is what carries you. The more systems you build, the less any single one matters, and the less reason you have to over-fit it in the first place.
And there is a bonus in the reframe: it is far harder to curve fit your way into genuine diversification than into a pretty ratio.
What to actually do this week
Think of yourself as running a machine that processes ideas. An idea goes in. It passes the tests or it fails them. If it passes, it goes to capital allocation and either earns a place or it does not. Then the next idea goes in.
The output of that machine is a portfolio, assembled from the small share of ideas that survived the whole journey. Which means the throughput of the machine matters far more than how hard you polish any single idea inside it.
I genuinely enjoy throwing away a system that fails. It is one less thing to spend time on, one less idea taking up space in my head, and one more slot open for the next test. The real skill is recognising a weak idea early and killing it without ceremony.
So:
Name the system you are stalling on. You already know which one it is.
Run it through the checklist. Six boxes.
If a box genuinely does not tick, that box is your next task, and it is a real one.
If they all tick, pick a deploy date, size it small, and go live.
Perfect is a hiding place. Good enough and traded wins.
You will make far more money with a good system you actually implement than with a perfect system you never do.
If you want the complete process behind this - the documented backtesting method, the capital allocation work that decides whether a system earns its place, and the live mentoring to keep you moving instead of polishing - that is what the Trader Success System is built to do.
Remember - you are only one trading system away!
Adrian Reid
Founder - Enlightened Stock Trading
The most dangerous overfitting often feels like responsibility. Every extra test produces another question, the acceptance criteria quietly move, and deployment remains safely out of reach. A predefined process, fixed stopping conditions, and out-of-sample validation turn research into a decision—rather than a permanent shelter from uncertainty.