Backtesting is not proof: what I actually trust before going live

Let me say the uncomfortable thing first: a backtest is not evidence. It's a description of the past written by someone who already knew how it ended. That's it. And I say that as someone who has spent a genuinely embarrassing number of hours running them on Confluence, because they're still the cheapest way to kill a bad idea. Just not the way to bless a good one.

The reason is simple and it's kind of brutal. You built the rules while looking at the data. Even if you swear you didn't tune anything — you did. You picked that market because you'd noticed something in it. You picked that timeframe because the other one looked ugly. You started the test in 2019 because that's where your data began, which happens to skip a regime that would've eaten the strategy alive. None of that feels like cheating in the moment. All of it is hindsight leaking in through the walls.

So here's the list of ways a backtest has actively lied to me, because each one taught me something specific.

Curve-fitting, obviously. Enough parameters and enough patience and you can make anything look profitable on history. The tell is fragility: nudge one setting a little and the whole edge falls apart. A real edge is a plateau — it works across a range of nearby settings, because it's describing something true about markets rather than something true about your particular file of numbers. If your result lives on a knife's edge, you didn't find an edge, you found a coincidence and gave it a name.

Look-ahead bias, which is sneakier than it sounds. It's rarely as blatant as using tomorrow's close. It's using a bar that hadn't finished forming yet. It's applying a filter you only knew to apply because you'd already seen what went wrong later. It's an indicator that quietly repaints. This one's nasty because the code looks correct and the equity curve looks beautiful, and beautiful equity curves should make you suspicious, not happy.

Costs. Spread, slippage, commission, and the fact that your fill is worse when you most want it. Strategies that make a small edge per trade and take a lot of trades are exactly the ones that die here, and they're also exactly the ones that look most impressive before costs. So costs go in first now, at pessimistic values, not as a haircut applied at the end to a number I've already emotionally committed to.

And the one that actually got me — the fill mirage. I had a setup that tested beautifully with limit orders. Really nicely. Then I looked at what the backtest was assuming, which was that a limit order fills any time price touches the level. It doesn't. In the real world, price touching your level and immediately reversing usually means you got nothing — the wick came, your order sat there, and the move left without you. The backtest was filling me on trades that never would have happened, at the best possible price, on the exact bars that made the strategy look good. Re-run with honest fill assumptions and the whole thing changed character. I've since made a habit of comparing what a strategy looks like with optimistic fills versus market-order reality, because the gap between those two numbers is a pretty good measure of how much of your edge is fictional.

Survivorship and sample size, briefly, because they're well-covered elsewhere: don't test on the assets that made it, and don't trust an edge built on forty trades. A run of luck is thirty trades long. Ask me how I know.

And the quiet one that ruins everything: testing until it passes. This one isn't a bug in the backtest, it's a bug in me. Run twenty variations and one of them looks great — that's not a discovery, that's arithmetic. Something always looks great. The only defence I've found is writing down every test before I run it and keeping the log, so I can see the twenty and not just the one. It's remarkable how much less exciting the winner looks when it's sitting in a list with its nineteen dead siblings. Verification is the actual skill in all of this, and it's mostly the discipline of not letting yourself off the hook.

Okay. So what do I actually trust?

One: it survives real costs. Pessimistic spread, honest slippage, conservative fills. If the edge only exists at perfect execution, it doesn't exist. This kills most things, and it should — it's the cheapest filter and the least negotiable.

Two: it survives windows I didn't tune on. Walk-forward. Fit the parameters on one stretch, test on the next stretch, roll it forward, and never let the test window inform the fitting. And hold one chunk of data completely back — touched once, at the end, never again. The moment you re-run on a holdout after a disappointing result, it stops being a holdout and becomes just more training data with extra ceremony.

Three: it survives forward in real time. This is the one that can't be faked, and it's why every idea of mine that gets past step two goes into a paper shadow that runs on live incoming data and logs what it would have done. No hindsight can leak in, because the future genuinely hasn't happened yet. It's slow and it's boring and it's the only test with actual integrity. It also catches an entire class of problem the historical tests never will — data feed weirdness, timing, the gap between what you thought the rule said and what the code does at 3am on a Tuesday.

Four: I can explain why it works. Not a story fitted afterward — a reason that would have made me predict the result beforehand. If the only answer is "the numbers say so," I've got a pattern with no mechanism, and patterns with no mechanism stop working the moment the market changes its mind about what it's doing. This one's the least rigorous-sounding and I weight it heavily anyway.

The honest state of things: Confluence is built, it's been through this wringer, and I have no live results and no return claims to make. Zero. That rule doesn't bend — a fabricated number would be worth less than nothing here, given what this thing is for. What I've got is a process I trust more than I trust any individual result it produces, which is genuinely the point.

Because here's where I landed after all of it. The backtest's job was never to tell me a strategy works. Its job is to tell me a strategy definitely doesn't, quickly and for free, so I stop wasting live money finding out. Everything that survives is still just a hypothesis wearing a nice equity curve. The market gets to decide, and it hasn't been asked yet.