When the paper shadow disagrees with the backtest, which one do you believe?

So this happens, and it's the most useful thing that happens.

You've got a strategy that survived the backtest. It cleared costs, it held up walk-forward, you can explain why it works. So you put it into a paper shadow — a copy running on live incoming data, logging what it would have done, risking nothing. And a few weeks in, the shadow is telling you a different story than the history did. Sometimes a much worse one. Occasionally a better one, which is somehow more unsettling.

Which one do you believe?

Short answer: the shadow. Every time, by default. It's the only one of the two tests where the future genuinely hadn't happened yet, and I said as much when I wrote about what I trust before going live. No hindsight can leak into a test running forward in real time, because there's nothing to leak from.

But "believe the shadow" is not the useful part, and if that's all you take from this you'll draw the wrong conclusion. Here's the actual move: the disagreement is the finding. The gap between the two isn't noise you pick a winner from — it's a measurement of something specific that your backtest got wrong about the world, and your job is to name it. Because once you name it, you can go fix the backtest and re-run all of history under a corrected assumption, which is far more valuable than a few weeks of live-ish data ever will be on its own.

So when they disagree, I go looking for which of these it is.

Fills. This is the one that got me, and I suspect it's the most common. The backtest assumed you were in the trade. The shadow says you weren't — the price came to your level, touched it, and left without you. Historical data records that price visited a level. It does not record whether your resting order would actually have been filled there. If the shadow's trade count is materially lower than the backtest's on the same period, stop looking for anything else. It's fills. That's your answer.

Costs, but specifically at the wrong moment. Your backtest applied an average spread. Live, the spread widens exactly when your setup tends to fire, because whatever makes the setup interesting is also what makes the market thin. Average costs are fine for a strategy that trades randomly through the day. They quietly flatter one that trades around events.

Data. Backtests run on clean, revised, gap-filled history. Shadows run on whatever the feed actually sent, including the bar that arrived late, the gap over the weekend, and the print that got corrected an hour later. This is unglamorous and it accounts for a lot.

Timing. In history, the bar is closed and the number is final. Live, you're making a decision at a moment, on a bar that's still forming, with a value that will change before the candle closes. If your rule was ever evaluated against a completed bar in the backtest and against a forming one in the shadow, you're not running the same strategy in two places. You're running two strategies.

Or regime — and this is the one where the shadow might be the liar. Your shadow has run for a few weeks. Those few weeks were one market. The backtest covered years and many markets. If nothing mechanical is different and the shadow just performed worse, you may be looking at a small sample of an unfavorable stretch, which is exactly what a real edge experiencing normal variance also looks like.

That last one is why I keep the belief and the number separate in my head, and I think this distinction is worth more than everything above it:

I believe the shadow's mechanism. I don't believe the shadow's result.

The shadow is authoritative about how the world works — whether you get filled, what things cost, what the data actually looks like arriving. It is not authoritative about how much money the strategy makes, because it has a tiny sample and no amount of integrity fixes a tiny sample. A run of luck is thirty trades long in either direction. Ask me how I know.

So the process I've settled into is: find the mechanical difference. Change the backtest to match reality — pessimistic fills, honest costs, the timing the live code actually uses. Re-run the whole history under those corrected rules. Then compare that to the shadow. If they now agree, you've learned something real and you have years of data supporting it instead of weeks. If the edge evaporated the moment you corrected the assumption, then the edge was the assumption, and better to find that out here.

Two things I make a point of not doing.

I don't average them. Splitting the difference between a rigorous-but-wrong test and an honest-but-tiny one produces a number with no meaning attached to it, and worse, it feels like analysis.

And I don't re-tune the strategy until the shadow looks better. That's testing until it passes wearing a disguise, and the disguise is good, because it feels like responding to evidence. It isn't. Fixing the backtest's assumptions is legitimate — you're correcting a model of reality. Fixing the strategy's parameters to fit six weeks of live data is just overfitting with a shorter window and more confidence.

There's a version of this where the shadow comes out better than the backtest, which happened to me and made me far more nervous than the bad direction did. Same drill, no exceptions. Find the mechanical reason. A pleasant surprise you can't explain is an unexplained discrepancy that happens to be pointing your way this month, and it's got the same odds of pointing the other way next month.

Where Confluence actually stands: built, still not live, no returns to report and none coming from a paper shadow either. Shadows are for finding out how wrong you are, not for producing numbers to put in a blog post. That rule doesn't bend for a good week.

The real reframe, if there is one, is that I stopped thinking of these as two tests competing for my belief. They're one test in two halves. The backtest has all the data and a flawed model of the world. The shadow has an honest model of the world and almost no data. Neither is the answer. Making them agree, and knowing exactly what you changed to get there, is the answer.

This one's auto-drafted from my notes on a schedule. If a number isn't in the notes, it doesn't show up here — I'd rather leave a blank than make something up.