What a paper shadow can't tell you no matter how long you run it

I've been running paper shadows on Confluence for a while now, and I wrote before about why I believe the shadow's mechanism over the backtest's result. I still do. But there's a follow-on point I skipped, and it's the one that actually matters if you're depending on a shadow to make a go/no-go call: there is a hard ceiling on what a forward test can tell you, and letting it run longer barely moves that ceiling.

The instinct is that time fixes it. Run the shadow another two months and you'll know more. And you will know a little more. But the thing you're short on isn't calendar time. It's setups.

Sample accrues at the rate the setup fires, not the rate the clock ticks

A high-confluence signal — the whole point of the thing — is rare on purpose. If it fires a handful of times a month, then a six-month shadow is a few dozen trades. Not a few dozen days. A few dozen decisions. And you cannot learn the shape of a distribution from a few dozen draws, no matter how honest each draw was. The mechanics can be perfect — real fills, real spreads, real data arriving late the way it does — and you're still trying to read a signal off a sample that a normal run of luck can swing either way.

So the shadow's headline number is close to meaningless for a long time, and "long time" here is measured in occurrences, not months. A selective strategy might need years of forward data to get a sample a backtest gets in one pass over history. That's the trade you made when you made it selective, and the shadow can't buy you out of it.

The thing you most want to know is the thing it under-samples worst

Here's the part that actually bothers me. What do you want a forward test to tell you? Mostly you want to know the bad case. The worst drawdown. How it behaves when the market does the thing it hasn't done yet. Tail risk.

And the tail is, by definition, the part a short sample sees least. A few months of shadow data is a few months of one regime — probably a calm-ish, trending-ish, nothing-broke regime, because most months are. The shadow will look fine. That's not evidence the strategy survives a real dislocation; it's evidence that nothing tested it. The scenario you're most exposed to is precisely the one your forward sample is guaranteed to be thin on until it happens for real, with real money, which is exactly when you didn't want to be finding out.

A backtest at least gets to include 2008, 2020, a few flash crashes. A two-year shadow might contain none of that. On the question of "what happens when it goes wrong," the backtest with its flawed assumptions can still be more informative than a clean shadow, just because it lived through more disasters.

It can't see the trades you didn't take

A shadow logs what the rules did. It's silent on what the rules filtered out that would've worked, and on the setups you tightened away during the shadow because a couple of them stung. That second one is subtle and I've caught myself doing it — nudging a threshold mid-run in response to a small sample, which is overfitting wearing the costume of learning from evidence. The shadow won't flag that. It'll just quietly start reporting on a slightly different strategy than the one you started testing, and the continuity in the log file makes it look like one clean experiment.

And it can't tell you how you'll behave

The shadow risks nothing, so it feels nothing. It never sits through a four-trade losing streak with actual money on the line and decides, at 2am, to turn the thing off one trade before the winner. That's not a strategy property. It's an operator property, and it's often the one that determines the real-world result. No amount of forward paper data measures it, because the whole point of paper is that it doesn't hurt.

So what is the shadow good for

Finding out how wrong the backtest's model of the world is. That's it, and it's a lot. Do you actually get filled at that level? What does it really cost when your setup fires? Is the data as clean live as it was in history? The shadow is authoritative on all of that, and those answers don't need a big sample — you learn them in the first few trades and they let you go fix history and re-run years of data under honest assumptions.

What the shadow can't do is be the evidence that the edge is real and sized the way you think. That needs a sample it accrues far too slowly to give you, and it needs to have lived through a bad market it probably hasn't seen. When people say "I paper traded it for three months and it worked," they've usually confirmed the plumbing and mistaken that for confirming the edge.

Confluence is still built and not live, no returns to report — and this is a chunk of why. The shadow's doing its job, which is telling me where the backtest lied. It's just not a job that ends with a number I'd stake anything on yet.

This one's auto-drafted from my notes on a schedule. If a number isn't in the notes, it doesn't show up here — I'd rather leave a blank than make something up.