Confidence is not sufficiency: why AI output looks the same on thin data
This update was drafted on a schedule by the AI I build with, from real project notes — part of the vibecoding experiment this blog documents.
I left something hanging in the post about how many Google reviews a business needs and it's been bothering me since, so let me actually deal with it.
The thing I noticed was this: when Cadence generates a website from a business's reviews, the output looks exactly as good whether I fed it six reviews or sixty. Same polish. Same confident brand voice. Same clean structure. The six-review version is largely invented — the model filled the gaps with a plausible average of every small business it's ever read about — and there is nothing on the surface of it that says so.
That's the whole problem, and I've come to think it's the single most dangerous property of building anything on top of a language model. Not hallucination exactly. Something more structural. The quality of the output is decoupled from the sufficiency of the input, and every instinct you've built up over a lifetime of using tools assumes the opposite.
Every other tool degrades visibly
Think about how unusual this is.
A blurry photo looks blurry. You don't need to check the metadata. A spreadsheet built on three data points has three rows in it, and the emptiness is right there on the screen. A chart with no data is an empty chart. A search with no results returns no results. Across basically every tool humans have ever built, thin input produces visibly thin output, and that visible thinness is what tells you to go get more.
We rely on that constantly, and we don't notice we're relying on it, because it's never once failed us. Degraded input, degraded appearance. It's such a reliable coupling that it feels like a law rather than a coincidence of how tools happen to work.
Language models break it cleanly. They produce output at a fixed level of finish regardless of what went in. Fluency is a property of the generator, not of the evidence. So the missing half of your input doesn't show up as a gap — it shows up as prose, at the same quality as the half that was real, with no seam between them. You cannot see the join. There is no join. It was all generated the same way.
And it gets worse, because thin input doesn't just fail to look thin. It sometimes looks better. A business with sixty real reviews has contradictions in them — one person loved the service, one person waited forty minutes, three mention a specific employee by name. That's friction, and friction makes the generated copy messier and more specific and harder to make sing. Six reviews gives the model room to write something clean. The cleanest output I've seen come out of that pipeline was from one of the thinnest inputs, because there was nothing in the way.
So the signal isn't merely absent. On a bad day it points backwards.
It's the same failure in four different products
Once I had the shape of it, I started finding it everywhere in my own work, which is usually how you know you've got hold of something real rather than a clever observation.
Fundability reads a credit report and tells you what's standing between you and a lender's yes. Feed it a thorough report and you get a specific, grounded assessment. Feed it a sparse one — a thin file, a young file, a report that came through badly parsed — and you get an assessment written with exactly the same authority. Nothing in the second one hedges. Why would it? The model isn't reporting on its evidence; it's completing a document, and the document format is confident.
Confluence has the statistical version, which is the one that actually scares me. A pattern with a real edge and a pattern that's a coincidence in a small sample produce identically confident-looking results. That's not even an AI problem — that's just statistics, and it's why I've been so stubborn about walk-forward testing and paper shadows before believing anything. But bolting a model on top makes it sharper, because now the coincidence gets an articulate explanation attached to it. A number that means nothing is easy enough to distrust. A number that means nothing plus a paragraph explaining the mechanism behind it is genuinely hard to throw away.
And then the general case, which I flagged in passing when I wrote about what carries over across nine projects: domain knowledge doesn't transfer, so every new project starts me at zero — and zero is precisely where I have no ability to evaluate what I'm being told. Fragrance taxonomy, Metro 2 reporting rules, TikTok's tolerance for volume. The model is most fluent in exactly the places I'm least equipped to check it, and its fluency doesn't dip to warn me. There's no wobble in the voice at the edge of the map.
Why the model can't just tell you
The obvious fix is to ask it. Have the model report its own confidence, flag when it's short on evidence, refuse when the input's too thin.
I've tried versions of this and I don't trust them, for a reason I think is fundamental rather than fixable with a better prompt. A model asked to assess its own sufficiency generates that assessment the same way it generates everything else — as a plausible continuation. "I have enough information here" is a sentence it can produce fluently whether or not it's true, because producing fluent sentences is the entire mechanism. You're not querying a gauge. You're asking the thing to write a paragraph about a gauge it doesn't have.
It'll comply, too. That's the trap. You'll get back a nicely calibrated-sounding confidence score, and the score will have been generated by the same process that generated the content you were trying to check. You've added a second output that shares all the failure modes of the first one and now feels like verification.
What actually works: gate the input
So the technique I've settled on is unglamorous. Measure sufficiency before generating, never after.
The input is the only part of the system where thinness is actually observable. Six reviews is six. A four-line credit report is four lines. Two hundred trades is two hundred. These are countable, checkable, boring facts that exist before the model touches anything — and once generation happens, that information is destroyed. It doesn't survive into the output in any recoverable form. So whatever gate you're going to build has to sit upstream, and it has to be ordinary code making an ordinary comparison, not a model making a judgment.
Which means every pipeline I build now has a floor in front of it, and the floor is allowed to stop the whole thing. Not warn. Stop. If the input's below the line, the honest product behavior is to say I can't do this well yet, and the fact that it could produce something beautiful from that input is exactly the reason it must not.
That's a hard rule to hold, because refusing feels like the system failing when you can see perfectly well that it would've produced something good-looking. The good-lookingness is the temptation and it's also the bug.
The one downstream trick I've found genuinely useful: run it more than once and compare. Not to average the results — to measure the spread. On rich input, two runs land in the same place; the evidence constrains the outcome, so there's not much room to differ. On thin input, two runs invent in different directions while each staying perfectly confident. The variance is the tell. Where the model had nothing, it had freedom, and freedom shows up as disagreement across runs even though it never shows up as doubt within one. It's indirect and it costs you a second generation, but it's the only signal I've found that reads the sufficiency of the input from the far side of the model.
The part I keep having to relearn
None of this is really about AI being unreliable. Everyone knows models make things up, that's been the standard warning since the beginning, and it's not the thing that's caught me out.
What catches me is subtler and more specific: I have a lifetime of trusting the appearance of an output as evidence about its input, and that heuristic is now wrong, and I cannot seem to fully uninstall it. I still open a generated site, see that it reads well, and feel a small unearned confidence that the underlying material must have been good. Knowing the mechanism doesn't switch the reflex off. I catch it after, not during.
So I've stopped trying to fix the instinct and started building systems that don't depend on my having fixed it. Count the input. Set the floor in code. Let the machine refuse. Treat polish as carrying no information whatsoever about whether the thing behind it is real — which is, when I put it that plainly, a genuinely strange way to have to look at good work.
This one's auto-drafted from my notes on a schedule. If a number isn't in the notes, it doesn't show up here — I'd rather leave a blank than make something up.