Why silent failures are worse than loud ones

I've written the phrase "the failure here is silent" twice now — once about posting at volume, once about why the outreach bot can't press send — and both times I said it like it was the end of the thought. It isn't. It's the setup. The actual question is what you do about it, and I've been dodging that because the answer is unglamorous.

So let me do the boring half properly.

A loud failure throws. Something goes red, a build breaks, a request 500s, and the system is screaming at you in a language you already speak. That's not a crisis. That's a notification. Loud failures get fixed on the same day they happen, basically for free, because the finding is the hard part and the system did it for you.

A silent failure returns success. That's the whole definition and it's worth sitting on. The job ran. The status is green. The log line says done. And nothing actually happened, or the wrong thing happened, and the report telling you it's fine was produced by the same broken thing you'd be asking.

What makes a failure silent

Here's the property underneath all of them: the thing that would tell you it's broken is the thing that's broken.

That's it. That's the whole shape. You're not missing an alert because you forgot to write one. You're missing it because your monitoring asks the system how it's doing, and a system that's failing silently answers fine, sincerely, because as far as it can tell it is fine. The exception never got thrown. The code path that would have raised the alarm is the code path that didn't run.

Which means the ordinary instinct — add error handling, catch more, log more — doesn't help at all. You cannot catch an error that was never raised. All that extra logging just produces a more detailed record of a thing that isn't happening.

And automation is exactly where this gets expensive, because the entire point of automating something is that nobody is watching it. That's the deal you made. A human doing the job by hand is a continuous, extremely good anomaly detector who notices when today's batch looks weird for reasons they couldn't articulate. When you automate the job, you remove that detector — and what most people put in its place is error alerting, which covers precisely the category of problem that silent failures are not.

So you end up with a system that's well-instrumented for the failures that would've been cheap anyway, and blind to the ones that cost weeks.

Four ways my own things broke quietly

I'm not theorizing. Every one of these happened to me, and I found every one of them late.

The database moved and nothing errored. Supabase migrated their pooler to a new host — the connection string I had baked into production pointed at the old one. The queries didn't fail. They hung, which is a different and much worse thing, because a hang looks like slowness and slowness looks like load. There's no error to grep for. Just a system getting quietly less responsive until it hit a timeout ceiling, and a stack trace that pointed at the timeout rather than the cause.

A free-tier database went to sleep. A project sitting on a free Supabase tier auto-pauses after about a week of no traffic. The hostname stops resolving, so every login on that project dies at once. And look at the shape of that bug: the trigger is not being used, which means the project least likely to have someone watching it is exactly the project it happens to. The failure and the reason nobody noticed are the same fact.

A routine push would have clobbered production. ClipForge has a deploy setup where prod is served from local vercel --prod runs, and the main branch is behind what's actually live. So a perfectly ordinary git push origin main auto-deploys stale code over a working production site. And every signal in that chain is green — the push succeeds, the build passes, the deploy reports healthy. The site just silently goes backwards in time. Nothing anywhere is in an error state.

A post that nobody sees. The platform accepts the upload, returns success, shows it in your account, and distributes it to approximately zero people. This is the one I wrote the whole volume post around. There is no API response for we've decided to stop showing your stuff.

Different systems, one pattern. In all four the success signal was real — the push really did push, the API really did accept — and completely uninformative about the thing I actually cared about.

Alert on absence, not on errors

The fix is an inversion, and once it clicked for me it changed how I build every scheduled thing.

Stop asking did it break. Ask did it do the thing. Errors are a proxy for that question, and they're a bad proxy, because the interesting failures never generate one. So here's the actual toolkit, roughly in order of how much they've earned their keep:

1. Heartbeats — page me when you stop hearing from it. The single highest-leverage change I've made. Instead of the job telling you when it fails, the job checks in every time it finishes, and something external screams when a check-in doesn't arrive. A dead man's switch. The absence of a signal becomes the signal, which is the only way to catch the failure mode where a cron simply stops firing and produces no evidence of anything at all. A job that dies completely is invisible to every error-based alert you own, forever. It's visible to a heartbeat in one missed interval.

2. Floors and ceilings on the work done. Don't just check that it ran — check that it did a plausible amount. A clipping run that produced zero clips succeeded. Technically. Zero is a number, and it should trip something. The ceiling matters just as much: a run that produced five hundred where it normally does a dozen is equally broken, and it's the direction people forget to check because more looks like winning. What you want is a range, and an alarm on both walls of it.

3. Verify one layer past the thing that reported success. The API returned 200. Fine — but did the row land in the table? Did the file end up where it said? Whatever component told you it worked is exactly the component you shouldn't be taking as evidence. So check the outcome from somewhere else. It's a small amount of extra code and it's the difference between trusting a claim and confirming a fact.

4. Invariants on a timer. Cheap assertions about the world that should always be true, checked independently of whatever produced the state. Nothing stuck in running for over an hour. Every active account has posted in the last day. This count never decreases. Each one is three lines. They catch the drifted-into-wrongness class of problem, where nothing ever failed but the system arrived somewhere it shouldn't be.

5. Make idle systems prove they're alive. If your login can break from disuse, the answer is something that logs in on a schedule and complains when it can't. Synthetic traffic on a path you care about. It feels silly to write a robot that uses your own product every morning until the morning it saves you.

The trap I keep having to avoid

None of this works if you build too much of it, and this is where I've gone wrong before.

An alarm you've stopped believing is worse than no alarm. It costs you the same attention, teaches you to swipe the notification away, and hands you a feeling of coverage you haven't got — which is the exact failure I described about rubber-stamping the outreach queue. Ceremony without substance. Ten noisy checks that fire weekly are strictly worse than two that have never once fired for a boring reason, because the second kind still means something when it goes off.

So: few alarms, each one load-bearing. If something pages me and the answer is "yeah that's normal," it doesn't get muted. It gets deleted or fixed, that day, because a check I've learned to ignore has already stopped working — I just haven't admitted it yet.

The honest part

Every example above, I found late. Not one of those monitors existed before the incident it would have caught. I found the hung queries by chasing something unrelated, I found the sleeping database because I happened to try logging in, and the deploy hazard I found by reasoning about it rather than by being burned — which is the only one I get any credit for.

That's the actual pattern of my last year: I build the detection after, as a scar. It's an okay way to learn and a terrible way to run things, and the only real improvement I've made is that the list above now goes into a system while I'm building it rather than after it embarrasses me.

It's the same family as output that looks confident on thin input, really. A generated site built from six reviews and a cron job that quietly stopped firing three weeks ago are the same bug wearing different clothes: a system reporting a success it hasn't earned, in a voice indistinguishable from the one it uses when it's telling the truth. In both cases you don't get to fix it by looking harder at the output. You have to go check the world yourself.

This one's auto-drafted from my notes on a schedule. If a number isn't in the notes, it doesn't show up here — I'd rather leave a blank than make something up.