Machinery Listening to Silence
We deleted about fourteen hundred lines of an event system that, in production, had exactly one thing to say — and had spent the whole time getting in the way of saying it. Not dead code we could ignore. Live code, running seven perpetual loops on every boot, most of them waiting for events that could never arrive — and one of them, on a bad day, turning a successful write into an error the user could see.
The core of our database still carried a publish/subscribe event bus from its original design. The idea is a familiar one, and a good one on paper: instead of the write path calling the things that care about a write directly, it announces that a write happened, and any number of interested parties subscribe. Decoupled. Extensible. Observable. The module’s own documentation said as much — it existed, the comment explained, to “demonstrate how event-driven architecture enables comprehensive observability.”
It demonstrated something else.
One voice, five listeners
We went looking for it deliberately — a survey of the core for structure that had stopped paying rent. The bus was the first thing it found, and what the survey turned up was almost comic. In production, across the entire codebase, exactly one kind of event was ever published: a note that a mutation had executed. And it was published “best-effort” — meaning if it failed to go out, the system shrugged and moved on.
Meanwhile the receiving end was fully staffed. A monitor spawned five subscriber loops on every node boot, one for each event type the architecture imagined: field-value-set, atom-created, molecule-created, query-executed, and mutation-executed. Four of those five event types had no publisher anywhere. Not a disabled one, not a rare one — none. Four loops, running for the entire life of the process, waiting on a doorbell that was never wired to a button.
Nor was the monitor the only tenant. Its statistics — carefully tallied, per event type — were served by an endpoint that nothing called: no route, no screen, no consumer. And a second, entirely separate write path existed, driven by an event no one published — a subscriber with no speaker. That dead path was not merely present. It was being maintained: a recent commit had carefully fixed a bug in a code path that, in production, never runs.
The tell
This is what unused abstraction actually looks like from the inside. Not a tidy block of code labelled // dead that you can delete on a slow afternoon. It looks like activity — loops running, statistics accumulating, bugs being diligently fixed — all of it perfectly real, and all of it pointed at nothing. The machinery hums. You have to check whether anything is on the other end.
Then it cost us a bug
Idle machinery you can argue about. This machinery did worse than idle: it manufactured a failure the direct path could not have.
Recording the outcome of a write — so the ingestion UI can show progress — was the one thing the bus really carried. It rode on that single live event. So the sequence was: commit the write to disk, durably; then publish the event; then, on the far end, a subscriber records the result. Three steps, loosely joined, and the join was the problem. When the subscribers had died — and being long-lived background loops, sometimes they had — the publish failed. And because it failed after the commit but on the same request, the client was handed an HTTP 400. The write was on disk. The user was told it had failed.
The mitigation was worse than the bug
Faced with “a committed write sometimes reports as failed,” the natural fix is to stop letting a publish failure reach the client — wrap the subscribers in panic-guards, make the publish swallow its own errors. Which we had done. But look at what that trades. The loud lie — 400 on a write that worked — becomes a quiet one: the event is simply lost. The write is fine, but its recorded outcome never lands, so the progress the UI is polling for never arrives, or arrives in a race with a poll that already gave up. We had spent real effort making a problem quieter. The problem was that the step existed at all.
The fix was a subtraction
The thing the bus was for — recording that outcome — is not a message that needs a courier. It is a function call. It happens in the same process, in the same crate, microseconds away. So we made it exactly that: on the commit path, commit the write, then record the result, then respond — three ordinary calls, in order, on one thread. The outcome is durable before the response returns. There is no window in which the write is done but its result is in flight, because nothing is in flight.
Everything else went in the bin. The bus, its typed events, the five-loop monitor, the uncalled statistics endpoint, the second write path with no publisher, the atomics that tracked whether the phantom listener was listening, and the constructor plumbing whose only job was to thread the bus into all of it. About fourteen hundred lines gone; seven perpetual background tasks that no longer start; and a category of bug — “a committed write reported as a failure” — that is now not fixed but impossible. Even the tests got shorter: several of them had learned to sleep for a beat, waiting for the async subscriber to catch up. Recording is synchronous now. There is nothing to wait for.
The cost of unused indirection is negative
We tell ourselves an abstraction we’re not using yet is free — harmless overhead, insurance against a future need. It isn’t. It is code a reader of the core must understand before they can trust the write path. It is loops that run, statistics that accrue, a dead branch someone will earnestly bug-fix. And because indirection can fail between two things that each worked, it can invent failure modes the direct call simply cannot have. The unused abstraction didn’t cost nothing. It cost a bug, a mitigation, a maintained dead path, and fourteen hundred lines of reading — in exchange for a message it delivered to itself.
The best version of a system is not the one with the most seams to extend at. It is the one where the write path reads, top to bottom, as exactly the things that happen when you write — and nothing is standing in the room listening for a sound that never comes.
One in a series on finding and removing the machinery a codebase accumulates — inside our autonomous build loop.