The Fix Was Subtraction
The slowest thing in our own tooling wasn’t a missing feature. It was a small tower of fixes, each one correct, each one added to survive a mistake nobody had noticed. Pulling the tower down was faster than anything we could have built on top of it — and pulling it down started with deleting the thing at the bottom.
We build LastDB with two apps that run on LastDB: a kanban board the agent fleet works from, and a Brain the fleet reads and writes. Both had gotten slow in the small, corrosive way that doesn’t trip an alarm. Showing a single card took the better part of twenty seconds. A single write to the Brain took five. Nothing was broken; everything was just heavy, all the time, and the fleet waded through it.
Two wrong stories
The first explanation was load. There was a release build running; the machine felt busy; “it’s a build storm” is the kind of sentence that ends an investigation, because it sounds like a diagnosis. Then someone actually looked: the CPU was sixty-two percent idle. Whatever was slow was not fighting for a processor.
The second explanation was the handshake — the little authentication step each command does before it talks to the node. The logs even said it failed: exchange failed — proceeding unattested. Obvious culprit; surely it was hanging. So we timed it. It failed in three hundredths of a second. A clean, fast failure that the command shrugged off and moved past. Not the villain either.
Both stories were confident, plausible, and wrong. The thing that finally told the truth was the least clever move available: put a timestamp on every line and watch where the seconds actually went. Reading one card by name, the command did a few quick things — and then sat for nineteen seconds inside a single step, hauling the entire collection of cards across the wire, more than a thousand rows, to pick out the one it had asked for and throw the rest away.
The assumption underneath
Why would a program scan a thousand rows to find one it can name exactly? Because somewhere in its history the client decided the store couldn’t filter — that “fetch everything and find it in memory” was the only shape a read could take. Written down once, that assumption never got questioned, because everything downstream of it worked. It was slow, but it was correct. And correctness is a great disguise for a mistake: it keeps the thing alive long enough for other code to grow up around it and depend on its shape.
The tower
And other code had. A full scan of a large, long-lived collection is not just slow — it’s occasionally flaky. The scan reads the collection page by page, and a row that’s plainly there can fall off a page mid-read, so every so often the scan came back empty-handed for a record that existed. So a retry loop had been added, to ride that out. But retrying every miss made a genuinely-absent record expensive, so a “fast-miss” heuristic had been added on top: if a page came back populated but without your row, stop — that’s an authoritative no. But a brand-new collection starts empty, and the retry loop would stall on it, so a cap had been added on top of that, to bound the wait.
Three layers, each a real fix to a real symptom, each locally correct. And every one of them existed to defend the scan — to make a read that hauls a thousand rows tolerable, predictable, and safe. None of them defended anything the reader actually needed. They were a tower built entirely on a foundation that shouldn’t have been poured.
The store could always do this
LastDB serves a keyed read: ask for a record by its name and it returns that one row, straight off an index, in a few hundredths of a second. It always could. The scan wasn’t a limitation the client was working around — it was performance the client was leaving on the floor. The whole tower had been built to survive the absence of a thing that was there the entire time.
So the fix was a keyed read where the scan had been. And here is the part worth keeping: the tower didn’t need to be ported, tuned, or carefully preserved. It needed to be deleted. The retry loop rode out a flake the scan produced — a keyed read doesn’t flake (we hammered it a hundred and eighty times, sequential and concurrent, and it never once missed). The fast-miss heuristic sorted “empty page” from “your row fell off the page” — a keyed read has no page. The cap protected against a stall that no longer exists.
We didn’t adjust any of it. We took it all out. Reading a card went from about twenty seconds to a tenth of one; the Brain write went from five seconds to one. The change removed more code than it added.
What the whole detour was actually about
Two lessons came out of it, and the small one is the famous one: measure before you name a cause. We had two fluent explanations — load, then the handshake — and both survived right up until a timestamp on each line made them impossible. A story that sounds like a diagnosis is not a diagnosis. The seconds are somewhere specific; go find where.
The larger lesson is about the tower. Complexity accretes to defend a wrong assumption. No one sits down to build a retry-loop-fast-miss-cap apparatus; it grows, one honest fix at a time, each responding to a real symptom thrown off by the mistake at the base. Every layer is defensible on its own. The whole is a monument to something that was never true. So when you find code that is slow and thickly wrapped in defenses, the useful question isn’t how do I make the defenses better. It’s what are these defending, and is it real. Sometimes the answer is that the thing at the bottom shouldn’t be there — and the best change you can make is to take it out.
The most satisfying fixes don’t add a clever thing. They remove a thing that was never needed, and quietly demolish the scaffolding that had grown up to survive it. A keyed read, and a subtraction.
More on distrusting a confident story until it’s proven: Prove It To Land.