[← Blog]

The N+1 That Looked Like Six Bugs

2026-07-02

For three days our own database kept greeting us with a blank window, and over those days we diagnosed — with real conviction — six different bugs. There was one. A textbook N+1 in the data browser, the most ordinary bug in software, billed in the one currency nobody was watching: file descriptors. Everything else we blamed was an echo of it, the universal recovery reflex — quit and reopen — re-armed it on every launch, and when we finally had the fix in hand, it took five releases to get it out the door, because our own release pipeline turned out to be hiding three bugs of its own. This is the whole descent, the experiment that ended the argument, and the proof at the end.

We build LastDB on LastDB, so the machine that kept dying held our own notes and our own task board. That is the point of dogfooding: when it breaks, it breaks us — and we get to feel exactly what a user would feel, which was an app that unlocked, showed its data for a breath, and went dark.

The most ordinary bug in software

Everyone knows the N+1: fetch a list, then issue one query per item on it. Every ORM tutorial warns about it, always with the same moral — it makes things slow. Ours didn’t make anything slow. Opening the Browse tab listed the schemas, then fired a count request for each one — on this node, 459 of them, all at once, straight at the embedded server. The counts exist so the browser can do something genuinely useful: hide the hundreds of empty inferred schemas and show you only the ones that hold data, each with a live record count on its row. A good feature, asked for in the worst possible way.

Here is the part the tutorials skip. Each in-flight request holds open connections, and each connection costs file descriptors — and macOS starts a desktop app with a budget of 256. Not 256 for connections: 256 for everything — the database files, the log files, the window system’s own plumbing, and then whatever is left for the network. The burst blew through it instantly. The server’s accept loop asked for one more descriptor, was refused, and died — and, tidy to the very end, unlinked its own socket on the way down. Seconds after unlock, the app had killed itself. The window went blank, every later request reported “network connection failed,” and the process sat there looking perfectly healthy.

An N+1 is not a performance bug. It is an unbounded fan-out, and it bills you in whatever resource runs out first. Usually that’s time, so we’ve all agreed to call it a performance problem. Ours ran out of descriptors first — and a resource bug does not present at its cause. It presents wherever the next allocation fails.

Two descriptors per question

Why did 459 requests cost roughly 900 descriptors? Because of an architecture decision we stand by, and a consequence of it we never priced in. LastDB’s desktop app is socket-only: the node listens on a local socket with filesystem permissions instead of a TCP port, so there is no open port on your machine for every other local process to poke at. The app’s UI reaches the node through an in-process bridge — and that bridge, faithfully simple, opened one fresh connection per request and closed it after. Both ends of every connection — the bridge’s side and the server’s side — live in the same process, against the same 256-descriptor budget. Two tickets per question, 459 questions at once.

The subtle loss is what we gave up without noticing. When a web page fetches over ordinary HTTP, the browser quietly imposes thirty years of accumulated etiquette: it caps concurrent connections per host at a handful and reuses them with keep-alive. Four hundred fetches queue politely over six connections whether the page author thought about it or not. Moving off HTTP-over-TCP onto our own bridge silently deleted that protection — nobody deleted it on purpose; it simply wasn’t there anymore. The UI code had always been storming; the browser had always been absorbing it. Swap the transport and the storm hits bare metal.

One more property worth naming, because it decides who hits this: the burst scales with the number of schemas, not the volume of data. A schema with 1,600 records costs exactly one request, same as an empty one. It is variety, not size, that grows the storm — and a knowledge-shaped workload on LastDB grows variety as a feature, one inferred schema per shape of thing it learns. Success, in other words, was the trigger.

Six faces

Which is how one bug wore six disguises. Over three days we confidently blamed, in order:

• An unlock/keychain regression. The window died right after unlock, every time — of course it looked like unlock. And here the investigation got genuinely confusing, because we found real bugs. Sensitive reads after unlock were re-deriving key material from the root instead of using the key the unlock had just placed in memory — a true bug, sincerely fixed. The app was also still touching the OS keychain from an earlier era of its life, which after a product rename produced genuine permission-prompt storms — also true, also fixed, secrets moved into the node’s own encrypted-at-rest store where no surprise dialog can interrupt a headless process. Two real bugs, two real fixes, zero effect on the blank window. Nothing anchors a wrong theory like finding true evidence for it.
• Data loss. The window was empty; the data was fine, every byte, the whole time.
• A stale build. We rebuilt; the failure looked identical. (Hold this thought — a stale build shows up later in this story as a real villain, just not this scene’s.)
• Warring background daemons. A real hazard — over the bad days we had accumulated supervisor scripts and watchdogs that could genuinely fight over one socket. We removed them all. The app kept dying, just with less clutter around it.
• A cloud-sync auth failure. Its retry loop was genuinely noisy in the logs, which made it a superb decoy — a bug-shaped object, loudly innocent.
• A haunted socket that “kept vanishing.” Which was true, and was the bug’s signature, not its cause.

Every theory had evidence, because the failure was real — only the explanation kept changing. We even shipped a release for one of the theories. The theory was wrong; the release was sincere. The tell we kept ignoring: if a symptom survives one clean rebuild, it is not a stale binary — and if the window dies right after the same tab opens, look at what that tab does.

The experiment that ended the argument

A theory that good deserved better than belief, so we made it falsifiable. First we reproduced the failure without the app: 459 concurrent requests fired at the node from the command line — the same burst Browse produces, minus everything else the app does. At the default budget, the result was total: all 459 connections refused, the accept loop dead, the socket file gone from disk. Twice, to be sure. Not a degradation — a total stop.

Then the other half of the A/B: relaunch the identical binary from a shell that granted it a 65,536-descriptor budget — one command, no code changes — and fire the identical storm. All 459 requests answered in about one second. Server healthy, socket present, window full of data. Same binary, same data, same click; only the budget changed. Every one of the six theories predicted the storm would kill the second run too — the daemons, the stale build, the haunted socket were all still “there.” They didn’t survive the toggle.

A diagnosis you can toggle is the only kind worth shipping a fix for. Sixteen hours of theorizing produced six candidate causes; two commands produced one answer. That asymmetry — and the embarrassment of it — is most of why we’re writing this down.

Five releases to ship one fix

The fix itself is small and boring, which is exactly what you want. Getting it into users’ hands was neither, because the moment we tried, our release pipeline began failing in unrelated ways of its own. It took five consecutive release attempts to ship one two-part fix, and every failure taught us something we’d rather have learned cheaper.

Attempt one shipped last month’s app. The desktop UI is compiled into the server binary at build time — which means a cached binary is a time capsule, and a release pipeline with a shared build cache can exhume one. Version .8 went out the door carrying a binary from weeks earlier: correctly signed, correctly packaged, wrong contents. Installing a “new” release silently downgraded the app. So we added a release-integrity gate: before packaging, scan the built binary and refuse to ship unless it literally contains the version string being released. Simple, mechanical, obviously correct.

Attempt two: the gate rejected a correct binary. Version .9 built perfectly — fresh compile, right version baked in — and the gate failed it anyway. The gate’s shell one-liner piped a string-scan of the binary into grep -q, under the shell’s strict pipeline mode. grep -q does something reasonable that becomes a trap: it exits the instant it finds a match. The scanner, still streaming a hundred-megabyte binary into a pipe nobody is reading, is killed by the operating system mid-write — and strict mode dutifully reports the pipeline as failed. The check failed because it succeeded: finding the match early is what killed the producer. The fix is one character of intent — count matches instead of exiting on the first — so the reader drains the stream and the producer exits in peace.

Attempt three: the smoke test couldn’t find its own script. Version .10 passed the repaired gate, then died in the post-build smoke test — the job that mounts the built disk image and verifies the packaged binaries. It exited with the shell’s tersest verdict, 127: command not found. The verification script it wanted to run lives in our repository; the smoke-test job downloads the built artifact but had never checked out the repository. A guard we’d added in one change assumed a filesystem another change had never provided. Meanwhile the binary inside that failed release was perfect — we pulled the artifact, verified it by hand, and used it while the pipeline caught up.

Attempt four failed on weather — the notary service that blesses macOS apps timed out mid-ceremony. Nothing to learn except that the cloud is someone else’s computer; retry.

Attempt five went green end to end — build, integrity gate, smoke tests on three install scenarios, publish. First fully clean run in the pipeline’s life, and the first release whose safety rails had all, themselves, been tested by failure.

What we changed

The fix has three layers, in increasing order of ambition:

• The server raises its own budget. First thing at startup, before any listener binds, the process lifts its descriptor limit from the desktop default to a working figure — clamped to what the OS allows, never lowering an already-generous one. A local server embedded in a GUI app cannot accept 256 as its operating budget, and it must not depend on how it was launched: double-click, login item, or shell all get the same headroom now. The startup log states the resulting limit, so the next investigator reads it instead of discovering it.
• The fan-out is bounded. Counts flow through a small worker pool — eight in flight, the rest queued — and stream into the UI as they resolve, populated schemas surfacing first. The burst stays eight deep whether you have ten schemas or ten thousand. This is the browser’s old etiquette, rebuilt on the new transport as an explicit decision instead of an inherited accident — and it protects the node from any future fan-out mistake in the UI, not just this one.
• The N+1 itself is scheduled to die. The endgame, queued as follow-up work: the schema list should carry its own record counts, so the browser renders complete from the one request it was always going to make — zero extra calls. And for nodes that grow truly large, the count belongs in metadata maintained on write — increment on insert, decrement on delete — making the answer O(1) no matter how big a schema gets. Bounding the storm was the fix; deleting the reason for the storm is the design.

We hardened the echo chamber too — launches refuse to steal a live socket, the watchdog probes what the app actually serves, failing sync backs off — but those were amplifiers. The disease was the N+1.

Proof, the only kind that counts

We’d already been burned this week by believing green dashboards, so the acceptance test for the fix was the hostile path, end to end: install the published release, launch it normally — the double-click path, the one that hands out 256 descriptors — and try to kill it. The startup log reported the raised limit. The same 459-request storm that had killed the server twice came back 459-for-459, all answered, socket alive. Then the human path: unlock at the password screen, open Browse, and watch it render seventeen populated schemas with live record counts — the hundreds of empty ones correctly hidden for the first time — while the node’s health check read steady underneath. The blank window is gone because the thing that blanked it can no longer happen, and we know because we did the exact thing that used to cause it, on the exact build users get.

The rules we kept

• When one incident appears to have six causes, suspect you are cataloguing symptoms. Root causes are usually singular; echoes are plentiful. Finding a real bug is not finding the bug — we fixed two true ones that changed nothing.
• Never relaunch first. Run the thirty-second checklist — descriptors, sockets, instances, log tail — name the cause, then launch once. On the wrong bug, the restart is the bug.
• Make the diagnosis falsifiable before shipping the fix. One controlled toggle — same binary, same storm, one variable — outweighs any volume of theory. If you can’t A/B it, you’re still guessing.
• An N+1 is an unbounded fan-out, not a slowness. It bills in whatever runs out first — time, descriptors, memory, quota — and it presents at the next allocation, not at its cause. Bound every fan-out at the source; better, move the loop server-side and delete it.
• Price your transport swaps. The browser’s connection pooling was load-bearing and invisible; replacing the transport deleted it silently. When you take over a layer, you inherit every courtesy it used to provide — enumerate them, or rediscover them like this.
• Fixed budgets that scale with success are ambushes. 256 descriptors, the kernel’s argument-size cap, an API gateway’s timeout: each is generous until the day your data isn’t small, and each fails at full speed. Find the ceilings before they find you, and log your headroom.
• Guards are code. Test the gate, not just the gated. A safety check with a false-negative mode is an outage generator pointed at your own releases.
• Verify on the hostile path. The fix isn’t real until the published artifact survives the exact scenario that killed its predecessor, launched the way a user launches it.

This is also the second invisible ceiling we’ve hit in a week — the kernel’s argument limit was the first. Both scale in exact proportion to success: more schemas, more data, more of everything, until a fixed budget nobody remembers agreeing to is suddenly the whole story. For the cloud-side half of this saga — three stacked bugs behind one sync outage — see Anatomy of a Sync Outage; for why our releases now demand proof of the user-visible capability, Prove It To Land.

Built with Brain and Kanban — open-source apps on LastDB — inside our autonomous build loop.

[← Blog]