Stuck Clusters and Quantum-Proof Keys

Why bother proving this

Brooker built Stability Sim earlier to teach metastable failure in the abstract. He says it got one lesson across: retries and queues make metastability easy to trigger. But he still hears a common objection. Real systems are production-proven, so if these behaviours existed, wouldn't operators have found them by now? He also says that after his earlier post on garbage collection and metastability, a well-known systems academic told him the idea was "a fantasy". Brooker says he has seen it happen in production but couldn't share the details, and he admits the original post was mostly hand-waving. Traces from real software are his answer to that objection.

Then a load spike arrives. Each running query uses about 12 MB of memory, so many queries running at once take memory away from the page cache. With less cache, queries read from disk and run more slowly. Slower queries stay in flight longer, so more of them pile up at the same arrival rate. That uses more memory, which shrinks the cache further. The loop keeps going after the spike ends. If 20 queries per second each held 12 MB for a few milliseconds, they would need little memory. If each now takes seconds, hundreds of megabytes stay tied up. (Those durations are illustrative. The article doesn't give them.)

Brooker likes this example because there is nothing distributed about it. One database on one machine is enough.

etcd: overload that looks like an outage

In the second example, a spike in writes overloads etcd's peers until they look unavailable to clients. The application then resends its requests. The resends add load to nodes that were already too slow, which keeps them looking unavailable. This is the classic retry storm. Brooker's point is that etcd is widely used, well-regarded software, yet "operational experience doesn't seem to effectively shake out these kinds of issues."

etcd: garbage collection as the loop

The third example backs his earlier GC argument. etcd is written in Go, which uses a garbage collector. The run starts at 500 reads per second and then spikes to 1,000. Brooker describes a self-reinforcing loop:

The system "doesn't recover without intercession": someone has to cut load or restart it. Brooker suspects a language without GC could show the same loop through rising allocator work, but says he hasn't reproduced that yet.

Raft losing writes over one lying disk

The post opens with the claim that a Raft cluster can lose writes if even one member's storage lies about fsync(). The text doesn't walk through the mechanism, so here is the standard reasoning (general knowledge, not taken from the article). Take three nodes, A, B and C. Leader A sends an entry. A and B report it as durable, which makes a majority, so the entry is committed and the client gets success. But B's disk only pretended to sync. B crashes and comes back without the entry, and then A fails. B and C still make a majority. Neither has the entry, so one of them can win the election and overwrite that log position. A write the client was told succeeded is now gone. Raft's safety proof assumes acknowledged writes survive crashes, and one node breaking that assumption is enough to break the guarantee.

How the animations were made

Brooker says he generated the animations with Kiro and KiroCrew, AI coding tools, mostly using Opus 5.5. The underlying behaviour comes from the real test environment, which captures the traces the animations replay. He presents animation as a way to build intuition for why systems work and why they fail. The article doesn't describe the test harness, the hardware, how long each run lasted, or how recovery was finally triggered.

What stays open

The post gives two or three numbers per example and no latency or throughput curves in the text. It doesn't say how big a spike each system needs before it tips over, and it doesn't cover mitigations such as retry budgets, admission control or memory limits on queries. The allocator version of the GC loop is still unproven, by Brooker's own account. What the post does establish is narrower but useful: these loops happen in ordinary configurations of software many teams already run.