Smaller Locks, Looser Clocks

Ten years of Magic Pocket at Dropbox

Ten years ago Dropbox moved its data off Amazon S3 and onto Magic Pocket, its own on-premises blob store. Facundo Agriel described the system's current scale: about ten exabytes, trillions of blobs, and billions of deletes a day.

He framed reliability around the durability model. It takes three inputs:

The model is only as good as the AFR you feed it. Agriel's lesson was that "failures don't care about your durability model." He gave two examples:

1. Heat. High temperature affected 30% of one drive population, and AFR rose sharply, but only for that drive revision under those conditions. A fleet-wide average would have hidden it. Dropbox built a system to drain those drives early.
2. SSDs wearing out together. The metadata tier was being written harder than the team realized. Those SSDs were installed together and wore at the same rate, so they were heading for a correlated failure. The durability math assumes independent failures, so this is exactly the case it misses.

He also stressed checksums and verifiers that keep confirming stored data is intact.

The I/O squeeze on hard drives

Demirbas found the drive economics the most interesting part. Drives now hold 40 TB. For a decade, higher density reliably lowered cost per byte. That trend is ending: cost is rising while density keeps climbing and I/O stays flat. Over the period Agriel measured, drive capacity grew 19x and bandwidth only 1.5x. Reading one drive end to end now takes 36 hours, which works out to roughly 300 MB/s on a 40 TB drive.

The consequence: if you buy a drive twice as large with the same bandwidth, each stored byte gets half the bandwidth. Buying capacity now buys less and less access to it, so operators will have to plan around I/O rather than capacity. Long full-drive reads also slow down rebuilding a failed drive, which feeds back into the repair-rate term of the durability model. That link is an inference, not something the notes state.

Supply is tight too. Agriel said Western Digital is essentially sold out, memory contract forecasts are up 3x, and Seagate's 2028 nearline capacity is already allocated.

Spanner Omni: Spanner without Google's infrastructure

Sebastian Kanthak presented Spanner Omni, a downloadable Spanner that runs on anything from one laptop to a multi-region cluster of thousands of VMs. Version 1 shipped on September 30, 2026. It is not open source and needs a paid license once the trial ends.

The engineering problem: Spanner depends on three internal Google systems that can't ship with it:

Illustrative numbers, not from the talk: suppose a quorum round trip is 40 ms and uncertainty is held to a few milliseconds. The commit wait finishes while replication is still in flight, so clients see no added latency. Push uncertainty past the RTT and the wait starts to dominate commit latency.

Getting under one RTT doesn't take hardware in every building. A single logically centralized time reference is enough if every node syncs to it. Each region keeps a fallback reference in case the central one becomes unreachable. The notes don't say how uncertainty bounds are computed or what happens if a region loses both references.

Lightning talks worth noting

The five-minute "Gong Show" talks got only brief notes:

The AI sessions, briefly

One AI talk was directly about database engineering. Todd Lipcon (Google) listed learned-systems projects that never shipped: learned B-tree indexes, ML cache eviction, Kepler query optimization, and an RL compaction scheme that gained 2%. His diagnosis was the "tax" of an embedded model: inference latency on every decision, plus permanent opacity and risk. His alternative: have the model write a C++ algorithm, review it, and ship the code. Google's cache-eviction work switched to this approach in 2024.

Several talks Demirbas attended aren't covered at all, including Salesforce's talk on logical corruption.

Why cascades are tempting

The appeal is real. You declare a foreign key from customer_users.organization_id to customer_organizations.id with ON DELETE CASCADE. After that, DELETE FROM customer_organizations WHERE id = 42 removes the organization and every user in it, atomically and with integrity checked. Application code never needs to know the shape of the relationship tree.

As the application grows, more tables point at organizations, and further tables point at those, all cascading. Deleting one row becomes a convenient handle on an entire customer's data. That same convenience is the problem: the work behind one statement now grows with the size of the customer.

The outage: cascade plus migration

Pang describes the failure as one PlanetScale has seen. Writes to most tables start timing out. The cause is a large cascading delete running at the same time as an innocent migration, such as adding a nullable text column.

Here is the mechanism step by step:

As an illustration (these numbers are not from the article), suppose a cascade takes four minutes to delete a large customer. A migration started one minute in turns that table into a three-minute write outage, even though the migration alone would finish in milliseconds. PlanetScale's fix in the incident was to cancel the offending transaction with its live-connections CLI.

PlanetScale has previously written about avoiding large deletes by dropping whole tables instead. Pang notes that cascade-driven deletion doesn't map easily onto that approach.

Unindexed foreign keys

Postgres doesn't automatically index the referencing column of a foreign key. When the column is indexed, the cascade uses that index to find children. When it isn't, deleting a single parent row can trigger a full sequential scan of the child table. Pang calls this confusing to diagnose, because the statement looks trivially cheap.

Cascades and logical replication

Built-in logical replication brings two more traps.

Orphans on the subscriber. On the publisher, deleting org 42 replicates as one transaction of ordinary row deletes: the org, then its users, then their children. The subscriber's apply worker runs with session_replication_role = replica. By default, that setting skips the internal triggers Postgres uses to check foreign keys and run cascades. This is deliberate, since the publisher already sent the child deletes. It breaks down when the subscriber has a child table the publisher doesn't know about. Pang's example is a subscriber-only user_activity_rollups table whose rows cascade from customer_users. When org 42 is deleted, its users disappear on the subscriber but their rollup rows stay, pointing at users that no longer exist. There is no error and no warning, only a violated foreign key. A publication with a hand-picked table list that omits a child table causes the same problem.

His guidance: if you cascade and replicate, publish the entire foreign-key tree. Use FOR ALL TABLES, or FOR TABLES IN SCHEMA (Postgres 15+) listing every relevant schema. Clean up subscriber-only children separately, and run orphan checks to catch what slips through.

The alternative: soft delete, then reap in batches

Pang argues that soft deletes are better in many ways for deleting customer data at scale. Deleting a customer becomes a single-row update:

UPDATE customer_organizations SET deleted_at = now() WHERE id = 42;

That replicates as one small change. Later, a background job "reaps" the real rows, perhaps after a retention window such as 30 days. It deletes in small batches during low-traffic periods and runs VACUUM afterwards so the freed space can be reused. Each batch commits quickly, so a pending ALTER TABLE, and the queries behind it, wait only briefly. A retention window also makes it possible to restore data after an accidental delete.

The costs are real, and Pang acknowledges them. Every read, including reads of child tables, must check deleted_at on the relevant rows. You also have to build and run the reaper. The article doesn't address how to enforce that filter reliably across a codebase. It also doesn't cover how soft-deleted rows affect unique constraints or index size.

The general rule

Pang presents all of this as a case of a broader OLTP (online transaction processing) principle: keep transactions short, small and bounded. As a starting point, he suggests a single transaction should touch hundreds to low thousands of rows, often far fewer, tuned to the workload. A cascade breaks that rule invisibly, because the number of rows it touches is set by the data rather than the code. That hidden size is the core of his objection.

Why the migration had to be fast

GitHub says traffic has grown so quickly that it is running into a hard ceiling on power capacity in its data centers and their region, along with supply-chain limits on hardware. The capacity left was enough to migrate safely but not slowly. Changing the storage architecture and the hosting platform at the same time looked too risky, so the team chose an order: move the data first, then change the architecture.

The problem: the slowest replica sets the pace

Spokes commits a write once a majority of replicas agree. In practice, though, it waits for every healthy replica to answer, up to a timeout, before moving on. GitHub does this so slow replicas don't silently fall behind. The cost is that the slowest healthy replica sets the latency of every write.

An illustrative calculation (not GitHub's figures): if a write needs three cross-region round trips, it holds the lock for at least 75ms. That caps the repository at about 13 writes per second, no matter how much hardware is behind it. A repository already near its old ceiling would fall further behind with every push, and its queue would keep growing.

The fix: serialize per reference

The key point is that Git's consistency guarantee applies to each reference separately. Once the server accepts an update to a branch, whether a fast-forward, a force push or a delete, no client should ever see an older value for that branch. Updates to different branches usually don't depend on each other.

So GitHub is shrinking the quorum's scope from the repository to the individual reference. Only writes that touch the same reference wait for each other. Each write still pays the cross-region latency, but writes to different branches can be in flight at once. Many developers and automated tools push to their own branches, so in practice most writes no longer queue behind one another.

Back to the illustrative numbers: with ten branches being pushed at once, the ceiling rises from about 13 to about 130 writes per second. GitHub describes the gain as roughly proportional to the number of references being written at once. In internal tests with realistic traffic, writes to different references got 3–4x higher throughput at lower latency. That is enough to move the last repositories to Azure.

Separating reads from writes

That adds some latency to each write. As with the cross-region hop, per-reference serialization keeps it from limiting throughput. The virtual replica also batches transactions on different references into a single set of network I/O operations, so a busy repository pays one object-store round trip for many concurrent writes. For now it also runs compaction, and read replicas pick up the compacted state.

With that in place, GitHub can add as many virtual read replicas as a repository needs. Each one fills its cache from Blob Storage and starts serving reads without joining the quorum. It treats object storage as the source of truth and reconciles its local state against it in the background. GitHub's distributed load-testing harness routinely found limits in the current architecture. Against virtual read replicas, the harness hit its own limits first.

Trade-offs and open questions

• Read freshness. Because read replicas reconcile in the background, they can lag, which suits only some read workloads. To serve every kind of read, reconciliation has to move in-line: before serving a request, the replica fetches only the reference state and data that changed since it last caught up. The article says this comes later.
• Write ceiling. Per-reference locking helps only when writes touch different references. Heavy contention on one branch, such as a busy main, is still serialized. The full-copy quorum also remains the upper limit on write throughput.
• Where this leads. GitHub argues that very little of a write needs ordering. Git objects are immutable and named by their contents, so they can be stored without any agreement step. Only the small set of reference updates needs ordering. The open problem is ordering that fast without moving the bottleneck somewhere else, and working out where durable reference state lives and how compaction and garbage collection scale. A final post promises the end state, which GitHub says still includes "a (small) consensus group."

Spokes itself stays. It now handles routing and load balancing, provides an internal API serving more than 100 billion operations a day, and runs "sweepers," background jobs that place, repair and rebalance repositories across storage nodes. GitHub says most of that carries over. The commit protocol does not: in GitHub's words, the protocol that reached a billion repositories won't reach the next billion.

The gap image signing leaves

It cannot catch problems that happened earlier. A malicious commit, a poisoned dependency, or a compromised build runner that swaps the base image all produce an image that is built, pushed and signed normally, so verification passes. The evidence that would expose these problems is scattered across separate systems: who approved the pull request, which runner did the build, which packages went in. Linking that evidence to the digest that finally lands in production is hard. Turnstile changes the question to "does this artifact satisfy the requirements for this environment?"

Attestations that chain together

• a subject, an immutable identifier such as an image digest or a commit hash;
• a predicateType saying what kind of claim it is, and a predicate holding the details;
• a signature from a trusted producer, such as a CI job or platform service;
• storage keyed by the artifact identifier, so anyone can fetch and verify it later.

The important part is how attestations link. Source control and build systems tend to drop the references between a commit, its pull request, the image built from it, and the deployment. Turnstile keys each stage's attestation so it points to the next. The attestation types are:

Here is how the chain works, using the article's example. A pull-request attestation's subject is a commit. Its predicate records that the commit merged into main through PR 42, with two approvals, and that the base branch was protected. A build-provenance attestation ties that same commit to a pipeline, a job and an image digest. Starting from a running image, you can walk back to the people who approved its code. Investigators can use the same trail after an incident.

Keyless signing

The trust question shifts from "who holds this key?" to "which identity made this claim, and was it allowed to?" That fits a world where CI jobs move and services get renamed. It also makes revocation possible: if a CI job is compromised, every attestation it produced during the affected period can be invalidated. Policies can also demand a specific producer. An SBOM signed by anything other than the approved scanning pipeline does not count.

Policies and verification summaries

Protecting the policies themselves

Scaling the hot path

Evaluations are mostly predictable key-value lookups, so Datadog chose DynamoDB for low latency, multi-availability-zone durability, and no manual sharding. The same artifact is often deployed to a service many times, so each process keeps an LRU (least recently used) cache of VSAs. The cache key combines the artifact digest, policy ID, software and control-plane versions, environment, and other input labels. On a miss, Turnstile loads the policy, fetches and verifies attestations, runs Rego, builds a VSA and caches it.

Across about 1 million evaluation traces in August 2026, the policy cache hit rate was 92.8% and the per-artifact hit rate was 75%. Evaluation latency was 36 ms at P50 and 481 ms at P99.

What the article leaves open

• Outages. The article doesn't say whether deployments fail open or closed when AMS is unavailable.
• Revocation and the cache. It doesn't explain how revoking a producer invalidates VSAs already in per-process caches.
• The two hit rates. It doesn't define how they differ.
• Tail latency. The roughly 13x gap between P50 and P99 presumably reflects cache misses, but it isn't broken down.
• Interoperability. The changes to in-toto and VSA are not described, so it's unclear how well the records work with standard tooling.