Rebuild Less, Ship Faster, Patch the Weights

Why the old setup broke

A replicated Git service has two hard duties. A server can't advertise a new branch tip until it can supply every object reachable from it. And the service must settle races when two pushes try to move one branch from the same old tip. OpenAI modeled two standard designs:

Two assumptions dropped

A unique packfile per client. Conventional servers compute exactly what each client lacks. Pando reuses stored packs, so a client may download objects it already has, but the server does little work.

The push path

Pando validates an incoming pack in a single decoding pass while streaming it to blob storage, without unpacking it into a local repository. A multi-key transaction then records a ref version in a metadata store (the new tip, the pack, and its dependencies) and atomically moves the ref pointer. The pointer is the only contended key, so only pushes to the same ref conflict. The metadata work is constant regardless of push size, because very large pushes write only some commits synchronously and defer the rest. The pusher can immediately read back its own write.

The key choice is what the server advertises. A normal server lists every branch so the client can omit objects the server holds. With millions of branches, that list can exceed 100 MiB. Listing more refs also makes packs smaller but harder to reuse, because another reader may lack objects the writer left out. Listing none would make every pack contain full history. Pando by default advertises only trunk, the designated main branch, and OpenAI reports most of the size reduction comes from advertising just one ref.

Worked example: an agent branches from trunk and pushes commits A and B. Its pack holds only what isn't on trunk, so anyone with that trunk version can fetch the branch with that one pack. When it later adds commit C, the pack contains A, B and C again. That redundancy costs client bandwidth, but the median pack was under 20 KiB and the 99th percentile under 3 MiB.

The fetch path

Compaction

Results

Pando has served more than 57 TB in one hour, about 127 Gbps on average, with over 99% of bytes from local disk cache. OpenAI says a petabyte a day takes only a handful of CPU cores. Clones and fetches saturated 10 Gbps links on standard cloud virtual machines. Its simulation had 3PC write latency and WAL read latency climbing as disks saturated, while Pando was unaffected with many thousands of concurrent pushers.

Building and migrating

What stays open

The simulation is OpenAI's own, and the article's text gives few absolute latencies. It doesn't name the metadata or blob stores, explain how merges changed, or quantify the storage cost of keeping every pack level. Tools needing a cross-branch snapshot won't get one. Some client improvements are upstream in Git, others remain in OpenAI's public copy, and any wider release of Pando's technology is only being explored.

How a change reaches the servers

Why full builds stopped scaling

A full build reads, encodes and compares every record, holding the whole zone in memory. Its virtue is that it never needs to know why a zone changed. Its cost grows with the zone, and some zones now exceed 20 million records, heading toward hundreds of millions. Traffic also rose, from about 250 record changes per second in 2022 to about 700, but Fattouche says the count was never the real problem.

Choosing the smallest safe build

Cloudflare made events carry the changed fields, then traced each setting through the code that generates DNS output. Some events stay vague: certain database functions only update a timestamp, which may be bookkeeping or may mean data changed elsewhere. Those still cause full builds. The planner never turns uncertainty into less work.

• Resources only: rebuild specific names. Ordinary changes, such as to A (address) or TXT (free text) records, usually go here.
• Resources and metadata: for nameserver records (which servers answer for the domain) and wildcards (catch-alls such as *.example.com).
• Apex and metadata: settings such as Universal SSL can use this.
• Purge: remove a zone that should no longer be published.
• Full zone: when nothing smaller can be proven safe.

The rule: every key a targeted build writes must match a full build's output byte for byte. Old state goes by scanning Quicksilver under each rebuilt name and removing keys the build did not write.

One name, several zones

During customer onboarding or offboarding, several internal zone objects can briefly exist for one public domain, so one metadata key is shared by all of them. It works like a directory of usable zone IDs. Suppose example.com has Zones 101, 202, 303 and 404, and the key lists the first three. A TXT change at www.example.com in Zone 202 touches only that zone's record keys. A wildcard change must also update the key, which is rebuilt from a freshly read list rather than patched, after the zone's records are written.

Planning twice

Merging work without losing messages

One unified scheduler replaced separate ones for record and full-zone work. If an A record changes at www.example.com and a wildcard changes at *.example.com before a build starts, one build covers both names plus metadata. Merging only adds work: one event needing the full zone makes the whole build full, and a purge overrides any rebuild.

Proving it

Equivalence tests run full and targeted builds from the same state and compare keys byte for byte. Every release also runs through a mirror Zone Builder, which consumes production events but writes to a separate Quicksilver serving no customers. A comparer samples 20% of zones; a dashboard screenshot shows a 0.31% diff rate across 234,313 zones, mostly timing differences. Rollout hashed the zone name, not its ID, so two schedulers never wrote the same metadata key.

Results

Over 24 hours, 92.95% of builds avoided a complete rebuild; 86.81% were resources only. Full builds were 7.04%, down about 40% from what would have been 11.88%. Those remaining average 0.35 seconds, mostly on new, small zones.

The median halving came next, from spending that headroom: shorter rebuild intervals on most plans, and a smaller Aggregator delay once databases rejected lagging replicas. So that gain is mostly less waiting, not faster building. The figures stop at the Quicksilver write, excluding distribution, and are not per-zone guarantees. A finer-grained histogram now puts p90 at about 5.5 seconds and p99 at about 12.

What remains open

Zones above a fixed record count never escalate to a full build, because it cannot finish and its retries block a partition. They rebuild the apex, metadata and named resources instead, accepting a gap: rare cases, such as enabling a zone on another network, need manual rebuilds of individual names. The post gives neither the threshold nor how often this happens. Cloudflare has also dropped its 2022 goal of eliminating full builds, since first publication genuinely needs every record.

Why this is hard

Weights are the numbers that define a neural network's behaviour. They carry no names or comments, and a single behaviour is usually spread across many of them, so you can't find an "if" statement and flip it. Swyx, the newsletter's author, offers an analogy: this is like ignoring source code and reading the 1s and 0s of a very small compiled executable, then saying what it does, fixing its bugs, or writing a new one directly.

What the researchers report

Going by the summaries, the demonstrations fall into three groups:

Commenters say the agent had access to a Python sandbox, and that it made about 50 attempts on this task. Swyx also flags that the claim of improving a model "in a general direction" with 50 steps may be an overstatement.

The looping objection

The strongest pushback concerns that baseline. A score of 7.9% looked implausibly low for a model this size, and reports say 93% of its answers hit the token limit, the cap on how much a model may generate. In other words, the model usually got stuck repeating itself. One critic adds that this came under poor sampling settings, the parameters controlling how a model picks each next word, and that the model card warns about this failure.

After the edit, token-limit failures fell to 33%. A rough calculation of ours shows why that matters. Before the edit, about 7% of answers ended within the limit while 7.9% were correct, so virtually every finished answer was already right, or truncated answers sometimes still scored. Afterward, about 67% finished and 48.1% were correct. Accuracy among finished answers did not obviously improve, and may have fallen to around 72%. The score could therefore rise simply because more answers finished, which fits the critics' reading and doesn't require any new math ability.

Other doubts about "native" reading

A detailed Reddit critique goes through the other experiments. These are claims from a commenter and not a formal review:

An early reproduction attempt

One Reddit commenter applied the idea to a smaller transformer checkpoint that finds links between objects on document pages. Astra had to analyze weights numerically first, then run limited tests, without changing the architecture. It checked for broken values, collapsed attention heads, zeroed output layers and matrix rank.

A naive edit, shifting an object bias by -0.35, cut false positives from 43 to 33 but also cut correct objects from 32 to 27, so a no-harm rule rejected it. A later, targeted edit to query and key weights gained 6 correct links and lost none across 102 pages. Recall (the share of true links found) went from 75.69% to 76.03%, and precision (the share of found links that are correct) from 89.09% to 89.49%.

That result is real but tiny, and comes from a small sample. It also shows the practical pattern: an edit needs held-out validation, because it can improve one metric while quietly damaging another.

What remains open

The main open questions are whether the method works on models without an obvious defect, whether it adds abilities rather than removing failures, and whether it reaches dense middle layers without relying on probes. Direct edits also already exist in a narrower form: Heretic 2.0, a refusal-removal tool released the same week, uses conventional numerical optimization. The test for Astra is whether it can do something those tools and a skilled interpretability researcher can't.

One caveat applies to everything below. The published text is an outline of the episode, not a transcript. It names many techniques without explaining them, so this piece marks where the detail runs out.

What the product does

The testing problem

Most AI teams improve a product by looking at where it failed for real users. They read the trace, spot the bad step, and turn the case into an eval, an automated test scored against an expected result. Legora cannot start that loop by reading what happened.

Its answer has three parts in the outline:

The citation result

The outline does not name the model, define "right", or give the size of the test. It also does not say what lesson Lauritzen draws. One plausible reading is that work built around a model's current weakness can be overtaken by the next release. That reading is this digest's inference, not his stated view.

Hallucinations, meaning confident fabrications such as invented sources, matter here because a false citation in legal work can be serious. The episode covers Legora's citation system and whether lawyers actually check citations, but the outline gives no answers.

Models and infrastructure

Lauritzen estimates that models already do about 60% of legal work well, and says the rest takes a smarter model. That is his estimate, and the outline gives no method behind it.

Other model and infrastructure choices are listed with little reasoning:

Tabular Review and long documents

Tabular Review is a spreadsheet-style product the hosts call a million-cell spreadsheet, and the outline says the industry copied it "down to the name". As an illustration of the scale, not a figure from the episode, 10,000 contracts with 100 questions each would make one million cells. One chapter covers what a user learns from the first 10 rows, which suggests checking a small sample before trusting the whole table. The outline does not confirm that.

Team size and open questions

Lauritzen once used a slide about the 300 Spartans, the small Greek force of legend, to argue that 20 engineers would be enough forever. Legora now has 100, and he still feels short-handed.

Other chapters raise questions without recorded answers in the text: whether model labs will cut off legal tools as they cut off coding tools, whether lawyers will work in Legora or directly through Claude, and what happens to law firm partnerships, the billable hour and jobs for junior lawyers. For those answers, and for how the testing methods really work, the episode itself is the only source.

The problem: four players and no safety net

The project began as an audit of everywhere Discord plays video, and it turned up four separate implementations. The post describes three kinds of use:

• The attachments player, the most common one, appears whenever someone uploads a video to a chat.
• The Quest player was the most feature-heavy, with reward markers on the timeline, end cards, progress rings and custom animations.
• Silent video surfaces use video as an effect or backdrop, such as the Orbs animation.

The two biggest players, attachments and Quests, shared no code and had no tests. Every feature had to be built twice and every bug fixed twice.

The missing tests made the duplication hard to fix. With no tests, nobody could say precisely what the old player did, so a replacement could silently drop behaviour. Writing a new player from scratch would have carried exactly that risk.

The silent surfaces turned out to be a smaller problem. They received a lightweight video effect component that plays a silent video without the player machinery, leaving the two big players as the real merger.

Why start with Quests

The team started with the harder player for two reasons. Quests had the richest feature set, so matching it early would prove the design could carry demanding requirements. It was also more contained than the attachments player, which appears across conversations, so the work could develop more quietly.

How the swap worked

The team worked in stages, never replacing the player in one step.

Define one interface. The team designed a single shared interface for both the old Quest player and the new DVP to target. An interface here means the agreed set of inputs and behaviours that callers rely on. With both players behind it, one could eventually stand in for the other.

Accept an early second customer. Partway through, the Clips team asked to build its new embed on DVP. Clips reached early access in late March, seven weeks before Quests shipped. Its needs went into the shared player, including a placeholder thumbnail shown while a clip loads and a download button in the control bar. The team says this felt like a detour but forced it to share as much code as possible.

Details that are cheaper to get right once

A shared player means a detail fixed once reaches every product. The post gives several examples:

• Smooth playhead: browsers report playback progress about four times a second, and not every old player handled that smoothly. DVP runs on its own clock so the marker animates fluidly. The post does not describe how the clock is implemented.
• Controls at any size: controls work from a 240-pixel tile up to fullscreen, with no click target below 24 pixels.
• Readable timestamps: time text sits on dark pills so it stays visible over bright and dark scenes.
• A time readout that holds still: in most fonts a "1" is narrower than an "8", so a running clock jiggles. Monospaced fonts fix that, but the team found them clunky here. DVP instead sets the CSS property font-variant-numeric: tabular-nums, which gives every digit equal width. In gg sans, Discord's brand font, those digits looked padded, so the readout switches to Roboto, Helvetica Neue or Arial.

New features built on the base

As an illustration, not Discord's actual settings: take a two-minute video with one tile every five seconds, giving 24 tiles. Hovering at 0:12 falls in the third five-second window, so the player displays only the third tile of the image.

On the frontend, turning previews on for Clips took two lines of configuration, passing the two file URLs to the player. Previews reached all Clips users and remain Clips-only for now, though other surfaces can enable them the same way.

Variable speed. DVP has seven playback speed settings in the base player. The post mentions 1.5x but does not list the rest.

Results and what stays unanswered

Test coverage grew from none to more than 400 unit tests and 40 Playwright tests. The post's headline number deserves some care. "Up to 94%" is the best case, and it measures code each product maintains, not total code, since the shared player itself still has to be maintained. The product-by-product breakdown appears in a chart whose figures aren't in the post's text.

The experiment also has limits. It shows parity on Quest engagement, but the post reports no playback-quality measures, such as startup time or stalling, and no statistical detail. The largest test is still ahead: the attachments player has the broadest exposure and is only partly rolled out. The post also says nothing about mobile apps, naming only web and desktop.