Thirty Thousand Merges a Day

Why git stopped fitting

The team already used git's partial-repository tools: shallow clones (recent history only), sparse checkouts (selected directories only), and promisor remotes (fetching missing objects later). It also used the standard workarounds of internal mirrors and "seeds," which are snapshots of the repo kept in blob storage. At the peak it ran several hundred git mirrors. As the merge rate climbed, seeds fell further behind the latest commit, and operations that need history, such as diff and blame, became fragile on partial clones.

The team's key observation was that it already used git as a centralized system with one authoritative copy, and almost every client was an agent or job that is always online. Git's offline-first design was working against it.

The design principles

How it works

Fetching is also different from git. In git, the client describes what it already has, and the server computes a custom packfile of what is missing. That is CPU-heavy, and a multi-gigabyte response cannot be resumed if it fails. Scion's client, scn, starts from a root tree, checks its local store, and requests only the missing trees and blobs it wants, recursively. Each request names a specific object that never changes, so any cache can answer it.

An illustrative example: a CI job that only needs services/api/ asks for the root tree, then that directory's subtrees, then the blobs inside. The rest of the repo is never requested. Sparse checkout falls out of the protocol with no special support.

The objects pass through three tiers that all speak the same RPC (remote procedure call) protocol:

1. scion-daemon: one per machine, shared by every job on that machine, seeded from a periodic snapshot.
2. scion-cache: regional servers that stream objects from memory, already in the format the daemon stores on disk.
3. scion-server: the authoritative servers, which read from Spanner or cloud storage.

Any tier could talk to any other, but the tiering is strictly enforced so that a failing regional cache cannot overwhelm the central servers.

Results

• Cache hit rate: above 99% at each tier.
• Infrastructure: 200 times smaller in CPU and memory than the old git mirror fleet.
• Checkout time: seconds instead of minutes.
• CI: a 25% gain in CPU capacity.

The role of Claude

Anthropic says Claude both caused the problem and made the fast fix possible. An early snapshot of Claude Mythos Preview rewrote the hand-written git-to-Scion ingestion pipeline without further instructions. The new version used 20 times less memory, ran 4 times faster, and preserved the pipeline's ordering invariants. Claude also found algorithmic speedups, such as an ancestry query that was missing an index, and helped build hermetic simulations (self-contained test environments) and fuzzing (testing with large volumes of random input).

The authors say every engineer on the team now rates Claude a better coder than themselves, but that much of their job was telling it "no." Examples include agents that scanned the whole commit log to find when a change merged instead of asking the review tool, and adversarial AI review that pushed toward needless defensive complexity. Humans still review every change, but they check intent rather than reading each line. Slack-based Claude "superfans" each watch one component's telemetry for subtle anomalies.

Open questions and caveats

Anthropic says Scion is not a general replacement for git. It only makes sense for enormous monorepos, and it requires its own code review, search and CI. A global merge-queue lock no longer works at this rate, so the team relies on testing after merge and before deploy, and lets Claude fix the conflicts that result. The post does not say how often those conflicts reach production. The team also gives no failure or incident data for Scion itself. Rather than releasing Scion, Anthropic suggests that git hosting platforms adopt its ideas: server-side RPCs, a different negotiation protocol, and sparse and shallow features that work well together. It expects other companies to hit these limits within six to twelve months.

The setup

After development ended, a separate "outer loop" evaluator ran 178 acceptance tests against each service. It used its own fake downstream services and captured Kafka messages itself. The agents never saw these tests. A run counted as accepted only if the service started and passed all 178.

The harness

The author uses Birgitta Böckeler's framing. A behavioral harness combines guides, which steer the agent before it acts, and sensors, which observe results and feed them back.

The guides were:

In Böckeler's terms, the checks were computational: deterministic tools decided pass or fail, not an LLM's judgment. Fixing the failures was inferential, because the agent had to interpret the results itself. The agent ran the loop through the project's test command, which did several things in order:

1. validated the configuration and examples
2. built the project and ran unit checks
3. started the mocks and launched the app against them
4. ran the checks

The agent reran failing scenarios, then ran the full suite as a final gate. Nothing ran automatically after each edit.

One design point matters. If an agent writes both the code and the tests, it can misread a requirement the same way in both. An "examples validate" step checked that the examples matched the specs. The agent was told to keep the contracts and examples as they were and fix the implementation, not weaken the expectations. That gave the checks a source that did not depend on the agent's reading of the brief.

Results (10 runs per model)

Costs are API-equivalent estimates calculated from token counts, not actual bills. Sol is OpenAI's flagship model. Luna is its cost-optimized model, roughly the old "nano" tier. Sol solved the brief from a single prompt every time. For Sol, the harness meant about 10.5× the tokens, about 3× the time and about 7× the cost, with no gain in acceptance. In exploratory runs, Claude Sonnet 4.6, Claude Opus 4.6 and Gemini 3.8 Flash also solved the task unaided on the first try.

For Luna, the harness added about 17.1M tokens, 19 minutes and $0.187 per attempt, and raised acceptance from 3/10 to 10/10. A harnessed Luna run still cost about 33% less than an unharnessed Sol run.

How the unharnessed runs failed

The failed runs were close. Every one still passed at least 144 of the 178 checks, and unharnessed Luna passed 94.8% of checks overall. The misses clustered at protocol details and integration points:

• The 202 timeout flow. Instead of 202, four apps returned 503, two returned 400 and one returned 429. Another created monitors but never reached the final 201 response. In 7 runs the missing monitor link also caused a later check to fail. The author counts those as knock-on effects, not separate bugs. He suspects the timeout detection matched too narrow an exception type, but the logs can't confirm it.
• Kafka wire contract. In three runs, every message arrived but was still rejected. Two runs used Spring's JSON serializer, which added an unexpected __TypeId__ header. One sent "categories": null where the contract allowed only an array or no field at all.
• Exact HTTP contracts. One run returned 415 where the brief required 400. Others sent invalid requests to a generic 500 handler.
• Configuration. One generated client read ORDER_API_API_KEY while the launch set ORDER_API_KEY, which likely broke product creation. Configuration checks in the harness catch this kind of mismatch.

The author calls the harness executable attention. Every one of these behaviors was in the brief. His reading is that requirements can "lose influence" during implementation, which he calls an attention lapse, and a red sensor brings them back. He says plainly that the experiment did not establish why the lapses happen.

Trade-offs and limits

The costs cover only machine generation. They leave out the human work of writing and maintaining the contracts, examples and harness, and the cost of diagnosing a bad result. The sample is small: 10 runs per model, one greenfield Java service, low reasoning effort, and an unusually complete brief. Guides and sensors were added together, so their separate effects can't be measured. The author warns against applying the exact figures to legacy code, other languages or vague requirements.

He also notes that Sol's 10/10 itself depended on the independent evaluator. His revised hypothesis is that model capability changes when sensors run, not whether they are needed. For cheaper models with inconsistent results, he would run the sensors throughout development. For frontier models, he would run them once as a final acceptance gate. He has not yet measured that final-gate-only setup. It is his next experiment, followed by weaker briefs, existing codebases, other languages and locally run open-weights models.

The problem

Cloudflare therefore sorts every operation into one of three planes:

• Data plane: queries. These must be fast and always served from the edge, meaning the data centers near users.
• Control plane: API operations such as create, delete and list. These may depend on central components.
• Background plane: analytics and billing. These can be eventually consistent and run anywhere.

Creating a database

Running wrangler d1 create (Wrangler is Cloudflare's command-line tool) took about 1.6 seconds end to end in the post's example. The flow:

If step 3 or 4 fails, the loop retries with new IDs. Reusing a database UUID with a fresh Durable Object would let two objects believe they own the same database. Why not wrap the whole flow in one Postgres transaction? Because that would hold a database connection open while waiting on a Durable Object that might be on the other side of the world, and that doesn't scale.

Heartbeats as the repair mechanism

Every Durable Object checks in with the control plane every 30 seconds, plus a random per-database offset (jitter) so they don't all arrive at once. The control plane replies with whether the database is active or deleted, along with the account's limits. These reads can tolerate staleness, so they are served from caches or replicas.

A worked failure case: Durable Object A is initialized, then the Postgres write fails. The retry creates Durable Object B and succeeds. Postgres is the source of truth and knows only B, so users can only reach B. Within about 30 seconds A heartbeats, learns its database doesn't exist, and marks itself deleted. Heartbeats also rewrite stale Workers KV entries. Workers KV offers no consistent reads, so each Durable Object acts as the single writer for its own entry. It also stores its active or deleted state locally, so it can reject queries even when Workers KV is out of date.

Keeping queries off Postgres

An HTTP query arrives at the Edge API Worker, which handles authentication, authorization, rate limiting and validation. It looks up the database's routing metadata in Workers KV and passes the query to the Eyeball Worker, a proxy that forwards it to the Durable Object. If Workers KV misses, which is rare, the lookup falls back to Hyperdrive: first its cache near the caller, then its cache near Postgres, and only then a replica. Queries coming from a user's own Worker skip the lookup entirely, because that Worker's D1 binding (a configured connection) already carries the routing information.

Enforcing limits asynchronously

Storage limits are 5 GB per account on the Free plan and 1 TB on Workers Paid. Durable Objects publish their sizes to a Kafka-like queue (Kafka is a widely used message queue). The number of queue consumers is capped, which caps the write load on Postgres.

Summing an account's storage on every check (SELECT sum(storage) ... WHERE account_id = ?) would get slower as the account adds databases. Updating one per-account total row on every change would make writers wait on each other for a lock on that single row. The fix uses three tables:

Durable Objects read their account's total on each heartbeat through Hyperdrive. The cache key contains only the account ID, so millions of databases on one account share a single cached value. Adding the database ID to the key would give a more precise limit but wreck the cache hit rate. To get a shorter cache lifetime than Hyperdrive's 60-second default, the query includes a dynamic_ttl parameter computed as floor(now / X seconds). The parameter changes every X seconds, which forces a new cache key.

Trade-offs and open questions

Limits react a few seconds late, and an orphaned Durable Object can linger for up to about 30 seconds. Cloudflare accepts both. The post gives no latency or throughput figures beyond growth multiples, and no load-test numbers. It also hints that storage enforcement may move to the ClickHouse-and-Quicksilver pattern later.

The launch

What Meta says it built

What independent testers found

Full improved on all ten evaluations, and its largest gains came on agentic tasks, where the model takes many actions to reach a goal:

Full ranks in the top three on 6 of the 10 evaluations. On Artificial Analysis's Coding Agents Index, Wonder Full running inside Muse Code scores 68. That ties Claude Code (Anthropic's coding agent) running Sonnet 5.5 and beats Opus 5.5 at 66.

Vals AI, another evaluator, ranks Full fifth on its Vals Index, ahead of every OpenAI model. On its private ProofBench math-proof test, Full scored 94 out of 100, up from Spark's 58, while the best models score 99–100.

Efficiency and cost

Both new models used about 54,000 output tokens per index task, roughly 10% fewer than Spark 1.3. That is more than OpenAI's GPT-6.1 Sol (38,000) and fewer than Fable 5.1 (78,000). Vals found Full needed about 30% fewer output tokens on Terminal-Bench and half the steps on its Vibe Code Bench for the same score.

Cost is mixed. Wonder averages $4.52 per index task, mainly because it reads a lot of input: 7.69 million tokens per task, against 6.44 million for Full. That puts it below Opus 5.5 ($5.98) and Sonnet 5.5 ($5.46) but well above GPT-6.1 Sol ($0.72). Full's cost per task can't be calculated because it has no published price.

Meta's self-reported numbers are stronger still. On SWE-Marathon, a long software-engineering benchmark, Meta reports 57.5% for Full against 46.3% for Opus 5.5. Treat these as vendor figures until others reproduce them.

The benchmark-gaming question

Labs are often accused of "benchmaxxing," training a model on the kinds of tests it will be judged by. Meta's answer is a plot comparing tasks covered in post-training with tasks that were not. It is meant to show gains on unfamiliar, out-of-distribution work as well, which Meta calls "big model smell."

Critics turned the charge around. Early users posted weak 3D browser demos built with three.js, a JavaScript graphics library. Wang admitted, "we did not explicitly train our model on 3d demo games… in retrospect we probably should've." YouTuber and developer Theo argues that 3D demos are themselves a benchmark that other labs have trained for.

How practitioners reacted

Opinions split. Theo, who had early access, rates it alongside Opus 5.5. He has Opus call Wonder for code and architecture review, but he does not recommend replacing Opus. Other early users praised how precisely it follows instructions and how it checks its own code. They found it weaker at frontend design, SVG graphics and creative writing. The analyst @scaling01 estimates Meta is still 3–6 months behind OpenAI and Anthropic and wants fuller system cards. Theo also notes that Wonder and Full behave like "very different models": Full matches Opus on Artificial Analysis while using fewer tokens.

Rough edges and open questions

On launch day, a bug drained Muse Code usage quotas far too quickly. Meta fixed it the same day and reset everyone's quota, and Wang said other issues remain. Several questions are still open:

• Full has no price, so its cost-effectiveness can't be checked.
• Without a detailed system card, Meta's safety claims rest on its word.
• Meta has not said how large the model is.
• Results on one partner's self-improvement test, its RSI Index, are still pending.

That partner described the models as strong at executing tasks but weaker at the intuition needed for scientific discovery.

The news

Meta's headline results: 18.9% more conversions and 12% higher click-through rate (CTR, the share of ad views that get clicked) across its ad surfaces. It also reports extra conversion uplifts of 5.8% on Facebook, 4.1% on Instagram and 1.6% for off-site conversions (purchases on advertisers' own sites), all "for targeted users over the past year". The post gives no baselines, confidence intervals or definition of "targeted users". It reports four advances.

1. A recommendation model that scales like an LLM

Language models improve predictably with size. Recommendation models usually don't, because their inputs are very varied. Some features are fixed facts, such as demographics or a representation of the ad's content. Others are sequences of past events: clicks, purchases, posts. And the data distribution keeps shifting.

Each Kunlun layer has three parts:

Several changes make it affordable:

Longer histories, bigger gains

GEM now uses user histories twice as long in production, and Meta has shown the infrastructure can handle 25x. It reports two findings. Prediction quality follows a power law in sequence length, meaning it keeps improving at a predictable, slowing rate as histories get longer. And larger models benefit disproportionately from longer histories. Meta argues the two effects reinforce each other: more compute justifies longer context, and longer context justifies more compute.

2. Getting a giant model's knowledge into fast models

GEM is far too slow to score every ad inside a live request. Instead, hundreds of smaller "vertical" models serve traffic and learn from it. Distillation, where students copy the teacher's soft labels (its predicted probabilities), was already in use. Meta calls it low-bandwidth: each training step passes only a compressed view of what GEM knows.

LoopFM, which Meta calls a "structure representation", adds a second channel. GEM's important internal layers are compressed into a compact vector describing each event it processes. Those vectors are lined up with the vectors GEM produced earlier for the same user, forming a memory of that user's intent across all surfaces. Downstream models read this sequence in both training and serving. These vectors come out of the same GEM computation that already produces the distillation labels, so Meta says this doubles knowledge transfer at no extra compute or latency.

3. One model across surfaces and goals

GEM now jointly predicts CTR across all surfaces and conversion rate (CVR) across the main advertiser goals, such as off-site purchases and app installs. Clicks are plentiful and conversions are rare, so patterns learned from clicks help the conversion predictions. Two things make this trainable:

• Cost-free data mixing. Training reads virtually from combinations of existing data pipelines instead of copying them into a merged dataset. The mix is kept close to the order data arrives in real time. Meta says this avoids instability when switching between domains and over-memorizing whichever domain it saw most recently.
• Domain experts. A shared expert learns what all surfaces have in common, while per-surface experts learn local patterns.

4. Training efficiency

The result is MFU doubled to 20–25% over 12 months while total training FLOPs grew 4x. That is still below figures often reported for LLM training. Meta's point is that recommendation workloads fit GPUs built for LLMs poorly.

What's left open

This is a summary post; a companion blog covers the training stack. Missing here: absolute metrics, ablations showing how much each technique contributes, how fresh Solaris embeddings are in practice, and how often the fallbacks fire. Next, Meta plans to train on organic (non-ad) engagement and on text, image and video, and to reach LLM-scale context lengths.