Today: cryptographic proof of which model answered, a disputed weight-editing result, a Git backend rebuilt for agents, legal testing blind, and benchmarks with a clock.
Picked from 22 new articles across 24 sources · arrow keys for other days
Today in 30 seconds
30s left
Anthropic's open-source FlashProve proves Llama 4 responses at 110 tokens per second, using 70–95x normal GPU time and 19 MB proofs. story 1 ↓
GPT-6 Astra edited 0.016% of a small model's weights, lifting its math score from 7.9% to 48.1%; the result is disputed. story 2 ↓
OpenAI's Pando Git backend cut median push latency nearly fivefold by dropping repository-wide snapshots and per-client packs; standard Git still works. story 3 ↓
Legora tests its legal agent without seeing customer data, using simulated client worlds built by three engineers and three lawyers. story 4 ↓
Frontier models pass at most 36% of Realtime-Bench's 50 tasks, which penalize slow decisions, so long reasoning loops can sink solvable work. story 5 ↓
OpenAI's new LifeVal benchmark measures how well AI helps with everyday tasks like finding childcare, managing finances and handling government paperwork.
1/5Anthropic Research· Nov 2
Anthropic's FlashProve proves Llama 4 responses at 110 tokens per second
Anthropic and academic collaborators released FlashProve, which cryptographically proves a given model produced a response without revealing its weights. One GPU proves a 400-billion-parameter model at 110 tokens per second, about 70–95 times normal compute.
Why it matters · Model identity could become something engineers check rather than trust, and the roughly 100x overhead puts a concrete price on auditing routers, evaluated models and possibly compromised servers.
Know this first
Context from the editor, not the article · 28 sec
1 · Why responses are unverifiable
An API returns only text. Good output doesn't identify its source, and re-running the model to compare results requires the weights, which are usually private, plus hardware most users and evaluators can't afford.
2 · Zero-knowledge succinct proofs
A zkSNARK lets one party convince another that a computation ran correctly using a short certificate. Checking it is far cheaper than the computation, and it reveals nothing beyond correctness. Some blockchain scaling networks already use them.
3 · Mixture-of-experts models
2/5Latent Space· swyx .ai· Nov 3
GPT-6 Astra edits network weights directly, but the headline result is disputed
MIT researchers report that GPT-6 Astra can read and rewrite small neural networks' raw weights without training. Its biggest math gain may mainly fix a looping bug, and critics say the method leans on known techniques.
Why it matters · If models can debug other models by editing weights, some training-loop work becomes cheaper; the evidence so far only supports small networks and narrow fixes.
Know this first
Context from the editor, not the article · 28 sec
1 · Weights and training
A neural network's behaviour lives in its weights, large arrays of numbers with no labels. They are normally set by gradient descent, which repeatedly nudges every weight to reduce error, rather than written or read by anyone.
2 · Self-improvement and autoresearch
Recursive self-improvement is the idea that AI systems could accelerate the building of better AI. Andrej Karpathy's autoresearch demo, about eight months ago, had a model run training experiments autonomously by writing code and reading results.
3 · Interpretability and ablation
Mechanistic interpretability reverse-engineers what specific weights compute. Ablation, a related technique, removes a behaviour such as refusal by finding the internal direction tied to it, then editing weights so the model can no longer express it.
3/5OpenAI· Nov 2
OpenAI rebuilt its Git backend, cutting median push latency nearly fivefold
OpenAI replaced its monorepo's Git server with Pando, which coordinates each branch independently and reuses stored compressed files across clients. Git workflows stayed the same; median push latency fell nearly 80%.
Why it matters · Coding agents multiply pushes and fetches, so source control can become the bottleneck; Pando shows which server guarantees one heavily loaded team could give up, and what that cost.
Know this first
Context from the editor, not the article · 30 sec
1 · Git objects and refs
Git stores history as immutable, content-hashed objects: blobs for files, trees for directories, and commits. Refs, such as branches, are movable pointers to commits. A Git service must keep objects and refs available and in step across servers.
2 · Packfiles and fetch negotiation
A packfile compresses many objects together, often as deltas. During a fetch, the client lists commits it has and commits it wants. A conventional server computes the difference and builds a custom pack, which costs CPU and resists caching.
3 · Why writes contend
Two pushes can race to move the same branch, so a server must pick a winner. Many systems impose one order on every update in a repository, which also makes pushes to unrelated branches wait on one another.
4/5Latent Space· Latent.Space· Nov 2
How Legora tests a legal agent without seeing its customers' data
Legora's CTO says client confidentiality stops the company reading its legal agent's traces, so a small team tests it in simulated worlds modeled on customers' work. These are podcast notes, with few details of the methods.
Why it matters · Most agent-debugging advice starts with reading production traces; Legora is a case of building at scale when that option is contractually closed, a constraint shared by many regulated products.
Know this first
Context from the editor, not the article · 27 sec
1 · Traces and evals
A trace records what an agent did: prompts, tool calls and outputs. Evals are repeatable tests that score that behavior. Teams usually mine production traces to find failures and turn them into evals.
2 · Why lawyers' data is closed
Law firms hold their clients' confidential material and owe duties to protect it. Legora says that data belongs to the firms' clients, so its engineers cannot read what the agent did with it.
3 · What legal diligence involves
In due diligence, lawyers review a company's documents before a deal, often in a data room, a secure repository of files. The work is high-volume and high-stakes, and every claim must trace back to a source.
5/5Import AI (Jack Clark)· Jack Clark· Nov 2
Frontier models pass at most 36% of real-time tasks, new benchmark finds
Researchers built Realtime-Bench, 50 web and game tasks where delay costs success. The best AI model passed 36% under real-world timing, against 70–80% for high-reasoning models with the clock paused.
Why it matters · If you build agents, this separates a model's ability from its ability under time pressure, and shows that a slow reasoning loop can fail tasks the model could otherwise solve.
Know this first
Context from the editor, not the article · 27 sec
1 · The reasoning-latency trade-off
Many current models can generate extra reasoning tokens before acting. More reasoning often improves answers but takes longer, which standard benchmarks rarely punish because the task waits for the model to finish.
2 · What a world model does
A world model is an AI system that predicts how an environment changes in response to actions. Interactive versions generate the next video frames as a user moves or issues commands, producing a playable simulation.
3 · How AlphaGo chose moves
DeepMind's AlphaGo paired a policy network, which suggests promising moves, with a value network, which estimates who is winning. Search used both to explore the game, a tidy setup for a game with fixed rules.
Most large language models use all their parameters for every token. A mixture-of-experts model replaces each layer's feed-forward block with many sub-networks and a router that activates only a few, so each token touches a fraction of the weights.
Image: Anthropic Research
Anthropic researchers and collaborators at EPFL, MIT and elsewhere have released FlashProve, an open-source prototype for 1: cryptographically showing that a specific model produced a specific response. On Llama 4 Maverick, a public 400-billion-parameter model, one Nvidia 2 GPU generates proofs at about 110 tokens per second.
The problem
Users can't tell which model answered them. A routing service could substitute a cheaper model, or an attacker could tamper with the token stream. Re-running the model to check needs private weights and expensive hardware. The goal is verification using a CPU and no access to the weights.
How it works
FlashProve uses a 3, a short certificate that a computation ran correctly. It is cheap to check and reveals nothing else.
The provider first publishes a 4, a roughly 6.5 MB fingerprint that pins down one exact set of weights while keeping them hidden. This takes about 39 minutes and lasts for about 16 million requests.
After generating text, the provider proves that the committed weights produce those tokens. A forged proof would require breaking widely believed cryptographic assumptions.
Three design choices make this practical:
Proving only what ran. Maverick is a 5 model, where a router activates a few sub-networks per token. Work scales with the ~17 billion parameters used per token, not all 400 billion, and the proof hides which ones ran.
Proving after the fact. Finished text is processed in one parallel pass, in batches.
Streaming weights. Weights stay in host memory, so one 80 GB GPU with ~290 GB of RAM is enough.
The numbers
A batch of 24 requests of 1,024 tokens each takes about 223 seconds. Each proof is 18.78 MB and verifies in about 17 seconds on 24 CPU threads. Per GPU, that is about 72 times the time taken by 6, a popular open-source inference engine, and about 94 times at vLLM's highest tested load.
The only earlier proof of this model that Anthropic knows of, VerInf, reached about 0.02 tokens per second with a 93.6 GB proof, on different hardware. One closed commercial system, Attestable, reports 53–77 tokens per second on a 31-billion-parameter model.
Who checks the code
Anthropic says Claude produced the improvements, and that its decisions can be hard to evaluate. To counter this, the team stated the security properties in 7, a language for machine-checked proofs. However, the fast C++ verifier and the prover have no formal guarantee.
Caveats
Proofs cover an integer version of the model choosing the single most likely token, so sampled or floating-point replies may not match.
Context is capped at 1,024 tokens.
Each proof checks one token, so 110 tokens per second counts the tokens in each window, not separately proven tokens.
Side-channel leaks, such as proof timing, are out of scope.
Anthropic researchers and collaborators at EPFL, MIT and elsewhere have released FlashProve, an open-source prototype for 1: cryptographically showing that a specific model, and no other, produced a specific response. On Llama 4 Maverick, a public 400-billion-parameter model, one Nvidia 2 GPU generates proofs at about 110 tokens per second. Each proof is about 19 MB and checks in seconds on a CPU. The Apache 2.0 release includes a prover, two verifiers, a technical report and machine-checked security proofs. Anthropic calls this intermediate progress.
Why it's hard
Output quality says little about its source. A routing service could quietly substitute a cheaper model. An attacker on a compromised server could slip malware into otherwise genuine replies. Anthropic also raises a more speculative case: a more capable model running in place of an authorized one and posing as the weaker model to avoid monitoring. Independent evaluators and regulators have a related need to know what is actually being served.
Re-running the model isn't a general answer, because it requires private weights and costly hardware. Provable inference asks for more: a check by someone with a CPU and no knowledge of the weights.
One alternative is trusted execution environments, hardware that attests to what it runs but depends on the manufacturer's tamper resistance. Anthropic says it already met a public September 2026 prototype goal through a separate project that signs inference traffic and recomputes it on a trusted server. FlashProve pursues the same aim in software alone, relying on computationally hard problems. Anthropic sees the hardware and cryptographic approaches as complementary.
How the proof works
FlashProve uses a 3 (zero-knowledge succinct argument of knowledge), a short certificate that a computation ran correctly. It promises four properties: honest proofs are accepted (completeness), false ones can't feasibly be forged (soundness), nothing leaks about weights or intermediate values (zero knowledge), and checking is cheap (succinctness). The architecture itself is not hidden.
The protocol has two phases. Offline, the provider publishes a 4, a short fingerprint binding it to one exact set of weights. Think of a sealed, tamper-evident envelope (an analogy): the contents stay hidden, but they can't be swapped later. Online, it proves that the committed weights yield particular output tokens for a particular prompt.
A worked illustration: suppose a reply contains "the cat sat". The claim for a checked position is that, given the preceding text, the committed model's highest-scoring next token is "sat". This is 5, always taking the top-scoring token. Because the text already exists, the prover can use 6, reading the whole sequence in one parallel pass rather than generating it token by token.
Internally, FlashProve follows an established recipe. It builds an 7, a protocol in which a verifier spot-checks a prover's long messages, then uses hash functions to turn it into a standalone proof, which Anthropic says also offers post-quantum security. Its building blocks include code proximity tests and 8, which in general let a prover show that values fit an error-correcting code or appear in a table. The post leaves the construction's details to the report.
Three engineering decisions matter most:
Prove only what ran. Maverick is a 9 model, where a router activates a few sub-networks per token. A token uses about 17 billion parameters, and prover work scales with those rather than all 400 billion. The proof still hides which experts ran.
Prove in batches, after the fact. The commitment to the weights is opened once per batch rather than once per request.
Stream the weights. Layers are proven in segments, with weights held in ordinary memory, so one 80 GB GPU with about 290 GB of host memory suffices for a model of about 232 GB.
Results
The tests used public 4-bit weights, a server with one H100 and 24 CPU cores, and requests of 1,024 tokens.
Commitment: about 39 minutes, about 6.5 MB, valid for about 16 million requests.
Proving: about 223 seconds for 24 requests, or 110 tokens per second, at 18.78 MB per proof.
Verifying: 17 seconds on 24 threads, 21 seconds on one.
Overhead: about 72 times 10, a popular open-source inference engine, per GPU (66–77 across runs), and about 94 times at vLLM's highest tested load. The vLLM baseline used the model's FP8 release on eight H100s.
System
Model
Reported speed
Notes
zkLLM (2024)
LLaMA-2, 13B
~2.5 tok/s in batch, ~0.001 per new token
One A100
DeepProve
Gemma 3
1.4 tok/s
31 tok/s distributed
VerInf
Llama 4 Maverick
~0.02 tok/s
93.6 GB proof, almost a day to verify
Attestable
Gemma 4, 31B
53–77 tok/s
One H100, code not released
FlashProve
Llama 4 Maverick
~110 tok/s
19 MB proof, 17 s to verify
Read these comparisons with care, because hardware and token accounting differ. The post says each request is a 1,024-token window and "each proof checks one token", and it describes post-hoc proving as covering a subset of tokens. The 110 figure matches 24 × 1,024 ÷ 223, so it appears to count window tokens, not tokens individually proven. zkLLM's two figures show how much that distinction can matter. Attestable's model is also dense, so it gets no expert-routing saving.
Who checks the code
Anthropic says Claude produced the improvements, but it doesn't separate Claude's contribution from the human authors'. It also says Claude's decisions can be hard to evaluate in large systems, so the team stated completeness, soundness and zero knowledge in 11, a language for machine-checked proofs. These hold in the 12, which treats hash functions as ideal random functions.
Two gaps remain. Humans must audit whether the Lean statements mean what's intended, aided by a visualizer, and the running code must match the protocol. An executable Lean verifier is proven to accept only what the soundness theorem's verifier accepts, but it is slow. The everyday C++ verifier has no formal guarantee, though it agreed with the Lean verifier on every proof tested. The prover, which zero knowledge depends on, has no Lean executable and needs separate audits.
What remains open
Exactness: proofs cover an integer model under greedy decoding. Sampled or floating-point replies can diverge, and the small expected error rate doesn't cover adversarial prompts.
Context: the limit is 1,024 tokens, against production windows of hundreds of thousands or more.
Scope: the system relies on mixture-of-experts structure, though Anthropic expects the progress to transfer to other architectures.
Leaks: side channels such as proof timing are out of scope.
Timing: post-hoc proving delays assurance and adds recomputation complexity.
Regulation: proofs can show hardware served inference, not that it wasn't also training on the side.
Remember
The 110 tokens per second counts 1,024-token windows, while each proof checks one token; compare speed claims cautiously.
Only about 17 billion of 400 billion parameters run per token, so expert routing drives much of the prover's efficiency.
Machine-checked proofs cover the protocol and slow Lean verifier, but not the C++ verifier or prover, which still need audits.
An MIT paper, "Sparks of 1," claims frontier language models can inspect and edit other networks' raw weights from a plain-English goal, with no training loop. The model doing the work is GPT-6 Astra.
This write-up rests on Latent Space's AINews summary and the Reddit debate it quotes, not the paper itself.
Why it's surprising
A network's weights are unlabeled numbers, and its behaviour emerges from all of them together. Improving a model normally means writing training code, running compute and reading loss curves. Karpathy's autoresearch demo automated that loop. This work tries to skip it. Latent Space's swyx compares it to reading the 1s and 0s of a very small executable and patching them directly.
What the authors report
Identifying what small models do from their raw weights, such as recognizing Atari-style game policies.
Writing small game-playing networks from scratch, including one for a simplified Breakout.
Repairing a failed car-control policy by editing 3.5% of its weights.
Raising a 0.8-billion-parameter Qwen model's score on 2, a math benchmark, from 7.9% to 48.1% by changing 131,072 weights, or 0.016%, with no gradient-based training.
Why critics push back
The math result draws the most fire. At baseline, 93% of answers hit the token limit because the model got stuck in reasoning loops. The edit cut that to 33%. By our arithmetic, about 7% of answers finished at baseline, close to the 7.9% score, so the gain mostly tracks more answers finishing. The same approach reportedly failed on Gemma 3 270M, a model without the looping problem.
One detailed Reddit critique calls the edit similar to 3, a known method that removes a behaviour by suppressing one internal direction. Here the target would be looping rather than refusal.
The critique also argues that:
Early and late layers can look like readable heatmaps, which makes them easier than the rest of the network.
Reading dense middle layers reportedly failed until the model could build its own 4, which test a network by running it.
Some written networks only beat weak baselines.
Other commenters note that the work ran in a Python sandbox, so "native" reading may overstate it.
Where that leaves it
One commenter's reproduction on a small transformer found a real but modest effect: recall rose from 75.69% to 76.03% and precision from 89.09% to 89.49%, with no previously correct links lost. An earlier naive edit improved a calibration metric but failed validation.
Swyx calls this a 0-to-1 capability while conceding it resembles interpretability that could be done by hand and is unlikely to scale soon beyond simple games and control policies. That caution looks right: the evidence shows useful targeted repair, not a replacement for training.
A paper from MIT CSAIL, "Sparks of 1," by Rikiya Takehi and coauthors including Alex Zhang, claims frontier language models can inspect and edit the raw weights of other neural networks from a natural-language goal, with no gradient descent or training loop. The model doing the work is GPT-6 Astra. Latent Space's AINews led with it, and swyx called it a "MONSTER discovery." Commenters on Reddit read it far more narrowly.
This account rests on the newsletter's summary and the discussion it quotes, not on the paper itself, so some experimental details are second-hand.
The problem
A neural network's behaviour is spread across its weights: unlabeled numbers with no function names or comments, and no single weight usually means anything alone. That is why model improvement normally runs through experiments. You write PyTorch or JAX code, spend compute, watch loss curves, read evals, and adjust data and kernels.
Karpathy's autoresearch demo, about eight months ago, showed a capable model could run that loop itself, which fed a summer of worry about 2, AI systems speeding up the creation of better AI. The new paper asks whether a capable model can skip the loop and edit the weights of a weaker model directly.
Swyx's analogy: ignore the source code and look at the 1s and 0s of a very small compiled executable, then say what it does, patch a bug from a failed trace, or write a new binary from scratch.
What the authors report
The reported demonstrations fall into three groups.
Reading. Astra identifies the behaviour of small models from raw weights, including recognizing Atari-style game policies.
Repairing. Given a failed control policy for a car, trained with 3 (a standard reinforcement-learning algorithm), Astra fixes it by editing 3.5% of the weights.
Writing. Astra produces weights from scratch for small multilayer perceptrons (MLPs, the simplest feed-forward networks) that play games, including 4 Breakout, a simplified Atari environment. Swyx says some of these beat independently trained models.
The headline experiment is on Qwen3.5-0.8B, a small open model. Astra changed 131,072 weights, about 0.016% of the total, reportedly over about 50 attempts. Its score on 5, a benchmark of 500 competition-style math problems, rose from 7.9% to 48.1%. The authors also claim instruction-following gains of up to 3×. The summaries don't explain how Astra chose those weights.
The case against the headline number
The strongest objection concerns the baseline. A score of 7.9% looked implausibly low to commenters for a model that size, and the reported reason is that 93% of answers hit the token limit because the model fell into reasoning loops under poor sampling settings. One critic says the model card warns about this failure mode. After the edit, token-limit failures fell to 33%.
Some rough arithmetic, ours rather than the paper's, shows why that matters. If 93% of attempts hit the limit, about 7% finished, which is close to the entire 7.9% baseline score. After the edit, about 67% finished and 48.1% were correct, so at most about 72% of finished answers were right, even if every correct answer came from a finished attempt. The score gain tracks completion. That doesn't prove nothing else changed, since the problems that finish at baseline may be unrepresentative, but it fits the critics' reading: a failure mode was repaired and no new math ability was added.
The same intervention reportedly failed on Gemma 3 270M, a model without the looping problem. That suggests the method depends on finding a specific fault.
Critics also link the fix to existing work. Arditi and colleagues showed refusal in chat models can be largely controlled through a single direction in the model's internal representations, and removing it is called 6. The critique characterizes Astra's edit as the same recipe aimed at looping. Ablation is mature enough to have tooling: Heretic 2.0, released the same week, replaces single-direction removal with per-matrix optimization aimed at making activations on harmful prompts resemble those on harmless ones.
The other experiments
A detailed Reddit critique goes through the rest of the paper:
Experiments 1 and 2: First and last layers can be spatially structured, with bands of large weights that resemble heatmaps of the input or output. Reading those may be far easier than reading a network generally.
Other commenters say the work happened with a Python sandbox, which makes "natively" a stretch. Swyx himself flags the 50-step improvement claim as a possible overstatement and calls the underlying activity a variant of 9 that could be done by hand.
An early reproduction
One commenter tried the idea on a small transformer checkpoint, requiring Astra to run numerical analysis first, then bounded forward passes, with no architecture changes. Astra checked for non-finite values, collapsed attention heads, zeroed readout layers and matrix rank before proposing reversible edits.
The first candidate shows the hazard. Shifting an object bias by -0.35 cut false positives from 43 to 33 but also cut correct objects from 32 to 27, so a no-harm rule rejected it. A later edit to parent query and key weights, along a class-projection subspace, produced a small gain on 102 development pages:
Metric
Before
After
Child-parent link recall
75.69%
76.03%
Precision
89.09%
89.49%
That amounted to 6 correct links gained and none lost. It is a single small informal test, but it suggests useful habits: treat an edit as a hypothesis, keep it reversible and validate on held-out data.
What to make of it
Swyx argues the field has gone "from 0 to 1" on a very efficient new capability, and that people will exploit it immediately. He also expects it not to scale soon beyond simple games and reinforcement-learning policies. The evidence reported here supports the second half more firmly than the first. A strong model can apparently act as an automated interpretability engineer for small networks, finding and removing some faults cheaply. Nothing here shows it can add a capability to a large model or match gradient descent at scale.
The open questions are how often edits succeed on models without an obvious pathology, how much depends on activation probes rather than weights, and whether the results survive sensible baselines.
Also in this issue
GDPx, a new SemiAnalysis benchmark, covers 206 agentic tasks across 11 economically valuable domains. Opus 5.5 leads at just 12%; Gemini 4 Carbon scores 10.9% at roughly a quarter of the cost. Removing benchmarks affected by safety safeguards reportedly changes the picture, and the benchmark isn't open source.
DeepSeek published a prototype agent harness with one long-lived router session and isolated parallel workers. In its own test, 10 completion time fell from 29.5 to 15.7 minutes against a single session using subagents, but it has been validated only in a small, trusted internal environment.
Remember
Check failure modes before trusting a benchmark jump: here, token-limit failures fell 60 points while accuracy rose about 40.
Whether a model can read weights alone remains unsettled; dense middle layers reportedly required running the network with probes.
Weight edits need engineering discipline: one naive edit improved a calibration metric but lost correct outputs, so validation rejected it.
OpenAI has moved all development in its 1, a single large shared code repository, onto Pando, a Git backend it built itself. Engineers and agents keep using standard Git workflows. Median push latency fell by nearly 80%, and Pando now delivers nearly a petabyte of data a day.
Why Git was straining
2, OpenAI's coding agent, lets one engineer run many changes at once, and each push can trigger several automated jobs fetching the same commit. By around the end of last year, fetches and pushes were slow, merges failed, and reliability sometimes dropped below 90%.
OpenAI modeled two conventional fixes. One keeps full replicas that vote on each branch update, which puts disks and network round trips on the push path. The other orders updates through one shared manifest in blob storage, so even unrelated pushes compete for it, and replicas can lag while indexing incoming data.
Two guarantees dropped
No repository-wide snapshot. Pando offers 3: fetching a branch shows you the history its writer had, and updates to each branch are strictly ordered, but two branches may not reflect the same moment. Pushes conflict only when they target the same branch. OpenAI says users and agents rely on per-branch order, not a global one.
No custom download for each client. A 4 is Git's compressed bundle of objects. Conventional servers build one per fetch. Pando serves packs it already stored, so a client may receive some objects it already has, but the server does little work.
How it works
When a new feature branch is pushed, Pando by default tells the client only about the trunk branch. The client therefore builds a pack depending only on trunk, and anyone with that trunk version can reuse it. The median pack made this way was under 20 KiB, and the 99th percentile under 3 MiB.
Pando streams the pack to blob storage, then atomically records it and moves the branch pointer in a metadata store. On fetch, it picks the stored packs a client needs and returns download links through 5, an existing Git protocol feature. A regional cache called Express serves them, with hit rates above 99%.
Catching up could otherwise take tens of thousands of packs a day, so background workers merge small packs into progressively larger ones. These layers also let Pando choose packs in logarithmic time.
Results and costs
OpenAI reports 57 TB served in one hour and clones saturating 10 Gbps links on standard cloud machines. Its comparison with the conventional designs comes from its own simulation, and the text gives no head-to-head numbers.
The costs: repeated pushes and stacked branches resend everything outside trunk, and peak performance relies on OpenAI's Git build, only partly merged upstream. Storage growth goes unreported.
OpenAI has moved all development in its 1, a single large shared code repository, onto Pando, a Git backend it built itself. Engineers and agents still use the Git protocol and client. In production, median push latency fell by nearly 80%, or almost 5x, and Pando now delivers nearly a petabyte of Git data a day.
The problem
2, OpenAI's coding agent, lets one engineer run many changes in parallel. That means more branches, pushes and merges, and more fetches from development environments, research jobs and continuous integration (CI). The load compounds: one engineer may have several changes ready for CI, and each push can set several jobs fetching the same commit. From around the end of last year, OpenAI saw slow fetches and pushes and failed merges, with reliability at times below 90%.
Any Git service has two scaling dimensions: many writers pushing to different branches, and many readers fetching the same history. Replication makes both harder. A replica cannot advertise a new branch tip until it can supply every object reachable from it, and competing updates to a branch need a single winner.
The designs OpenAI rejected
OpenAI modeled two conventional designs.
3, a protocol for coordinating a commit across nodes, keeps complete replicas. A push is copied to all of them, and a coordinator takes a majority through lock, pre-commit and commit rounds. Reads can be served locally and are consistent, but replica disks and network round trips sit on the push path, and replicas must absorb incoming packs alongside read traffic.
A 4, a record of changes written before they are applied, uses a shared manifest in blob storage. Writers upload 5 (Git's compressed bundles of objects) in parallel, then publish through 6, an atomic update that fails if the value has changed. Every push, even to an unrelated branch, competes for that one manifest. Batching helps at the cost of queueing. Replicas must also read and index each pack before serving it, so they can fall behind under heavy writes.
Two assumptions Pando drops
Pando gives up two things most Git systems treat as requirements.
The first is a repository-wide snapshot of refs. Pando offers 7: fetching a ref means observing the history of the writer who produced it, with updates to each individual ref strictly ordered. Pushes conflict only when they target the same ref. If branches foo and bar both move during a full clone's negotiation, one client may see foo at version 1 and bar at version 2, and another the reverse. OpenAI says users and agents depend on the order of updates to a single ref, not a global order.
The second is a unique packfile for each client. Pando reuses stored packs, so a client may download objects it already has, but the server does almost no computation.
The push path
Conventional servers advertise every ref to a pushing client, so the client can leave out anything the server already has. With millions of branches, that advertisement can exceed 100 MiB. It also makes packs hard to reuse, because another client may lack the objects the writer omitted.
Pando advertises by default only a designated trunk branch when a new feature branch is pushed. The client's pack therefore depends only on trunk, and anyone holding that trunk version can fetch the branch with that single pack. In OpenAI's monorepo, packs produced this way had a median size under 20 KiB and a 99th percentile under 3 MiB.
Pando validates the pack with a single decoding pass while streaming it to blob storage, never unpacking it into a local repository. A multi-key transaction then records a "ref version" (the new tip, its pack, and its dependencies) and atomically moves the ref pointer, so the writer can immediately read its own push. The ref pointer is the only contended key, and each push takes a constant number of metadata operations regardless of size. Very large pushes write only a subset of commits synchronously. Throughput grows by adding Pando servers, metadata capacity or blob stores.
The fetch path
Pando maps a client's "haves" and "wants" to ref versions, then selects packs covering what is missing. It does not assemble new packs. It uses 8, a Git protocol feature that lets a server send download links, and returns an empty inline pack plus those links. Express, a regional caching service, serves the bytes. Consistent hashing routes each pack to an Express server, with cache hit rates above 99%. OpenAI reports clones and fetches saturating 10 Gbps links on standard cloud virtual machines.
Compaction
Sending one link per push would not work, since catching up on trunk could take tens of thousands of packs a day. Each push therefore creates an "L0" pack, and background workers group packs into geometrically larger covers, currently up to L4, similar to compaction in 9, a storage design that merges small sorted files into larger ones. A full cover can be repacked into a single, better-compressed file. Covers also act like a skip list, so choosing packs takes logarithmic metadata reads in the distance between client and target.
An illustration, with a ratio OpenAI does not state: if each level spanned 10 times the one below, a client 12,345 pushes behind could ideally fetch 15 packs (one L4, two L3, three L2, four L1, five L0). Pando also keeps packs at every level instead of garbage-collecting them, partly because reading one object from an L0 tends to touch less data than from an L4.
Results and evidence
Beyond the latency figure, OpenAI reports better merge reliability and 57 TB served in one hour, more than 99% of it from local disk cache. It says a handful of CPU cores can handle a petabyte a day.
The comparison with 3PC and WAL comes from OpenAI's own simulation of 1,000 clients pushing to distinct branches, with 3PC's consistency relaxed for fairness. It shows 3PC write latency and WAL read latency climbing as disks saturate. OpenAI also claims many thousands of concurrent pushers without degradation. The text gives no head-to-head numbers, and these are modeled designs, not named products.
How it was built and deployed
OpenAI says one engineer designed the core, teams of agents implemented it in Rust, and a handful of engineers deployed it, 10 weeks from kick-off to production, with production traffic starting in April. Validation included a Git conformance suite and 10, which replays controlled executions to expose race conditions. Agents using GPT-6 Astra also wrote 11 and 12 formal specifications that found bugs. Pando first ran as a read- and write-through proxy to the old system, deduplicating requests and easing its load while workloads moved over. Hundreds of integrations were migrated too.
Trade-offs and open questions
Repeated pushes to a branch, or stacked branches, resend everything outside trunk, costing client bandwidth. Tools that assume a global ref snapshot lose it. The best performance also relies on OpenAI's Git build, though several fixes are already upstream, and more are in its public repository. These include parallel pack downloads and "explicit haves", which separate a branch's advertised tip from the history a pack may omit.
The article does not report storage growth from keeping every pack level, or how Pando fares outside this monorepo. OpenAI says it is exploring ways to share the technology.
Remember
Advertising just one ref yields most of the pack-size reduction while making packs reusable by other clients.
Scaling came from removing work: Pando servers neither unpack nor regenerate packs, mainly negotiating and selecting stored ones.
Check whether your tools assume a repository-wide ref snapshot; that guarantee is what Pando traded for parallel writes.
Legora, a legal AI company, cannot read a single 1 of what its agent does for a lawyer. A trace is the record of an agent's prompts, tool calls and outputs, and here the material belongs to law firms' clients. CTO Jacob Lauritzen discussed how the company copes in a Latent Space episode.
The source here is the show notes and chapter list, not the transcript, so it offers claims and a few numbers but little mechanism.
The problem
Agent teams commonly improve products by reading production traces, finding failures and turning them into 2, which are repeatable scored tests. Legora can't start there, even though lawyers use its agent for high-stakes work such as reviewing 10,000 contracts in one table or running 3.
Testing in simulated worlds
Three engineers and three lawyers build test environments modeled on customers' work. The notes compare them to the "worlds" of 4, where an agent acts in a simulation and gets scored. One chapter mentions a 10,000-file 5, a repository of deal documents.
Some signal from production survives. Chapters cover product metrics, "paraphrasing traces" and narrowing failures to 10 or 20 types, though the notes don't explain how the paraphrasing works.
The number that stands out
On one citation problem, the model was right 70% of the time, then 99% after one release, and Legora didn't have to change anything. The notes don't define the task or test set, so treat this as an anecdote. Still, it raises a question for teams on frontier models: how much should you invest in working around a weakness the next model may remove?
Lauritzen also estimates that models already do about 60% of legal work well, and says the rest takes a smarter model. The notes give no method behind that figure.
Engineering positions
Team size
Lauritzen once used a 300 Spartans slide to argue that 20 engineers would be enough forever. There are now 100, and he still feels short-handed.
The episode also asks whether model labs will cut off legal tools as they did coding tools, and what happens to the billable hour and junior lawyers. The notes don't record the answers.
Legora, a legal AI company, cannot read a single 1 of what its agent does for a lawyer. A trace is the record of an agent's prompts, tool calls and outputs, and here the underlying material belongs to law firms' clients. CTO Jacob Lauritzen discussed how the company builds under that limit in a Latent Space episode with swyx and Basil Chatha.
One caution first: the source available here is the episode's show notes and chapter list, not the transcript. It gives claims and a few numbers, but rarely the mechanism behind them. Where the notes stop, this piece says so.
The problem
Teams building agents commonly improve them by reading production traces, spotting failures and turning those failures into 2, which are repeatable scored tests. Legora can't start from that loop. Its product also operates at a scale where individual mistakes are hard to spot by hand: lawyers use it to draft and edit Word documents, review 10,000 contracts in a single table, run 3, and coordinate work among lawyers, clients and other agents.
Testing without the data
The substitute is synthetic. Three engineers and three lawyers build test environments modeled on customers' work, which the notes compare to the "worlds" used in 4, the training method where an agent acts inside a simulated environment and is scored on the outcome. One chapter mentions a 10,000-file 5, a repository of deal documents, as a test setting.
Production signals still exist in reduced form. Chapters cover product metrics for when you can't read traces, "paraphrasing traces," and narrowing failures to 10 or 20 types. The notes don't say who or what does the paraphrasing, or how privacy is preserved. Another chapter covers obtaining anonymized examples and floats the idea of buying a law firm for its data.
Legora's privacy stance also shapes its infrastructure. It runs everything on Azure, Microsoft's cloud, and doesn't use ElevenLabs, a voice AI company, for transcription. The notes link the Azure decision to data privacy but don't give the reason for skipping ElevenLabs.
The 70-to-99 story
The sharpest number concerns citations. On one citation problem, the model was right 70% of the time, then 99% after one release, and Legora didn't have to change anything. The chapter frames it around a period when evals didn't seem worth the effort.
The notes don't define the task or the test set, so treat this as an anecdote rather than a benchmark. It does raise a practical question for any team on frontier models: how much effort should go into working around a weakness the next model may remove? The notes don't record Lauritzen's answer. Hallucinations and whether lawyers actually check citations get their own chapters, also without detail in the notes.
What the model can and can't do
Lauritzen estimates that models already do about 60% of legal work well, and says the remainder takes a smarter model. That is his estimate, and the notes give no method behind it. He also says agents couldn't do certain things nine months ago, though the list isn't given.
Legora resisted a 6, a menu letting users choose the underlying model, for years before adding one. On drafting, Lauritzen finds Fable and Opus too chatty and likes Astra's concise writing. The notes don't identify the versions or test conditions, so this is one practitioner's taste rather than a measured comparison.
Product and engineering choices
7 is Legora's spreadsheet interface, which reaches a million cells and which, per the notes, the industry copied down to the name. As an illustration of that scale, not a figure from the source, 10,000 contracts with 100 questions each would yield one million cells. A chapter discusses what a lawyer learns from the first 10 rows, and the notes don't elaborate.
Other pieces compose. Agents can create lists, and a list can start another agent. A legal intake bot works in Slack. Legora builds products from these same pieces.
The notes also flag several technical positions:
Team and open questions
Legora once planned to stay small: Lauritzen used a 300 Spartans slide to argue that 20 engineers would be enough forever. There are now 100, and he still feels short-handed. Chapters also cite 100 legal engineers, and 13, who work directly on customer deployments, help onboard large clients.
The episode also covers matters the notes only pose as questions: whether labs will cut off legal tools as they cut off coding tools, whether lawyers will work in Legora or through Claude, and what happens to partnerships, the billable hour and junior jobs. The answers are in the audio, not in this source.
Remember
Simulated worlds can partly replace production traces, but only if domain experts help build them; Legora pairs three lawyers with three engineers.
A 70% to 99% jump with no product change suggests some workarounds may expire at the next model release.
Lauritzen's 60% figure for legal work is a personal estimate with no stated method, so don't read it as a measurement.
Researchers at UT Austin, Ohio State, the University of Illinois at Urbana-Champaign, NVIDIA, Northwestern and Steadyworks have built 1, a test of whether large language models (2) can act when delay itself causes failure. The best score under real-world timing was a 36% pass rate, from OpenAI's GPT-6 Astra on its high setting. With the clock paused, models using high reasoning reach about 70–80%.
Why timing matters
Conventional evals wait for the model to finish. A ticket sale or an arcade game doesn't, so extra reasoning has a cost and an agent must decide how much it can afford.
How the test works
The benchmark has 50 interactive tasks: 30 on the web (travel, e-commerce, service, security) and 20 games (arcade, puzzle, strategy, simulation). It looks for three abilities. Buying a ticket needs all of them: noticing whether availability has changed, cutting deliberation short as the window closes, and choosing whether to wait or buy now.
Each model can run under three conditions:
Wall clock: real-world latency, as in deployment.
Paused clock: the environment can be paused, giving a baseline for ability without time pressure.
Token clock: the world advances a fixed amount per generated token, so a wordier model acts later regardless of how fast its hardware is.
The field also included models from Anthropic, Google, DeepSeek and Moonshot AI. Jack Clark, who writes the Import AI newsletter, calls the benchmark "an effective proxy for where AI agents will break." The newsletter doesn't give other models' scores or token-clock results, and its paused figures aren't a like-for-like comparison with the 36%.
Also in this issue
Genie 4. Google DeepMind detailed its latest 3, a system that generates an interactive environment, even from a single photo. New features include natural-language edits while a simulation runs, better physics and people, and native audio. Gemini 3.8 Flash, a fast LLM, also sits in the loop as a supervisor: it breaks goals into steps, checks frames to confirm that each step succeeded, and replans when needed. Access is limited to trusted testers for now. Clark still finds it "quite janky," and no quantitative results are reported.
Metis. This startup's position paper proposes keeping claims about the world in an explicit store, called an 4. A trained policy and 5 would steer reasoning toward reducing uncertainty, and an independent judge would have to approve any change to a claim's status, with evidence attached. The aim is auditable reasoning with fewer hallucinations, but the paper reports no results.
Space weapons. Clark also reads a US Defense Intelligence Agency report on anti-satellite weapons, which says China's launches rose from 22 in 2016 to 93 in 2025. He links this to AI through space-based data centers, which would make orbit more contested.
The main research item in this issue of Jack Clark's Import AI newsletter is a benchmark that asks a question most evals avoid: what happens when thinking costs time? 1, built by researchers at UT Austin, Ohio State, the University of Illinois at Urbana-Champaign, NVIDIA, Northwestern and Steadyworks, tests whether large language models (2) can complete tasks where, in the authors' words, "delayed actions directly affect success."
The headline result is that the best model passed 36% of tasks under real-world timing. With the clock paused, models using high reasoning settings reach roughly 70% to 80%.
Why timing is a separate problem
A conventional eval freezes the world while the model works, so a model can reason for as long as it likes. Buying a concert ticket or playing an arcade game offers no such pause. An observation may be stale by the time the model has finished reading it, and a perfect decision can arrive too late to matter. Extra reasoning therefore has a cost, and a capable agent has to decide how much it can afford.
How the benchmark works
Realtime-Bench has 50 interactive tasks:
30 web tasks across travel, e-commerce, service and security
20 game tasks across arcade, puzzle, strategy and simulation
It probes three abilities, which the authors illustrate with a ticket purchase:
RT-Perceive (temporal state awareness): tracking changes in the world from observations. The agent must follow the sale's status and judge whether the availability it saw is still current.
RT-Deliberate (adaptive deliberation): changing the length and depth of reasoning to fit the time available. The agent should stop deliberating as the purchase window closes.
RT-Act (time-aware action selection): timing actions to suit the situation. The agent chooses between waiting for sales to open and buying while tickets remain.
The design's most useful feature is that it runs each model under three clock conditions:
Wall clock: the environment runs under real-world latency, as it would in deployment.
Paused clock: the environment can be paused at will, which gives a baseline for what the model can do without time pressure.
Token clock: the environment advances at a fixed rate per generated token. An action takes effect at the simulated time reached when it is committed, so more tokens mean later execution, however fast the inference hardware is.
As an illustrative example, with made-up numbers: if the 3 advanced one simulated second per 100 tokens, a model spending 3,000 tokens on reasoning would act 30 simulated seconds after it started. That penalty stays the same whether the model runs on a fast or a slow server. The token clock thus isolates how economically a model thinks, whereas wall-clock results also reflect the serving stack, an inference from the design rather than a claim in the newsletter.
The results
The field included OpenAI's GPT-6 Astra, GPT-5.6 Sol, GPT-Realtime-2 and OpenAI Decisions; Anthropic's Claude Fable 5.1 and Claude Opus 5; Google's Gemini 3.8 Flash, Gemini 3.1 Flash Live and Gemini 3.8 Live; Jev 1.13; DeepSeek's V4.1; and Moonshot AI's Kimi-K3.
GPT-6 Astra at its high setting scored highest on wall clock, at 36%. The newsletter doesn't give scores for the other models, category breakdowns or token-clock results, and points to the paper and an official leaderboard for those. Several tested models have "Realtime" or "Live" in their names, yet the top slot went to a general model on a high setting.
Clark calls the benchmark "an effective proxy for where AI agents will break." The gap between roughly 36% and 70–80% suggests that many failures come from timing rather than from a lack of underlying ability. That comparison is loose, though, since the newsletter's paused figures describe high-reasoning models in general rather than a matched pair.
Genie 4: a model supervising a 4
Google DeepMind also published details of Genie 4, the latest in its series of world models, which can turn a photograph or a 2D drawing into an interactive environment. It is a research preview for some trusted testers and will eventually reach subscribers to Google AI Ultra, the company's subscription tier.
Google highlights four changes:
Real-time modification: natural-language commands can change the weather, move the camera or add a person or object, and Gemini adapts the prompt to fit the scene.
Physics and people: simulation of aerodynamics and collisions, plus characters that can be steered interactively. Google calls the people simulation the "most dramatic leap."
Audio: Google says this is its first real-time world model to generate audio natively, including dialogue that adapts on the fly.
Complex events: Gemini 3.8 Flash, a fast LLM, runs inside the loop as a guiding agent. It watches the scene, breaks goals into steps, checks incoming frames to verify that each step succeeded, and revises its plan when the world shifts or the user intervenes.
The last change is the most interesting architecturally. It hands causality and multi-step planning to a supervising LLM, using a monitoring loop to compensate for a generator that can drift. Clark still rates the technology "quite janky," citing robotic voices and a dreamlike inconsistency in the environments. The newsletter reports no quantitative evaluation.
Metis: making reasoning explicit
Metis, a startup building AI architectures for better reasoning, has published a position paper, "The Age of Reasoning." Its goal is to do for open-ended problems what AlphaGo did for board games. Clark notes that it offers little technical detail.
Metis argues that LLMs lack two things AlphaGo had: an explicit, persistent 5 (a maintained record of what the system holds to be true) and a criterion for choosing reasoning moves that make progress toward the goal.
Its proposal is to store claims and hypotheses in a core database, then train a policy and a 6 on real-world data to steer reasoning toward reducing long-term uncertainty. An independent judgment system must approve any change to a claim's evidential status. "Model-generated content cannot enter the accepted epistemic state merely by being plausible," the authors write. Evidence attaches to the claims it supports, so the reasoning can be audited.
As an illustration of ours, not the paper's: a model might read a study and propose that a drug lowers blood pressure. That sentence would stay a candidate until it was linked to its source and passed the judge. No results are reported, so reliability gains remain a claim. Clark observes that such a system would be deeply inhuman, since people reason from a messy mix of memory, intuition and rules, which makes today's LLMs closer to us.
Space as contested infrastructure
Clark also reads "Challenges to Security in Space," a 100-plus-page report by the US Defense Intelligence Agency, and links it to AI through proposed space-based data centers. The report says China went from 22 launches in 2016 to 93 in 2025 and may field space-based 7 by the late 2020s. It also describes a Russian satellite repeatedly approaching a US national security satellite, and warns that a nuclear detonation in space would create an electromagnetic pulse affecting satellites within range. The AI connection is Clark's own speculation; the quoted passages concern military hardware.
Remember
Agent evals with paused environments can overstate deployment performance; time-sensitive tasks need testing under real timing.
Token-based clocks separate how economically a model reasons from how fast its serving infrastructure runs.
Genie 4 and Metis both add an explicit checking layer around generative models, though neither reports quantitative results here.
Experiment 3: This may follow known 7 analysis from Elhage and colleagues, which splits attention weights into which tokens attend to which and what they then contribute to the output. If so, the model is applying a published recipe.
Experiment 7: Direct interpretation of dense intermediate layers reportedly failed several times until the model could create its own 8, which is activation-based interpretability, because it involves running the network.
Game writing: The critic argues the Go opponent, a 512×6 MLP, is very weak, so beating it says more about the baseline than about weight programming.
Drafting: he treats drafting a long contract as a search problem.
Retrieval: he advises against default 6, the off-the-shelf vector representations of text used for search.
Model choice: Legora resisted a 7 for years before adding one. He finds Fable and Opus too chatty for drafting and prefers Astra's concise writing, which is a practitioner's opinion rather than a measured result.
Infrastructure: data privacy led Legora to run everything on Azure, Microsoft's cloud.
Interface:8, its spreadsheet for reviewing documents at up to a million cells, was copied by the industry down to the name, according to the notes.
Drafting is a search problem. Lauritzen treats drafting a long contract this way, and ties a good 8, the software around the model that manages tools and context, to search.
Retrieval. He advises against default 9, off-the-shelf vector representations of text used for search, and the episode touches on 10, a retrieval method that compares query and document at the token level instead of squeezing each into one vector.
Long documents. A chapter asks whether a million-token contract should go through 11, splitting it into chunks and combining the per-chunk results, or one big context window. The notes don't give his answer.
Open-weight models. Chapters cover 12, further training of a released model, for legal research, and model arbitrage, a term the notes leave undefined.
