A Rebuilt Leaderboard and a Billion Ads a Second · Morning Gist
Thursday, October 295 stories · 16 min
A Rebuilt Leaderboard and a Billion Ads a Second
A rebuilt AI benchmark gets a new leader, Meta speeds up ad retrieval 100x, and practitioners describe how agents are changing engineering work.
Picked from 19 new articles across 23 sources · arrow keys for other days
Today in 30 seconds
30s left
Meta's Andromeda now scores over a billion ads per second per GPU server, a 100x gain from redesigning the model and serving system together. story 1 ↓
Artificial Analysis rebuilt its index with hidden prompts and four separate areas. Eight closed models now beat the best open one. story 2 ↓
Gemini 4 Carbon ranks #1 on that index but trails Claude Opus 5.5 at coding. Its hallucination rate rose from 4% to 11%. story 3 ↓
Anthropic's platform leads explain running agent code away from its credentials, using one authorization system and evaluating models with real harnesses. story 4 ↓
Martin Fowler says developers are becoming supervisors of coding agents, and explains why enterprises are slow to adopt them. story 5 ↓
Anthropic has reportedly dropped coding interviews, which raises the question of how to hire when AI writes much of the code.
1/5Engineering at Meta· Oct 28
Meta's Andromeda ad retrieval now scores over one billion ads per second per GPU server
Meta rebuilt Andromeda, the first-stage system that picks candidate ads. It now uses a cheaper mixture-of-experts model and runs the whole pipeline on one GPU server, reaching over one billion ads per second, 100 times the 2024 version.
Why it matters · The post is a production example of a broader point: at this scale, moving data through memory and over the network costs more than arithmetic, and fixing it means redesigning model and system together.
Know this first
Context from the editor, not the article · 41 sec
1 · Retrieval, then ranking
Recommendation systems usually work in two stages. Retrieval cheaply narrows millions of candidates (here, ads) to a few hundred in milliseconds. Ranking then scores those survivors with a much more expensive, more accurate model. Retrieval has to be cheap per item, so its models have traditionally been simpler and less accurate.
2 · Where GPU time actually goes
A GPU runs work as kernels, which are small programs launched one after another. Between kernels, results are written to and read back from HBM, the GPU's main memory. On-chip registers and shared memory are far faster but tiny. When each item needs little arithmetic, moving data, not computing, sets the speed limit.
3 · Towers and experts
2/5Interconnects (Nathan Lambert)· Nathan Lambert· Oct 28
Artificial Analysis rebuilds its AI ranking, and open models fall behind
Artificial Analysis, the most-cited independent ranking of AI models, released version 5 of its Intelligence Index. It now has eight closed models ahead of the best open one. Nathan Lambert argues the index's design choices largely explain that result.
Why it matters · One number like this shapes which models engineers adopt and what labs train for. Knowing its biases helps you read leaderboards critically when choosing a model.
Know this first
Context from the editor, not the article · 36 sec
1 · Benchmarks and indices
A benchmark is a fixed set of tasks used to score AI models, such as coding problems or exam questions. An index combines many benchmark scores into one weighted number. That makes models easy to compare, but every choice of weights favors some skills over others.
2 · Open vs. closed models
Closed models, such as OpenAI's GPT and Anthropic's Claude, can only be used through the company's paid service. Open-weight models, such as Moonshot's Kimi or Zhipu's GLM, publish the trained parameters, so anyone can download them, run them on their own hardware, and fine-tune them on private data.
3 · Hill-climbing and benchmaxxing
3/5Latent Space· swyx .ai· Oct 29
Google's Gemini 4 Carbon tops a major leaderboard, but cyber defenders get it first
Google DeepMind released Gemini 4 Carbon, a model built for cybersecurity and long multi-step tasks. Independent tests rank it first overall. It is still behind rivals on coding, and only vetted security teams can use it for now.
Why it matters · A model aimed at finding and exploiting software vulnerabilities shows where AI security tooling is heading, and the independent scores show where Google's claims hold up.
Know this first
Context from the editor, not the article · 43 sec
1 · How models get compared
AI labs publish scores on benchmarks, which are fixed sets of tasks with checkable answers. Independent groups such as Artificial Analysis, Arena and Vals rerun models themselves. Their numbers usually count for more than a lab's own, because a lab picks which tests to report. Some tests use private questions, so models can't have seen the answers during training.
2 · Agents and harnesses
A coding agent is a model in a loop. It reads files, runs commands, checks the output and tries again. The software that runs this loop and supplies the tools is called the harness. Examples are Claude Code and Google's Antigravity CLI. The same model can score quite differently in different harnesses.
3 · Tokens and pricing
4/5Latent Space· swyx .ai· Oct 27
How Anthropic's platform team launches Claude models and runs agents safely
On the Latent Space podcast, Anthropic's heads of platform engineering and product describe how they launch models like Opus 5.5, make reasoning harder to copy, and run agents with the controlling program and the code-running sandbox kept apart.
Why it matters · If you build on large-model APIs or design agent runtimes, these are concrete patterns: keep agent state apart from execution, keep secrets out of sandboxes, use one generic authorization system, and evaluate with real harnesses.
Know this first
Context from the editor, not the article · 37 sec
1 · APIs, agents and harnesses
Developers reach models like Claude through an API: send text, get text back. An agent is a model calling tools in a loop, such as running code or reading files. The harness is the program around the model that runs that loop, stores the conversation and decides what the model sees next.
2 · Distillation and reasoning traces
Distillation trains a cheaper "student" model on a stronger "teacher" model's outputs. Reasoning models write step-by-step working, called thinking or a chain of thought, before they answer. That working is very valuable training data, so whoever can collect it at scale can copy much of the model's skill.
3 · Sandboxes
5/5Martin Fowler· Martin Fowler· Oct 28
Martin Fowler on AI coding: developers are becoming supervisors of agents
Martin Fowler gathers views from a conference and fellow writers. Coding agents are moving developers from writing code to supervising it. Most people and companies aren't there yet, held back by unreliable tools, messy data and token costs.
Why it matters · It gives a senior engineer a working model of how their job is changing, from writing code to supervising agents, and of the practical limits.
Know this first
Context from the editor, not the article · 35 sec
1 · LLMs and coding agents
A large language model (LLM) predicts text and can write code. A coding agent wraps an LLM in a loop: it edits files, runs commands and tests, reads the results and tries again. The wrapper code is called a harness. 'Agentic engineering' means building software mainly by directing agents like this.
2 · Who is Martin Fowler
Fowler is chief scientist at Thoughtworks, a software consultancy, and a long-time writer on software design and Extreme Programming. His 'Fragments' posts collect short commentary on other people's writing, so this piece is a roundup rather than one argument.
Retrieval models often use two "towers": one network turns a user into a vector, another turns an ad into a vector, and a cheap step compares them. A mixture-of-experts (MoE) model has many specialised sub-networks plus a router that switches on only the few relevant ones for each input. This saves computation.
The retrieval model: user-expert towers, a context-aware ad encoder, and a router choosing the top-K experts per ad.1 / 3
Meta has rebuilt Andromeda, its system for ads 1. Retrieval is the first stage of choosing which ads a person sees. It cuts a huge pool of eligible ads down to a short list in milliseconds, and a slower, more accurate ranking stage then scores that list. Meta says the new Andromeda scores every eligible ad and processes over one billion ads per second on a single GPU server. That is 100x the 2024 version, which ran on NVIDIA's 2 chip. Meta credits Andromeda with a 2.4% rise in conversions on Facebook since 2025.
The problem
Generative-AI tools let advertisers create hundreds of millions of new ads, so the candidate pool grew sharply. The 2024 system handled tens of millions of ads per second per GPU; Meta needed billions. At that rate, the limit stopped being arithmetic. It became data movement. GPU work runs as a chain of small programs called 3, and each one wrote intermediate results to 4, the GPU's main memory, generating terabytes per second of traffic. On top of that, targeting checks and final selection ran on separate CPU machines, and shipping candidate ads between them and the GPUs saturated the network.
The model change
Meta adopted a 5 design, borrowed from large language models: many specialised sub-networks plus a router that activates only the relevant few.
Several "6" now act as experts, each modelling a different kind of signal, such as past ad engagement or LLM-derived features.
Ads are encoded together with real-time serving context.
For each ad, an attention-based router picks the most relevant user experts.
Meta says this approaches ranking-stage quality with about 300x fewer floating-point operations than the 2024 model.
The system change
That compute saving made two changes possible:
One fused kernel now does the whole per-ad job: fetching stored feature vectors, running the model and scoring. Intermediate values stay in fast on-chip memory and never touch HBM.
Everything runs on one server. Index matching (checking advertisers' targeting rules), scoring and 7 (keeping the best K ads) share one GPU server. The client sends a small request and gets back only the winners. Network traffic falls about 100x.
Tuning with NVIDIA
A joint optimisation effort sped each component up 2–5x:
The scoring kernel went from 2.4 to over 10 billion ads/sec, using about 85% of peak memory bandwidth. Meta says that is as fast as a plain dot-product scorer.
8-bit integer arithmetic added up to 2x more, with accuracy Meta calls on par.
Index matching passed 100 billion ads/sec.
Meta argues the gains came from designing model, system and hardware together. It does not publish latency figures or offline quality metrics. Next, it is porting the work to NVIDIA's 8 platform.
Meta has published the second generation of Andromeda, its system for personalised ads 1. Retrieval is the first stage of ad selection: it picks a short list of promising ads from everything a user is eligible to see. Meta says the new version scores every eligible ad and delivers over one billion ads per second on a single GPU server. That is 100x the 2024 Andromeda, which ran on NVIDIA's 2, a chip that pairs a CPU and a GPU in one package. Meta also says ranking quality is close to that of its much more expensive later-stage models. Since 2025, Andromeda has lifted conversions on Facebook by 2.4%. A conversion is an advertiser's desired outcome, such as a purchase.
Why the old design ran out of road
The 2024 system was already GPU-native. It used hierarchical indexing, custom operators and a purpose-built model, and Meta says that model had four orders of magnitude more capacity than its predecessor. Since then, generative-AI tools have helped advertisers create hundreds of millions of new ads, so the candidate pool grew sharply. The 2024 system evaluated tens of millions of ads per second per GPU. The new target was multiple billions per GPU.
At that rate, three problems appeared together:
The model: retrieval mixture-of-experts
The new model borrows 7 from large language models. An MoE has many specialised sub-networks ("experts") and a router that activates only a few per input. Meta changed three parts of the model:
User side. One "8" (a network that summarises a user as a vector) became an ensemble of specialised user towers, each acting as an expert. They draw on different signals: the user's sequence of ad engagements, representations of user cohorts, features derived from LLMs, and engagement across Meta's different apps.
Ad side. The ad tower encodes each ad together with real-time serving context, so the ad's representation already reflects the situation it would be shown in.
Interaction layer. This is where user and ad meet. It went from dense neural networks to an attention-based MoE gating network. For each candidate ad, the router picks the top-K most relevant user experts and combines only those.
Meta reports the model needs about 300x fewer 9 (floating-point operations, a measure of compute) at inference time than the 2024 model, while approaching ranking-level quality.
The system: one kernel, one box
The 300x cut in compute created room for two system changes.
A fused kernel. In 2024, only the model's own operations were merged into a single kernel. Now one kernel does all the work for each user-ad pair: it gathers embeddings (stored vectors for features), runs the MoE interaction, and produces the score. Intermediate values stay in GPU registers and per-block shared memory, which are fast on-chip storage, and are never written to HBM.
An illustrative analogy: picture a factory where every workstation boxes up half-finished parts and ships them to a warehouse, and the next station fetches them back. Fusion puts all the stations on one bench, so parts pass straight from hand to hand.
Co-location. Index matching, scoring and top-K selection now all run on the same GPU server. The client sends a small payload (user features plus a targeting query) and receives only the final top-K ads. Meta says this cuts network traffic by about 100x and makes the system limited by GPU compute again rather than by the network. Moving index matching to the GPU was hard. Targeting rules span tens of feature categories, and the index must update in real time as ads are created, changed or expire. Meta redesigned the index structure for parallel execution while keeping exact targeting behaviour. The post does not describe that new structure.
Pushing toward the hardware ceiling with NVIDIA
In early 2026, Meta and NVIDIA set out to see how close the pipeline was to the GH200's limits. NVIDIA supplied optimised kernels, and Meta integrated and benchmarked them. Each component got 2–5x faster:
Gating (interaction) kernel: 5.3x faster, from 2.4 billion to over 10 billion ads/sec. That uses about 85% of the GPU's peak HBM bandwidth. Meta says this is as fast as a plain dot-product scorer on the same hardware. In its view, the extra model complexity is "essentially free" on 10, the GPU's dedicated matrix-math units. Switching from 16-bit floats to 8-bit integers () gave up to another 2x, with accuracy "on par".
The 10-billion figure appears earlier in the post as a property of the production model. It seems to be the optimised kernel measured in isolation. The production end-to-end figure is over one billion ads/sec per server.
Trade-offs and open questions
Meta's central argument is that the biggest gains come from designing model, system and hardware as one stack. The evidence fits that view: the FLOP reduction is what made full fusion possible. The cost is tight coupling. The model shape, kernel and data layout now depend on one another and on a specific GPU generation.
The post leaves several things out:
offline quality metrics for "near ranking-level";
latency numbers;
how K or the number of experts is chosen;
how the GPU index handles consistency during real-time updates;
how much of the 2.4% conversion lift comes from this generation specifically.
Next, Meta is porting the work to 12, NVIDIA's next CPU-GPU platform, and studying how the ratio of CPU to GPU compute affects workloads like Andromeda.
Remember
At billions of items per second, moving data through memory and over the network limited speed more than arithmetic did.
Cutting model compute 300x is what made fusing the whole per-ad pipeline into one GPU kernel possible.
Moving targeting checks onto the GPU server cut network traffic about 100x and made GPU compute the limit again.
Hill-climbing means making small changes over and over to push a score up. Benchmaxxing is slang for training a model to do well on particular benchmarks, often with data that looks like the test, so its score rises faster than its real-world ability.
Model rankings on Artificial Analysis Intelligence Index v5, which weights four categories equally.1 / 3
Artificial Analysis, an independent company that tests AI models, has released version 5 of its 1. The index is a single score that ranks the most capable current language models. Nathan Lambert, an AI researcher who writes the Interconnects newsletter, calls it the most prominent independent measure of model intelligence. He uses the release to explain why designing such a score is so fraught.
What v5 measures
The new index gives equal 25% weight to four areas: knowledge work, coding, science and general knowledge. Each area is made of specific tests. Examples include 2, which gives a model tasks to complete in a command-line environment, and 3, short for Humanity's Last Exam, a set of very hard expert questions. The test prompts are now private, meaning kept secret. The stated reason is that labs releasing 4 models (models whose parameters anyone can download) appear to be "5," or tuning their models to known tests.
The result
Eight closed models now rank above the best upcoming open model, Kimi K3.1 Pro. Open models trail most on the science tests. Lambert says the earlier version flattered open models too much: the best OpenAI and Anthropic models really are more flexible and reliable. But he argues that v5 also makes a choice. It measures the far frontier of open-ended, agent-style work, where large closed models do best. It may miss the routine workflows where open models are becoming economically important.
The dilemma
An index has to reduce many uneven abilities to one number, and every way of combining scores builds in a view of what matters. Lambert says open models trail partly because recent gains come from 6, where models practice in simulated task "environments" built to resemble the tests. Doing that across many domains takes infrastructure that not every open lab has. He expects the next open releases, such as GLM 6 or Kimi 3.5, to close much of the gap within two to three months.
He calls hidden prompts "defensible" but costly. Outsiders can no longer find bugs in the benchmark, which happens often with public ones.
Who has a stake
Lambert says OpenAI and Anthropic largely stopped caring about these indices long ago. Their focus is internal research and big customers. For smaller labs, looking strong on the index is "existential." Evaluators have conflicts too: Artificial Analysis is paid by labs to verify their results, and 7, a leaderboard based on human votes, is paid by labs for pre-release testing. Lambert trusts non-profits, academics and government bodies to be fair, and frontier startups to be accurate. His only remedy is more indices with different biases.
Artificial Analysis, an independent company that tests AI models, has released version 5 of its 1. The index is a single score that ranks the newest frontier language models, meaning the most capable ones available. Nathan Lambert, an AI researcher who writes the Interconnects newsletter, calls it the most prominent independent intelligence metric, with Epoch AI's Capabilities Index (ECI), a similar aggregate score from a research group, "a distant second." He uses the release to argue that building such indices is harder, and more political, than it looks. In his words: "Hill-climbing is easy, knowing what to hill-climb is hard."
What changed in v5
Artificial Analysis says v5 reflects "real-world usage today." It weights four categories at 25% each:
Science: Terminal-Bench-Science (15%), the hard subset of CritPt v2 (10%)
General: AA-Omniscience v1.1 (15%, split into 10% for accuracy and 5% for not hallucinating, i.e. not inventing facts) and HLE-Diamond (10%)
Terminal-Bench gives a model tasks to complete in a command-line environment. 3 is short for Humanity's Last Exam, a set of very hard expert questions. The article doesn't describe the other components in detail.
The second change is a move to private benchmarks, where the test prompts are kept secret. The stated reason is evidence that labs releasing 4 models are "5," meaning they tune models specifically to known tests.
The result
In the new ranking, GPT 6.1 Terra places above Claude Fable 5.1. Lambert reads this as a sign of "strain and confusion," the kind he says every benchmark shows. The chart that drew attention compares open and closed models. Eight closed models beat Kimi K3.1 Pro, the strongest model that is to be released with open weights. Open models trail most on the science evaluations, Terminal-Bench-Science and CritPt v2.
The benchmarker's dilemma
Lambert's main point is that every index must boil down to one number. That makes it easy to quote and compare, but any way of combining scores builds in a worldview. Real-world use is "jagged," in his word: models are strong at some tasks and weak at others, and the mix keeps changing. Some benchmarks measure the absolute frontier: open-ended work through a 6 (the software wrapper that lets a model use tools and run multi-step tasks), which often favors large models. Others measure narrower skills that repetitive agent workflows need. Lambert believes Artificial Analysis is tilting toward the first type, a "general agentic collaborator."
That tilt widens the gap between open and closed models. Lambert calls it "an okay decision." He warns, though, that as businesses come to depend on everyday open-model workflows, the index could miss "where the core economic engine of the industry is." He also concedes that the previous version flattered open models too much. The best OpenAI and Anthropic models really are ahead on flexibility and reliability.
Why open models trail, in Lambert's account
The best open models benchmax "a little" and the worst "a lot."
Open models cover a narrower range of skills, as their weak science scores show. Lambert thinks trying to cover everything could be a strategic mistake for open labs.
Many recent training gains come from 7 (RL). In RL, a model practices tasks in simulated "environments" and is rewarded for success, and gains come from environments that closely resemble the evaluations. Keeping pace across many domains needs a sophisticated RL stack that not every open lab has. Some labs buy training data, which arrives with delays.
He argues the open-model economy was never about winning at the very frontier. Its goals are to bring the cost of popular tasks down to roughly the price of the hardware, and to let companies own their models, specialize them on internal data, and control their own agents. Benchmarks capture these goals only partly. He predicts that the next round of open releases, such as GLM 6 or Kimi 3.5 in two to three months, will make "substantial steps" on the index and surprise people again, as he says has happened for years.
Private benchmarks: defensible, with a cost
Lambert finds the move to hidden prompts "directionally defensible." It is easier to build training data for a test when you know what it looks like. He notes it isn't impossible with hidden data, though. The cost is trust. With public benchmarks, the community finds bugs and pushes fixes, which "happens a lot." With private ones, you rely entirely on the builder. With this much money at stake, he would rather have transparency.
Incentives on both sides
Lambert says Anthropic and OpenAI stopped caring much about these indices months or even years ago. They focus mainly on internal uses, such as automating their own AI research, and on customers spending billions. For small and mid-sized labs, though, looking strong on these indices is "existential," because their brand rests on catching up to the frontier.
The evaluators have incentives too. Artificial Analysis earns money from labs to verify their results. 8, a leaderboard built on human votes between model answers, earns money from pre-release testing for labs. Lambert trusts non-profits, academics and government bodies most to be fair. One example is 9, the US government's Center for AI Standards and Innovation. But he trusts startups like Artificial Analysis and Arena most to be accurate at the frontier. He compares the situation to a long-running problem in research: how hard should you tune your comparison baseline? More tuning makes results more trustworthy, but it usually makes your own method look less impressive.
What's left open
His only remedy is more indices with different biases, such as coding-only rankings like CursorBench, and growth from players like Vals AI and Intelligence.ai. He admits that attention is still heavily skewed toward Artificial Analysis. He doesn't propose how to judge which evaluator to trust. He calls that question, the "dilemma of a benchmarker of benchmarkers," more sociological than scientific.
Remember
A single AI score bakes in a worldview: v5 rewards open-ended frontier work, which widens the gap for open models.
Hidden test prompts make it harder for labs to game a benchmark, but outsiders can no longer check it for errors.
Leading closed labs largely ignore these indices; for smaller labs, looking strong on them is existential.
Models read and write tokens, which are word fragments of roughly four characters. Providers charge per million input tokens and per million output tokens. Output costs more. A model that 'thinks' at length writes more output tokens, so the cost per finished task can differ a lot from what the list price suggests.
Image: Latent Space
Google DeepMind has released Gemini 4 Carbon, two weeks after its previous model, Gemini 4 Argon. Google pitches Carbon as the strongest model for cybersecurity and for long, multi-step tasks. For now, only vetted security teams can use it, through Google's 1. Google promises wider developer access "as soon as possible".
The cyber pitch
Security work has three stages: finding a vulnerability (a flaw an attacker could use), exploiting it, and patching it. Google reports a score for each:
Finding vulnerabilities: 88.2% on SEC-bench Pro
Writing working exploits: 54.8% on ExploitGym
Patching: 71.0% on CWE-bench (2 is the standard catalogue of software weakness types)
A model that is good at exploitation helps attackers too. Giving defenders first access lets them patch before everyone else gets the model.
What independent testers found
3 is a third party that reruns models on its own tests. It ranks Carbon #1 on its Intelligence Index v5 with a score of 59, just ahead of Claude Opus 5.5 (58) and GPT-6 Astra (57). The rebuilt index runs 60% on private tests, which makes it harder to game but impossible for outsiders to reproduce.
Carbon does not lead on coding. It scores 59 on the Coding Agent Index, against 62 for Opus 5.5 running in 4. On FrontierSWE, a set of real software-engineering tasks, it gets 61.2%, behind GPT-6 Astra (65.5) and Opus 5.5 (62.3). Carbon's coding runs used Google's own 5, 6. The harness is the software that lets a model edit files and run commands, so these comparisons change two things at once.
Cost and verbosity: a full index task costs $7.76 with Carbon, versus $10.04 for Opus and $6.25 for Astra. Carbon uses 29% fewer output 7 (the word fragments models are billed by) than Opus. List price is $4/$20 per million input/output tokens, halved during a launch discount that has no end date.
Hallucination: this means confidently stating false things. Carbon's measured rate rose from Argon's 4% to 11%. That is still the second-lowest rate measured, far below the roughly 50% for Opus 5.5 and Astra. Some users worry that even this rate is too high for security work.
Google's own safety claims
Google says successful cheating on the Terminal-Bench 4.0 benchmark fell to 0.0%. It also says "overreach" (doing more than asked) fell 26% and unsupported claims roughly halved. No independent group has checked these numbers.
What to watch
Most outsiders can't test Carbon yet, and its coding results come from a single harness. The commentator scaling01 still puts Google about two months behind the next OpenAI and Anthropic releases.
Google DeepMind has announced Gemini 4 Carbon, two weeks after its previous model, Gemini 4 Argon. Google pitches Carbon as the best available model for cybersecurity and for long-horizon tasks, meaning jobs that take many steps over a long session. The general public can't use it yet. It goes first to vetted security teams through the 1, Google's early-access scheme for "trusted cyber defenders", and Google promises wider developer access "as soon as possible". These numbers come from the Latent Space AINews roundup of 28 October. That roundup collects the lab's own claims alongside independent measurements, and the two don't always agree.
Why a cyber-first release
Security work has three stages: finding a vulnerability (a flaw an attacker could use), exploiting it (writing code that actually abuses the flaw), and patching it. A model that is good at the first two can help attackers as much as defenders. Google's own reported scores cover all three stages:
SEC-bench Pro (finding vulnerabilities): 88.2%
ExploitGym (writing working exploits): 54.8%
CWE-bench (patching; 2 is the standard catalogue of software weakness types): 71.0%
Giving defenders a head start is a deliberate choice. One Reddit commenter defended the delay along those lines: if a model this capable at exploitation is released to everyone at once, maintainers have "zero time to react and patch". Other users are less patient. They say even the earlier Argon preview was so rate-limited that they couldn't get three prompts through.
Specs and price
Carbon has a 1-million-token context window. That is how much text the model can consider at once, roughly several thick novels. It can also produce up to 1 million 3 of output. It accepts text, images, video and speech, and replies in text. 4 lists standard pricing at $4 per million input tokens and $20 per million output tokens. A 50% launch discount, with no end date, brings that to $2 and $10. Cached input costs $0.20 per million tokens. Caching applies when the same prompt prefix is sent repeatedly and the provider reuses its earlier work.
What independent testers found
Overall. Carbon is #1 on the Artificial Analysis Intelligence Index v5 with a score of 59. The index averages nine evaluations, and Carbon leads four of them. The top of the board is tight: before Carbon, Claude Opus 5.5 scored 58 and OpenAI's GPT-6 Astra 57. v5 is a rebuilt index, and 60% of it now runs on private tests. That makes it harder to game, but outsiders also can't reproduce it. On LocalLlama, a Reddit forum for people who run models on their own hardware, users complained that the rebuild widened the measured gap between open-weight models (whose parameters anyone can download) and closed ones from 3 to 12 points.
Coding, where it does not lead. On Artificial Analysis's Coding Agent Index, Carbon scores 59 running in Google's 5, against 62 for 6 with Opus 5.5. On FrontierCode v1.1 it scores 51% to Opus's 56%. On FrontierSWE, which uses real software-engineering tasks, it gets 61.2%, behind GPT-6 Astra (65.5), Opus 5.5 (62.3) and Anthropic's mid-tier Sonnet 5.5 (61.9). It does set a narrow record of 9.6% on the very hard SWE-sweep benchmark. One caveat: the coding-agent results were measured only in Antigravity CLI, so nobody yet knows how Carbon performs in other 7.
Cost per task. List prices can mislead, so Artificial Analysis measures what a full run of its index tasks actually costs. Carbon averages $7.76 per task, compared with $10.04 for Opus 5.5 and $6.25 for GPT-6 Astra. It uses about 123,000 output tokens per task, 29% fewer than Opus. Vals found the opposite pattern against Google's own previous model: Carbon writes 64% more output tokens than Argon, mostly on finance, tax and legal tasks.
Hallucination. A hallucination is a confident answer that is false. Here the picture is mixed. Carbon leads AA-Omniscience, a knowledge test that penalises wrong answers, with a score of 54. But its measured hallucination rate rose from Argon's 4% to 11%. That is still the second-lowest rate measured, and far below the roughly 50% recorded for both Opus 5.5 and GPT-6 Astra. Google separately claims "unsupported claims" fell about 2x compared with Argon. The roundup doesn't reconcile the two numbers, which probably measure different things. Some users worry about hallucination specifically in security work, where an invented vulnerability wastes an analyst's time.
Elsewhere:
Arena, which ranks models by blind human votes on pairs of answers: #1 in Text Arena (1528), #2 in Agent Arena at a median $1.80 per task, #4 in WebDev (1768).
Vals: #2 overall, #1 on its Finance Index and on a legal-research benchmark.
AA-Briefcase, a knowledge-work test scored on 8 (a chess-style rating built from head-to-head comparisons): 1609. That is 126 points above Argon but behind the leader at 1678.
Andon Labs: #1 on Vending-Bench 2, a long-running business simulation. Carbon also matches the human baseline for 3D spatial understanding on Blueprint-Bench 2.
Google's safety claims
Google reports that on Terminal-Bench 4.0, successful cheating fell to 0.0% and attempted cheating to 0.4%. Cheating here means the model gaming the test, for example by tampering with checks instead of solving the task. Google also reports that "overreach" (doing more than asked) fell 26% and "under-service" (doing less than asked) fell 29%. These are Google's numbers, and no outside group has verified them.
What's still open
Coding harnesses. The coding comparison mixes two variables: model and harness. Carbon ran in Antigravity CLI and Opus in Claude Code, so the 59-vs-62 gap isn't a pure model comparison.
The Google-vs-rivals gap. The commentator scaling01 argues that Google is still about two months behind the next OpenAI and Anthropic releases. Sundar Pichai said on Alphabet's earnings call that he expects "continued acceleration because of RSI" (recursive self-improvement, meaning AI that speeds up AI development). A Gemini engineer described overnight agent loops that inspect training data, which make launches feel like "periodic checkpoints".
Unverified rumours. Talk of a stronger, unreleased Gemini model is unconfirmed.
Access. Until access widens beyond Fairwind members, most independent results come from a small set of evaluators.
Remember
Carbon leads Artificial Analysis's overall index by one point, but trails Opus 5.5 and GPT-6 Astra on coding.
Its measured hallucination rate nearly tripled from Argon, though it is still far below Opus 5.5 and Astra.
Coding comparisons mix model and harness, and Carbon was tested only in Google's own harness.
A sandbox is an isolated computer environment, usually a container or lightweight virtual machine, where untrusted code can run without touching the host. Agents that write and run code need one, because nobody can guarantee the generated code is safe.
Image: Latent Space
The podcast Latent Space interviewed Katelyn Lesse and Angela Jiang, who lead engineering and product for the 1, Anthropic's developer APIs and the infrastructure behind them. The episode came out a day after the Opus 5.5 model launch. Most of what they describe is plumbing that model users never see.
A model launch is now mostly safety and API work
Opus 5.5 answers complaints that its predecessor's writing was convoluted. Anthropic says it also costs less to run, though the saving depends on how much a workload reuses cached prompt text. Lesse says most of the launch effort went elsewhere: deciding whether each model requires Anthropic to keep request data for later abuse review, running verification programs that relax safeguards for approved security and biology researchers, and resisting 2.
Distillation means training a cheaper model on a stronger model's outputs, especially its step-by-step reasoning. Anthropic's defense is to require preserved 3: clients must send the model's earlier reasoning back unchanged on later turns. Jiang says the motive is safety, because a copied model would run without Anthropic's 4. The change broke harnesses that compress long conversations (called 5). Anthropic shipped its own compaction feature that works with the new rule, warned open-source maintainers ahead of time, and gave existing customers a grace period. It reports distillation is now "significantly harder."
That grace period ran on 6, an internal authorization record saying "this account may use this model or program." Anthropic built it more generically than the first use case needed. It now also handles restricted models and a single switch that can pull a model back from everyone.
The brain and the hands
The hosts' favorite idea concerns 7, an API that gives developers a ready-made agent harness instead of raw model output. The agent is split in two:
The brain: the session log and the harness. It persists for the whole session.
The hands: a 8 that runs generated code. It is created only when needed.
An earlier design put the whole agent inside a sandbox. Every request then had to wait for startup, and a crashed sandbox killed the agent. Now a failed sandbox is just a failed step that can be retried. Credentials never enter the sandbox: a proxy outside it attaches them, so generated code has no key to steal.
Also worth knowing
Before launch, customers with strong automated test suites (9) try candidate versions. One early Opus 5.5 build was sent back for being too lazy. A year ago the core API was a single Python program in one cloud region. CPU is now scarce, and the episode lists a Rust rewrite, but the transcript gives no numbers for it or for the brain/hands split.
The podcast Latent Space interviewed Katelyn Lesse and Angela Jiang, who lead engineering and product for the 1. That is Anthropic's developer-facing layer: the APIs and the infrastructure underneath them. The episode came out a day after Opus 5.5, which the hosts call the most widely praised model on the market. It went out alongside Anthropic's IPO filing. The interesting parts for engineers are about what it takes to ship a model as an API, and how Anthropic builds the runtime that its agents run on.
A model launch is mostly platform work
Lesse and Jiang say the most visible changes in Opus 5.5 answer customer complaints. People found Opus 5's writing convoluted, so the team cleaned it up. Customers also wanted more "intelligence per dollar." Anthropic says the new model costs less to run than its predecessor, but the saving depends on the workload and how much it reuses cached context. Cached context means prompt text the API has already processed and can reuse cheaply. The team changed caching prices on purpose to push customers toward integration patterns that save money for everyone.
Lesse says most launch work now goes into things that barely existed a year ago:
Per-model data-retention rules. An earlier model, Fable, requires Anthropic to keep request data so it can review traffic afterwards; Opus 5.5 does not. Retention matters because abuse spread across a network of accounts can't be spotted from tokens as they stream past. It needs review after the fact, by automated systems and by humans.
Verification programs. Approved users in cybersecurity and life sciences get relaxed safeguards for "dual-use" work, meaning knowledge that can serve attack or defense.
Distillation defenses, covered next.
Making reasoning harder to steal
Distillation means training a cheaper model on a stronger model's outputs. The team says extracting a model's chain of thought, its step-by-step working, has long been one of the most effective ways to distill it. A paper on stealing reasoning traces from proprietary models was disclosed to Anthropic privately before publication, and the hole has since been closed. Jiang says the motive is safety, not revenue: "if you distill from us, you actually pay us." Anthropic runs 2 on what its models produce. These are filters that flag dangerous output, and a distilled copy would run without them.
The fix is to require preserved 3. A thinking block is the segment of a response that holds the model's reasoning. Under the new rule, a client sending a multi-turn conversation must pass earlier thinking blocks back exactly as they were received, and Anthropic now applies checks across multiple accounts too. The guests mention "prefix-locked thinking" and limits on replaying thinking tokens across different models. They don't explain the details.
The trade-off is breaking customers. Lesse sorts the damage into two levels. The first is an annoying integration change that still lets you do everything you could before. The second is losing a capability outright, which they work hard to avoid. One case: harnesses that compress long conversations to fit the context window, called 4, broke when they couldn't rewrite old thinking blocks. Anthropic shipped a compaction feature inside the API that works with preserved thinking. It also contacted open-source harness maintainers ahead of time, and gave existing customers a grace period. By Anthropic's own measurements, 5 has become "significantly harder."
Trust grants: one authorization system for everything
The grace period ran on 6. A trust grant is Anthropic's internal record saying "this account may use this model, program or exception." A service on the request hot path, meaning it runs on every request, checks for it. Lesse calls it ordinary authorization, much like IAM (identity and access management) in AWS. The difference is that it was built to fit Anthropic's own system. Jiang says they deliberately built it more generically than the first use case needed. It now handles access to restricted models, the verification programs, grandfathered customers, and a "single button" that can pull a model back from everyone. She describes product planning here as "deeply probabilistic": build a portfolio of hedges against events you can't predict.
Early access is about seeing other harnesses
Before launch, Anthropic tries out candidate versions internally (it calls this "ant-fooding"). Then it gives early access to API customers such as Cursor and Cognition, picked because they have strong 7, meaning automated test suites for model behavior. Outside harnesses show behavior that Anthropic's own products never exercise. The most useful feedback is negative. One early Opus 5.5 version was too lazy, so it went "back in the oven." Lesse notes that customers disagree about where laziness ends and too much initiative begins. Anthropic's tools for this are parameters such as effort level, whose default in Claude Code is now medium and varies by customer segment. The guests admit these knobs have become a "mishmash."
The brain and the hands
The idea the hosts liked best is about 8. This is a set of APIs where you define an agent, start a session, and get a working harness without building one. By contrast, the 9 returns raw model tokens. Managed Agents splits the agent into two parts:
The brain: the session log and the harness. It lasts for the whole session.
The hands: a 10 that runs the code. It is created only when needed and can be thrown away.
An earlier design ran the whole coding agent inside a sandbox behind an API. Every request had to wait for the environment and the agent to start before the first response. And if the sandbox died, the running agent died with it. Once the two are split, a crashed sandbox is just a failed tool call that the harness can retry somewhere else.
An analogy (not from the article): think of an architect and a hired contractor. If the contractor quits, the architect still has the plans and hires another one.
The split also helps security. Service credentials, such as API keys for the tools an agent calls, are never placed in the sandbox. They are attached by a proxy outside it, and network controls limit where sandbox traffic can go. Generated code can't leak a key that isn't there. Lesse contrasts this with the much-reported cases of models trying to escape during training. There, a model is shut in a box with an impossible task and an incentive to break out. In production, "the harness is over here, the hands are over there," with online safeguards running on top. The design questions the hosts suggest for any agent runtime: which parts must survive a failure, which parts run untrusted work, and which secrets actually have to be present where the code runs.
Memory, scale and what's left open
"Dreaming" is a memory feature built from the same building blocks: one agent reviews another agent's past work and writes memories back. The team says memory work has slowed in favor of safety. They keep the base harness deliberately simple so developers can add their own verifiers or run agents in parallel.
On scale: when Lesse joined about a year ago, the Messages API was a single Python program running in one Google Cloud region. CPU is now a tight constraint, and the episode's timestamps list a rewrite of the Messages API in Rust. The transcript excerpt ends before that segment, so there are no performance figures. The episode also gives no latency or reliability numbers for the brain/hands split, and no details on how the anti-distillation checks work.
Remember
Run agent state and code execution separately: sandboxes can then be thrown away and never hold credentials.
Anthropic fights distillation by requiring clients to send the model's earlier reasoning back unchanged, accepting some pain for customers.
One generic authorization system, built more broadly than first needed, now handles model access, grace periods and pulling models back.
Frontier models are the most capable models. Labs offer them only as paid APIs, priced per token (a token is roughly a word fragment). Open-weight models publish their parameters. Anyone can run or fine-tune them on hardware they control, which trades capability for cost and control.
Thoughtworks' nine-cell grid: where each AI workload runs, by differentiation and security/sovereignty scores.
Martin Fowler, chief scientist at the consultancy Thoughtworks, has published a roundup of short notes. Together they sketch how AI is changing software work, and why most people and companies haven't caught up.
A gap in experience
Fowler starts with AI researcher Andrej Karpathy. Karpathy says the public sees "a derpy chatbot", most professionals see a modest gain, and a small group feels "the curve going vertical." Fowler sees the gap himself. He uses free chatbots on purpose and finds them useful but unreliable. He had just come from 1, an O'Reilly "unconference" where attendees set the sessions, and he describes it as a bubble of early adopters. Coaches there run one- to two-week courses in 2, which means building software mainly by directing AI coding agents. Agents are programs that edit code, run tests and fix their own errors in a loop. The coaches said the hardest part was emotional, because the tools challenge what developers think their job is.
Why enterprises lag
One attendee said corporations have "a Swiss-army knife, but are only using the toothpick." Fowler thinks they have reasons. Outages cost more for them, workflows built over years change slowly, and security is a real worry. He mentions the "3": an agent that can read private data, sees untrusted input, and can send data out. Above all, many firms lack clean data for agents to work with.
From doing to supervising
Annie Vella borrows a 1960s distinction from automation research. In direct control, a human closes the feedback loop: change code, see results, fix. In 4, the machine closes the loop and the human monitors it. Coding agents push developers toward supervision. She maps this onto the five-level model of Endsley and Kiris. It runs from no assistance, through autocomplete suggestions, then agents you must approve, then agents you can veto, to full automation. Her caveat is that the classic model assumes the machine reports honestly what it did. AI agents don't reliably do that, so they need more oversight. She proposes treating the 5 (software development lifecycle) as "a control system we build and supervise." Fowler agrees.
Adam Tornhill adds that only people who confuse programming with writing code need to fear for their identity. Tim Bray's objections are ethical and economic instead: job cuts, climate cost, and who profits.
Paying for tokens
Model APIs charge per token, a small chunk of text. Thoughtworks' Andy Nolan describes how the firm will manage that cost. It rents frontier models, the most capable ones, for hard reasoning. It runs 6, whose parameters are public, on its own chosen capacity for sensitive or high-volume work. Each workload is scored on domain differentiation and on security & sovereignty, then placed on a nine-cell grid. The forecast cuts 2028 costs from $16–40M to a figure the post prints as "$6.8-1M", which looks garbled.
Martin Fowler, chief scientist at the consultancy Thoughtworks, has published his latest "Fragments" post, a roundup of short commentary. Read together, the items tell one story. A small group of developers already work mainly by directing AI coding agents, programs that wrap a large language model (LLM) in a loop of editing code, running tests and fixing errors. Most people and most companies are far behind that group. Several writers try to describe what the job is turning into.
The gap in perception
Fowler opens with Andrej Karpathy, a well-known AI researcher. Karpathy says the public mostly sees "a derpy chatbot", most professionals see a modest gain, and "a small sliver" feel "the vertigo of the curve going vertical". Fowler tests this himself. He deliberately uses free-tier chatbots and finds them often useful but unreliable enough to undercut that usefulness. He admits these aren't frontier tools or a strong 1 (the scaffolding code around a model). His point is that most people don't have those either.
He had just come from 2, an invitation-only "unconference" run by the publisher O'Reilly, where attendees set the sessions themselves. He calls it a bubble of people at the bleeding edge. He reports under the Chatham House rule, which lets him repeat what was said without naming who said it.
Teaching the new style, and its emotional cost
Some attendees run one- to two-week immersion courses in 3, meaning building software mainly by directing agents. Fowler compares them to the Extreme Programming (XP) immersion classes he helped run around 2000. XP is a practice-heavy agile method built on pair programming and test-first development. The coaches said the hardest part was emotional, not technical. The tools challenge what developers think their job is. One coach said his main work was walking people to a coffee shop so they could absorb it. Fowler remembers an XP student who walked out on day one because the version-control system, CVS, didn't lock files.
Why enterprises lag
One attendee put it this way: corporations "have acquired a Swiss-army knife, but are only using the toothpick." Fowler thinks they have good reasons. When a startup's website goes down, that's one thing. When a factory stops, that's another. Workflows that involve many people and many years of habit change slowly. Security worries are real. He cites the "4", the risk that comes when an agent can read private data, sees untrusted content, and can send data outward, all at once. Above all, useful enterprise AI needs clean, well-organised data, and many large firms lack it. He was struck that Foocamp barely discussed this. He reads that as a sign of how invisible the problem is to people on the leading edge.
Bubble or 1997?
Torsten Slok, chief economist at the investment firm Apollo, argues that outside seven big tech companies there is no data yet showing productivity gains that justify the spending. The developer Laurie Voss replies that valuations are a bubble that will pop, but that judging AI on early data is like judging the internet in 1997. Fowler agrees the boom resembles the dotcom era, and thinks this one is bigger. He is less sure whether we are in 1997 or 1999, just before the crash. Voss offers a heuristic: where you can build something that checks whether an answer is correct, AI can reach that answer much faster than you can. Fowler adds that "investment" also covers where you spend your own time.
Identity and ethics
Adam Tornhill, founder of the code-analysis company CodeScene, enjoys agentic coding because it lets him take on bigger problems. Infrastructure updates and unfamiliar build systems stop blocking him. He expects writing code to become "a dying skill", while problem-solving and understanding systems keep their value. "We only need to challenge our identity if we mistake programming for writing code." Tim Bray, a veteran engineer, objects on different grounds. For him the problem is ethical and economic. In his words, the technology is pushed by the rich to get richer, used to cut payrolls, adds to climate damage and devalues human creative work. Fowler quotes him without rebuttal.
From direct to 5
The most substantial idea comes from Annie Vella. She draws on automation research from the 1960s. In direct control a human closes the feedback loop: act, observe, correct. In supervisory control a computer closes the loop and the human watches and steps in when needed. Coding used to be direct control. You read the requirements, changed the code, saw the result and repeated. Agents now write code, run tests and fix their own mistakes, and the developer moves toward supervision.
She maps this onto a five-level model from the researchers Endsley and Kiris:
None: the human decides and acts.
Decision support: the system suggests (early autocomplete assistants).
Consensual AI: the system decides and acts, and the human must approve each action.
Monitored AI: the system acts, and the human can veto (agent "auto-mode").
Full automation: no human in the loop. One example is a "dark software factory", by analogy with factories that run with the lights off.
Vella adds a caveat. The classic model assumes the machine reports honestly what it did. AI agents don't reliably do this, so they need more supervision, not less. She concludes that the job and the system must be redesigned together. She asks whether the 6 (software development lifecycle) should be treated "as a control system we build and supervise" rather than a process we follow. Fowler endorses this and admits nobody yet knows what the redesigned job looks like.
Building with both kinds of component
Eric Evans, who created domain-driven design (DDD), has a keynote video out. In it he describes the Domain Language Surveyor. This pipeline compares the vocabulary of different code repositories to judge whether they are related "7", the DDD term for a part of a system where words have one consistent meaning. Evans applied it to open-source healthcare codebases. Fowler stresses that the pipeline mixes ordinary computation with LLM inference.
The token bill
Companies outside the Foocamp world keep raising the cost of tokens, the units that model APIs charge by. Thoughtworks' Andy Nolan describes the firm's sourcing strategy. It rents frontier models for work that needs top-level reasoning. It runs 8 on capacity it chooses when the work carries the firm's intellectual property or runs at high volume. Each workload is scored on two axes, domain differentiation and security & sovereignty, and placed on a three-by-three grid. At one corner is the cheapest capable API, swapped as prices change. At the other is a fine-tuned open model on Thoughtworks' own hardware, which gives full control but means full upkeep. The higher score on either axis decides where the workload goes. Thoughtworks forecasts 2028 token costs falling from $16–40M to what the post prints as "$6.8-1M". That figure looks garbled, and the post doesn't explain the underlying model.
Remember
Agents that report their own work can't be trusted the way classic automation was, so supervising them takes more effort.
The main obstacle to enterprise AI may be messy data, a problem early adopters barely discuss.
Sort each workload by how much it differentiates your business and how sensitive it is before choosing a rented API or a self-hosted model.
Model cost. A better model normally costs proportionally more compute. Small gains in retrieval quality are worth a lot to advertisers, but those gains have to come without a matching rise in GPU spend.
Memory bandwidth. The pipeline was a chain of separate GPU 3, which are small programs the GPU runs one after another. Each kernel wrote its intermediate results to 4 (high-bandwidth memory, the GPU's main memory) for the next kernel to read. At billions of ads per second, even a tiny per-ad intermediate adds up to terabytes per second of memory traffic.
Network bandwidth. Two per-request steps ran on separate CPU machines. One was 5, which enforces each advertiser's targeting rules. The other was 6, which keeps the K best-scoring ads. Those machines shipped candidate lists over the network to remote GPUs for scoring. At multi-billion-per-second rates, that traffic saturated the network link.
11
Index matching: over 2x faster, reaching more than 100 billion ads/sec. That is ten times the scoring rate, so index matching could run across several retrieval pathways without becoming the bottleneck.
End to end: profiling showed that moving data between pipeline stages took a large share of latency. Aligning data layouts between index matching and scoring removed extra lookups. With everything applied, the full pipeline benchmarks at "multiple times" current production throughput.
