Today's theme is engineering discipline around models: measuring token spend, keeping routing in plain code, and using generation only where it earns its cost.
Picked from 52 new articles across 23 sources · arrow keys for other days
Today in 30 seconds
30s left
Cloudflare cut its AI code review cost from $2.43 to $0.77; fixing prompt caching and wasted calls delivered about 40% of the savings. story 1 ↓
Meta's 1,300-line code router dispatches 14 specialist agents, shrinking data-column prep from days to under an hour. story 2 ↓
OpenAI's new models jumped from 5.4% to 86.6% accuracy on a COBOL migration benchmark, at 39% lower cost. story 3 ↓
Cloudflare's Clairic extracts typed fields with confidence scores without generating tokens; its claims pipeline ran 3.2x faster and 49% cheaper. story 4 ↓
Meta replaced per-ad IDs with shared semantic tokens in its ad models and reports a 0.35% conversion lift. story 5 ↓
1/5Cloudflare· Ryan Skidmore· Oct 19
Cloudflare cuts its AI code reviewer's cost per review from $2.43 to $0.77
Cloudflare cut the cost of each AI code review by 68%, to $0.77. It made the opening prompt reusable across every repository, removed wasted model calls and retuned its models. Median review time fell from 4m09s to 1m32s.
Why it matters · Agent costs grow quietly as context and call counts creep up. This is a measured playbook for finding where the tokens go and checking that quality survives the cuts.
Know this first
Context from the editor, not the article · 41 sec
1 · Cloudflare's AI code reviewer
Cloudflare runs an internal AI reviewer on every merge request, which is a proposed code change. A coordinator agent hands the change to specialist sub-reviewers for code quality, security, performance, internal standards and documentation. A risk tier (trivial, lite or full) decides how much review each change gets.
2 · Why agent calls get expensive
Language models are billed per token, a word-sized chunk of text. An agent works in a loop: the model asks for a tool call, such as reading a file, then gets called again. Each new call resends the whole conversation so far. Cost therefore grows with both the number of calls and the length of the context.
3 · Prompt caching basics
2/5Engineering at Meta· Oct 19
Meta made its AI coding agents more reliable by giving the coordinator less to do
Meta replaced one all-knowing AI agent with a simple, rule-following router that hands each task to one of 14 narrow specialist agents. The agents stopped mixing up their knowledge, and setup work fell from days to under an hour.
Why it matters · If you build AI agents for real codebases, this shows a pattern that works in production: let code make the routing decisions, and keep each model call small and narrow.
Know this first
Context from the editor, not the article · 40 sec
1 · What an AI agent is
An AI agent is a large language model (LLM) wrapped in a loop. It reads instructions, calls tools such as code search or file edits, looks at the results, and decides the next step. Everything it knows for a task has to fit in its context window, the text it can see at one time.
2 · Too much context hurts
A bigger context window does not mean better answers. When a prompt holds many similar but different rule sets, models often apply the wrong one or blend them together. People call this context interference, or cross-contamination. It is one reason multi-agent designs split work into separate, isolated calls.
3 · Orchestrators and subagents
3/5OpenAI· Oct 19
OpenAI's new models score far better on COBOL, Fortran and other rarely used languages
OpenAI built two tests of coding in rarely used languages: building a SQLite-style database in 20 languages, and translating COBOL programs without changing behavior. Its newest models went from 27% to 87% on the first and from 5% to 98% on the second.
Why it matters · Huge amounts of banking and scientific code run on COBOL and Fortran, and few engineers still know them. Reliable AI migration would change how that code gets modernized.
Know this first
Context from the editor, not the article · 38 sec
1 · Low-resource languages
Models learn to code from public code and documentation. Python and JavaScript have huge amounts of both. COBOL (1959, built for business records), Fortran (1957, built for scientific number-crunching) and Visual Basic have far less, so models write them poorly. Yet these languages still run banks, payment networks and weather models.
2 · Why legacy migration is risky
Old systems often lack documentation, and their original authors are gone. Business rules may exist only in the code. Rewriting such a system in Java or Python means reproducing every quirk exactly, down to how numbers get rounded. Small differences can silently change financial results.
3 · How coding benchmarks score
4/5Cloudflare· Alex Reneau· Oct 19
Cloudflare's Clairic pulls typed values from messy text without generating tokens
Cloudflare released a preview of Clairic, a model that reads text or images and returns typed values with confidence scores in a single pass. It's built for extraction jobs where people now use slower, costlier chatbot-style language models.
Why it matters · Asking a large language model to "return these fields as JSON" is one of the most common production uses, and fast typed extraction with confidence scores changes how you build and check those pipelines.
Know this first
Context from the editor, not the article · 41 sec
1 · How LLMs produce output
Large language models (LLMs) like GPT or Claude write answers one token (a word fragment) at a time. Each token needs another run of the model. Long outputs are slow, you pay per output token, and the result is free text. That text may not parse as valid JSON, and it may contain plausible but invented values.
2 · System One vs System Two
Psychologist Daniel Kahneman split thinking into System 1 (fast, intuitive) and System 2 (slow, deliberate). A startup called TypeSafe borrowed the idea a month ago with Jev, a "System One" model. Jev answers multiple-choice questions with probabilities in one quick pass instead of reasoning in text. Cloudflare later built a similar decision model called Clef.
3 · Calibrated intervals
5/5Engineering at Meta· Oct 19
Meta swaps per-ad IDs for shared semantic tokens in its ad models
Meta now describes each ad by a short string of shared "meaning" tokens instead of giving it its own ID. New ads can borrow what the model learned from similar ones, and Meta reports 0.35% more conversions.
Why it matters · It shows the LLM idea of a small shared vocabulary of tokens moving into large production recommenders, and it gives a cheap way to judge a tokenizer before running a full training cycle.
Know this first
Context from the editor, not the article · 44 sec
1 · How ad recommenders work
Ad systems run in two steps. Retrieval narrows millions of ads down to a few hundred candidates. Ranking then scores each candidate, typically by predicting click-through rate (CTR, the chance a person clicks) or conversion rate. Models have traditionally learned a separate vector for every item ID from engagement history: clicks, purchases and impressions.
2 · The cold-start problem
A brand-new ad has no engagement history, so its ID vector starts out as noise. The model can't tell how good the ad is until many people have seen it. The same applies to niche, low-traffic "tail" items, which may never collect enough data. ID-per-item systems also can't see that two different IDs describe nearly the same product.
Anthropic can cache the computed start of a prompt, called a prefix, so later requests that begin identically reuse it. A prefix is stored only where a cache breakpoint ends it. Writing costs 1.25× the normal input price, reading costs 0.1×, and an entry expires after five minutes unless something reads it.
Three sub-reviewers share one cached opening request; one writes the cache and the other two read it.1 / 4
Cloudflare's internal AI code reviewer now costs $0.77 per review. That is 68% less than the $2.43 it averaged at its most expensive, in late August. The median review also got faster, from 4m 09s to 1m 32s.
Why it got expensive
The reviewer is a 1 agent that hands each 2 (a proposed code change) to specialist sub-reviewers for code quality, security, performance and internal standards. Between March and August, cost per review roughly doubled without any architectural change. Bigger code changes explained a little of it. Most of it came from moving every reviewer to pricier Anthropic models, which did better in Cloudflare's own evaluations. Tokens (the word-sized chunks models are billed by) per review tripled, from 0.91 million to 2.99 million.
Most of the spend was on cache reads and writes, which is context the model had already seen. Agents resend the whole conversation on every call, and the code quality reviewer averaged 16.6 calls per review.
The caching fix
Anthropic's 3 reuses the start of a prompt (a 4) at a tenth of the normal input price. It only stores a prefix up to a 5, and writing one costs 1.25× the normal price. Every session began with 11k–21k identical 6, yet only 14–26% of first requests hit the cache.
The cause was 7, the open-source agent harness the reviewer ran on. Its first version put the shared prompt and job-specific details (working directory, date) into one block with a single breakpoint at the end, so the shared part was never stored on its own. Splitting the block raised first-request cache hits to 91–99.5%.
The team also removed wasted calls. Sub-reviewers had been trying to use a tool they weren't allowed to call. Models had been reading files one at a time. Cancelled jobs had kept running. Three reviewers now share one cached opening request.
Models and effort
Code changes saved $0.65. Model and effort changes saved $1.06. "Effort" is a setting for how much the model reasons. Moving to Sonnet 5.5 cut per-run cost by 47–62% at the same per-token price, because it used far fewer tokens.
Quality
Serious findings fell sharply, so the team checked whether real problems were being missed. For code quality at high effort, authors now act on 59% of serious findings, up from 34%. Model judges rated important findings per review as roughly unchanged (0.30 against 0.28 before), while inflated severity ratings dropped 70%. Performance, still at medium effort, did lose real findings, and security may have. Both will be tested at high effort next. The samples are small and the latest window covers only one day.
Cloudflare's internal AI code reviewer cost $1.19 per review in April. By late August it cost about $2.46, even though the architecture hadn't changed. Ryan Skidmore reports that after a cost push starting in mid-September, an average review now costs $0.77. That is 68% below the $2.43 averaged over the two most expensive weeks. The median review also got faster, from 4m 09s to 1m 32s, and the 1 (the slowest 1% of reviews) fell from 15m 20s to 4m 41s.
How the cost doubled
The reviewer is a 2 agent that hands each 3 (a proposed code change) to specialist sub-reviewers. Risk tiers (trivial, lite and full) set how intensive the review is. Holding the mix of tiers fixed, cost per review rose from $1.22 in March to $2.32 in August. Some of that was larger changes: the share of full-tier reviews went from 59% to 65%, which the team attributes to engineers writing bigger changes with better AI tools. Most of the rise, though, came from model choices. In August every reviewer moved to Anthropic models because they did better in internal evaluations, which raised cost per job from $1.38 to $2.03. The internal-standards reviewer, which Cloudflare calls 4, moved to Claude Opus 4.7, and its cost went from $0.25 to $0.88. It then cost more than the coordinator.
Caching was already working by the usual measure: the cache hit rate rose from 85.7% to 97%. But 5 per review tripled, from 0.91 million to 2.99 million, and a high hit rate on three times the tokens still costs more.
Where the money went
Before the work started, spend split into output tokens (23%), cache writes (43%) and cache reads (34%). Most of the money went on context the model had already seen. The code quality reviewer averaged 16.6 model calls per review. Over those calls it wrote 72k tokens to cache and read back 1.08 million, because each call resends the whole conversation. That left two goals: make each session's first request read from the cache instead of writing to it, and use fewer calls.
The caching bug
Anthropic's 6 stores the computed start of a prompt (a 7), but only up to a 8. Writing costs 1.25× the normal input price, reading costs 0.1×, and entries expire after five minutes unless read. Every session opened with 11k–21k tokens of tool definitions and agent prompt that were identical in every repository. In principle these should almost always have been cache hits. Logging showed the first request hit the cache only 14–26% of the time.
The cause was in 9 v1, the open-source agent harness the reviewer ran on at the time. It put the agent prompt and an environment section (working directory, date) into one block, with the only breakpoint at the end. Because the directory and date differed between jobs, the shared part was never stored on its own.
As an analogy: picture a library that only shelves whole books. Every review's book started with the same 20 pages but ended differently, so no two books were ever the same and nothing on the shelf could be reused.
On September 16 the team added a marker at the end of each agent prompt and a plugin hook that splits the block there, giving the shared half its own breakpoint. To keep that half identical across jobs, they disabled the per-repository skills list and froze tool descriptions. First-request cache hits jumped to 91–99.5%. The same day they moved the Codex reviewer's reference index into its prompt, so it no longer spent three turns reading files. Its calls per run fell from 15.9 to 9.6 and its cost from $1.12 to $0.72. Together, these changes cut cost per review from $2.47 to $2.02.
Fewer calls
Smaller fixes on September 21 and 23 took off about $0.20 more:
Hiding a tool sub-reviewers couldn't use. Between 57% and 76% of sessions tried to post the final review comment, were rejected by the server, and spent a call recovering. Removing the tool also saved about 2.9k tokens per request.
Parallel file reads. In 64% of turns the model had read just one file. Reviewers now load the shared context and all patches up front, up to 150 KB.
The code quality, security and performance reviewers now also send identical first requests: tools, shared rules, then the full merge request, with each reviewer's own instructions last. A cache entry can't be read until the response that wrote it starts streaming. So two of the reviewers wait up to 20 seconds for the first, saving about nine cents per reviewer.
Models and effort
Splitting the period at each change, code changes account for $0.65 of the $1.67 saving and model and effort changes for $1.06. The author notes this is a before/after comparison, not a controlled experiment. "Effort" is a setting for how much the model reasons. The coordinator moved to medium effort, which finished full-tier jobs 20% faster, and then to Opus 5.5. A randomized split showed no measurable change in outcomes compared with Opus 5. The per-token price cut explains only about $0.20; cache reads per review are down 82%, writes 54% and output 67%.
On October 3, Sonnet reviewers were split 50/50 between Sonnet 5 and 5.5. At the same per-token price, 5.5 was 62% cheaper per run for code quality, 58% for performance and 47% for security, with similar numbers of serious findings. The savings came entirely from using fewer tokens. At medium effort, code quality makes 1.7 calls per run instead of 6.2. High effort costs 61% more per run (about two cents per review) but posts 58% more critical and warning findings, so code quality moved to high.
Did quality drop?
Serious findings fell 40% for code quality, 56% for security and 71% for performance. The team checked whether that meant missed problems in two ways.
First, they counted a finding as acted on if the author's next push changed the flagged lines. As a baseline for coincidental edits, lines 30–40 away changed about 15% of the time. The share of code quality's serious findings that authors acted on rose from 34% to 59%, so acted-on findings per 1,000 reviews held roughly steady (132 versus 145, within the margin of error). Across all reviewers, acted-on findings rose from 229 to 321 per 1,000 reviews.
Second, models from two providers judged 657 blinded findings, with a third model settling disagreements. Important findings per review went from 0.28 to 0.30. Findings rated more severe than deserved dropped 70%, from 0.46 to 0.14 per review.
There were losses. Code quality at medium effort produced about a quarter as many acted-on findings as at high. Performance found about 64% fewer important issues (roughly one fewer per 24 full-tier reviews). Security fell about a quarter, though that isn't statistically significant. Both are still at medium effort, and testing them at high is next.
The caveats are real. A change to flagged lines doesn't prove the finding was right, the judges are themselves models, each group has only 35–66 findings, and the "now" window is one day long. Reliability did improve: reviews ending without a verdict fell from 0.79% to 0.27%, and security retries from 3.5% to 0.04%.
Remember
A 97% cache hit rate hid the real problem: each session's first request, the expensive one, mostly missed.
Newer models cut cost mainly by making fewer calls and using fewer tokens, not through lower per-token prices.
Fewer findings can mean less noise. Check whether authors act on them before assuming quality fell.
In a multi-agent system, an orchestrator receives the request and decides which worker agents (subagents) run, and in what order. A common design makes the orchestrator itself an LLM that plans freely. The alternative is to make it ordinary deterministic code: the same input always produces the same output.
How the thin orchestrator sequences shared specialist subagents through workflow templates.1 / 2
Meta has described an AI agent system whose central coordinator, called the 1, deliberately does no thinking about the actual work. It is 1,300 lines of code that route more than 20 kinds of workflow to 14 specialist AI agents. Meta calls this the thin orchestrator pattern.
The problem
The agents automate "data column onboarding," which means adding a new data field to Meta's recommendation pipelines. That one change touches three code repositories, three programming languages, and up to six pipeline stages that must stay in sync. Python configuration files generate C++ code, which feeds data services, which fill analytics systems.
Meta first built a single 2 agent that held all the domain knowledge, including rules for more than 10 data sources. It worked in demos but 3 (produced confident, wrong output) in production. With too much in its context, the agent mixed things up. It used one subsystem's naming rules in another, or one data source's code pattern for a different source.
A second single-agent approach succeeded 70–80% of the time, but only when an engineer hand-wrote a prompt for each task. Without that, success fell to 10–20%. Meta concluded the problem was the architecture, not the prompts.
How the new design works
The orchestrator is ordinary 4 code: the same input always gives the same result. For every request it runs seven fixed steps:
read the chosen agent's instructions
check for duplicates
collect required fields
infer what it can, such as which repository is involved
check for warnings or required human reviewers
hand off to a specialist
report the result
Only step 6 uses an LLM. It runs inside a 5 that sees only its own domain. A schema agent knows the configuration registries, a population agent knows data-source code generation, and a validation agent knows builds and tests.
New use cases come from 6: fixed sequences such as Schema → Validation → Population → Testing. Supporting pipeline migration took a new template and no new agent code. More than 25 knowledge files load when a subagent runs, so an edit applies to the next session without redeploying anything.
Results
Preparing configuration for a new column now takes under an hour. It took three to four days by hand and about a day with Meta's earlier tool. About two-thirds of the system's changes land without significant human edits.
Caveats
Meta argues that a thin orchestrator limits hallucination because routing never depends on LLM reasoning. But this is a case study, not a controlled test. There's no matched accuracy comparison, "significant edits" isn't defined, and the model isn't named. The approach also depends on workflows that can be listed in advance.
Meta's engineers have described an AI agent system whose central component deliberately does almost nothing clever. A 1,300-line 1, the coordinator that receives requests, routes more than 20 kinds of workflow to 14 specialized AI agents. It does no domain work itself. Meta calls this the thin orchestrator pattern and argues it is the reason the system works in production.
The job being automated
The task is "data column onboarding" in Meta's recommendation systems, which means adding a new field of data that flows through the pipeline. The pipeline is 2: the system is defined by code files rather than by settings in a database. Python configuration files generate C++ code. That code feeds data processing services, and those services fill analytics systems. Adding one column touches three code repositories and three programming languages. Each column passes through up to six pipeline stages, and they must all stay consistent. Done by hand, preparing the configuration for a new column took three to four days. An earlier config-driven tool cut that to about a day.
Why one smart agent failed
Meta's first attempt was the obvious one: a single LLM agent loaded with everything. That meant configuration schemas, code generation rules, pipeline routing logic, validation requirements, and data population patterns for more than 10 data sources. It worked in demos. In production it started hallucinating, meaning it produced confident but wrong output.
The team says that past a certain amount of context, extra knowledge made results worse. The agent cross-contaminated concepts. It applied field naming rules from one subsystem to another, or used the code pattern for one data source when filling data from a different one. Meta cites a preprint from Mount Sinai's AI department with a similar finding. At batches of 80 tasks, orchestrated multi-agent systems kept 65.3% accuracy while single agents fell to 16.6%, and the multi-agent systems used up to 65 times fewer tokens (the units of text a model processes). That paper's authors attribute the gap to task isolation.
Meta also tested a different single-agent approach. It succeeded 70–80% of the time, but only when an engineer hand-wrote a custom system prompt for each job. Without that, success dropped to 10–20%. The team's conclusion was that the problem was the architecture, not the prompts.
The two layers
The redesign splits the system in two:
A 3 orchestrator. It classifies what the user wants, gathers the required fields, runs safety checks, and hands off to a specialist. It never reasons about code, generates configuration, or edits the codebase.
Fourteen 4, each a narrow expert with its own isolated context. A schema agent knows the configuration registries. A population agent knows how to generate code for each of the 10 data sources. A validation agent knows the build systems and quality checks. No agent sees the whole picture.
Every request goes through the same seven-step protocol:
Read the chosen subagent's instruction file.
Check whether the target already exists, to avoid duplicates.
Gather required fields from the user.
Automatically work out what can be inferred, such as repository type and existing configurations.
Run a reliability gate: check for active warnings or required human reviewers.
Delegate to the subagent with the gathered context.
Handle the result, surface any errors, and confirm success.
Only step 6 involves LLM reasoning, and it runs inside a subagent that has only its own domain knowledge loaded. The other six steps are ordinary code.
Workflow templates, not new agents
The orchestrator keeps what happens separate from how each step is done. Phase order comes from 5, which are fixed state machines (predefined sequences of stages with set transitions). They limit what the LLM can decide:
Column onboarding: Schema → Validation → Population → Testing
All templates draw on the same 14 subagents. When the team needed pipeline migration, a whole new use case, it wrote a new template that ran existing agents in a different order. That took no new agent code.
Each subagent has the same five-part structure:
the knowledge files it loads (typically a few thousand tokens each)
a checklist of inputs the orchestrator must collect first
a step-by-step workflow
patterns for finding relevant code in the 6 (Meta's single shared code repository)
a table that maps common errors to fixes
Meta says this makes the 15th subagent look structurally the same as the first.
Knowledge lives in more than 25 files, grouped by workflow domain. Subagents load them when they run, rather than having them built into the orchestrator. An edited knowledge file takes effect in the next session without redeploying anything, and the same files serve several platform versions of the agent.
An illustrative analogy (ours, not Meta's): the orchestrator works like a hospital triage desk following a checklist. It confirms who the patient is, collects the forms, checks for allergy flags, and sends the patient to cardiology. It never tries to diagnose. The cardiologist never sees the dermatology notes, so they can't mix the two up.
Results
Configuration prep for a new column: under an hour, compared with three to four days by hand and about a day with the earlier tool.
About two-thirds of the system's code changes land without significant human edits.
Meta says the system reaches this by trading generality for deep domain knowledge. The orchestrator is general and the subagents are specialists.
Meta's argument for why it works
The authors give four reasons:
Routing is not reasoning. "Which subagent handles this?" can be decided by keyword matching or classification, so an LLM isn't needed for it.
Isolation prevents cross-contamination. The agent working on data source A never sees source B's patterns, so it can't apply the wrong one.
Failures stay contained. A failed subagent gets retried or escalated to a human without corrupting the orchestrator or any other agent.
Each layer improves on its own. A better model upgrades all 14 agents at once, a knowledge-file edit applies immediately, and a new template adds a capability.
The strongest argument, Meta says, is how the alternative fails. When a heavyweight LLM orchestrator hallucinates, the error spreads downstream: it routes to the wrong agent, collects the wrong fields, or skips safety checks.
What to be cautious about
This is an internal case study, not a controlled experiment. Meta gives no before-and-after accuracy figure for the new system on the same task set. The 65.3% vs 16.6% numbers come from someone else's preprint, not from Meta's measurements. The "two-thirds land without significant edits" figure does not define "significant," and the article doesn't say how often the other third fails or why. It also doesn't name the model used, or explain how intent classification can be deterministic when users write requests in free-form language.
The pattern also has an obvious cost. It works because the domain is well mapped and the workflows can be listed in advance. Open-ended tasks that don't fit a template would still need some planning by an LLM.
Next, Meta plans to extend the system to more pipeline domains. It is also exploring automated feedback loops that spot failure patterns in production and propose edits to subagent knowledge files, with a human reviewing every change.
Remember
Adding more knowledge to one agent made it worse: it mixed up rules from similar subsystems.
Only one of seven orchestration steps uses an LLM; routing, duplicate checks and safety gates are plain code.
New use cases came from reordering existing agents in a template, with no new agent code.
A coding evaluation gives a model a task and then runs automated tests against what it produces. The test pass rate is the share of tests that pass. A benchmark is close to saturated when top models score near 100%, so it can no longer separate strong models from stronger ones.
saturation
When top models score near 100% on a benchmark, so it can no longer tell strong models apart.
Image: OpenAI
OpenAI says its newest models are much better at writing and translating code in 1. These are languages like 2, 3 and Visual Basic, which have little public code for a model to learn from but still run critical systems. OpenAI cites figures that COBOL supports 95% of ATM transactions and 80% of in-person card payments, and that Fortran runs weather forecasting and nuclear safety analysis.
The problem
Coding models are weak in these languages, and the systems written in them are hard to maintain. Documentation is thin, the original authors are gone, and business rules often exist only in the code. Any fix or rewrite has to keep the exact existing behavior.
Test one: build a database in 20 languages
4 asks a model to build a database engine compatible with 5, a small, widely used SQL database. The model writes it from scratch in one of 20 languages over a long coding session. To grade every language the same way, the database file serves as the common interface. The model's program takes a database file and a SQL query and writes out a new database file. The real SQLite then checks whether the contents are correct. The tests come from SQLite's own test suite.
At maximum 6 (a setting that lets the model think longer), the average pass rate rose from 26.8% for GPT‑5.6 Sol to 87.4% for GPT‑6 Astra. COBOL went from 10.9% to 72.1% and is still the hardest language. Fortran went from 21.4% to 83.5%. The cheaper GPT‑6 Luna reached 40.3% at about $0.77 per attempt, versus $3.48 for GPT‑5.6 Sol. OpenAI is publishing this benchmark on GitHub.
Test two: migrate COBOL faithfully
The second test asks for COBOL programs to be rewritten in C++, Java, Kotlin, Python or Rust. The subtle part is matching COBOL's semantics. For example, COBOL drops extra decimal digits unless the code explicitly asks for rounding. A rewrite that rounds will run without errors and still produce different financial totals. The hidden tests demand byte-for-byte identical output, and a migration passes only if every test passes.
GPT‑6.1 Sol scored 86.6% at low reasoning effort, up from 5.4% for GPT‑5.6 Sol, and cost 39% less. At higher effort it reached 97.9%, so the test is nearly maxed out.
Caveats
Both benchmarks are OpenAI's own. The size of the COBOL test isn't disclosed, and the article doesn't say it will be released. The headline baseline, GPT‑5.6 Sol, wasn't OpenAI's strongest earlier model: GPT‑5.6 Terra scored 37.9% on MultiLangBench. OpenAI itself says real migrations still need expert review and testing against the system's actual business rules.
OpenAI reports that its latest models are much better at writing and migrating code in 1. These are languages like 2, 3 and Visual Basic, which have little public code and documentation for a model to learn from. To show the gains, OpenAI built two evaluations and published results for models from GPT‑5.4 through GPT‑6.1. On both tests the newest models jump from failing most tasks to passing nearly all of them.
Why these languages matter
OpenAI cites outside figures to show how widely this code is used. COBOL, a 1959 language built for business data processing, supports more than 40% of online banking systems, 80% of in-person credit card transactions and 95% of ATM transactions. Fortran, a scientific computing language from 1957, runs weather forecasting and nuclear reactor safety analysis.
The trouble is not only that models have seen little of this code. The systems themselves are hard to work on. Documentation is often incomplete, the original developers have left, and important business rules may exist only in the code. Whether a team is fixing a bug or porting a system to Java, it has to keep the behavior the business depends on. OpenAI says developers and enterprise customers told it that coding models struggle here.
Test one: build 4 from scratch, in 20 languages
5 asks a model to build a database engine compatible with SQLite, the small, widely embedded SQL database, in a chosen language. Twenty languages are tested. This is a long coding session, not a single function. The model has to organize a codebase, manage state, handle errors and add features without breaking earlier ones.
Twenty languages raise a fairness problem: how do you run the same tests against a Fortran program and an OCaml program? OpenAI's answer is to make the database file the shared interface. The candidate program receives an existing database file and a SQL query, and it writes out a modified database file. The real SQLite then opens that output and checks that its contents are correct. As an illustration, the test might pass in a database with a users table plus INSERT INTO users VALUES (3,'ana'), and then confirm with real SQLite that the row is there. The tests are adapted from SQLite's own test suite. Scores are the test pass rate plus the API cost of the run.
Average pass rates across all 20 languages, at maximum 6 (a setting that lets the model spend more computation thinking before it answers):
GPT‑5.4: 17.4%
GPT‑5.5: 25.8%
GPT‑5.6 Sol: 26.8%
GPT‑5.6 Terra: 37.9%
GPT‑6 Luna: 40.3%
GPT‑6 Sol: 57.7%
GPT‑6.1 Sol: 87.0%
GPT‑6 Astra: 87.4%
Astra scores above 83% in 19 of the 20 languages. By language, it moved COBOL from 10.9% to 72.1%, OCaml from 25.6% to 85.8%, Fortran from 21.4% to 83.5% and Visual Basic from 27.5% to 89.4%. COBOL is still the hardest language.
Luna is the cheaper option. It reaches only 40.3%, but it beats GPT‑5.6 Sol in every language at an estimated $0.77 per sample versus $3.48, about 78% less. OpenAI does not give Astra's cost. The article also never explains what the Sol, Terra, Luna and Astra names mean, though the results suggest they are tiers that trade cost against capability.
OpenAI is releasing the MultiLangBench setup, the test suite and the per-language results on GitHub.
Test two: migrate COBOL without changing behavior
The second evaluation gives a model a COBOL program and asks it to rewrite it in C++, Java, Kotlin, Python or Rust so that it does exactly what the original did.
The hard part is COBOL's specific semantics, not just its syntax. OpenAI's example is 7, meaning numbers stored with a set number of decimal places. COBOL discards extra decimal digits unless the code explicitly asks for rounding. Suppose (an illustration, not from the article) a field holds two decimal places and a calculation produces 10.236. COBOL stores 10.23. A Python rewrite that "helpfully" rounds stores 10.24. Both programs run without errors, yet across millions of transactions their totals drift apart. Rules about how missing records are handled carry the same kind of hidden business meaning.
Grading is strict. Hidden tests require byte-for-byte identical observable output, and a migration counts as a success only if every test passes.
GPT‑5.6 Sol scored 5.4% at low reasoning effort. GPT‑6.1 Sol scored 86.6% at the same setting, at 61% of the API cost (39% cheaper). At higher reasoning effort it reached 97.9%, which OpenAI says is close to saturating the evaluation, meaning the test can no longer show much difference between top models.
What to make of it
Taken at face value, these are large capability changes. A move from 5% to 87% on behavior-preserving migration is the difference between a toy and a tool worth trying. Some caveats apply:
Both evaluations are OpenAI's own. MultiLangBench is being published, so others can inspect and rerun it. The article does not say the COBOL migration benchmark will be released, nor how many programs it contains or how large they are.
The baseline was chosen favorably. The headline comparisons are against GPT‑5.6 Sol, but GPT‑5.6 Terra already scored 37.9% on MultiLangBench. Against Terra, Luna's lead is small.
The settings differ. The top MultiLangBench numbers use maximum reasoning effort, and costs are estimates.
A near-saturated benchmark stops being informative. 97.9% shows the test is too easy for the newest model. It does not show that migration is solved.
OpenAI says plainly that production use still needs expert review and testing against the system's real business rules and dependencies. Real legacy systems also bring databases, job schedulers and decades of dependencies that self-contained test programs don't capture.
OpenAI says it wants models to bring general engineering skills to more languages: reading unfamiliar repositories, planning changes, using build tools, running tests and debugging. It is asking developers for feedback on where the models still fail.
Remember
Using the database file as the shared interface lets one test suite grade engines written in any of 20 languages.
Migration is hard because of semantics like COBOL truncation: a rewrite can run cleanly and still produce different results.
The baselines were chosen favorably. GPT-5.6 Terra already scored 37.9%, and the COBOL test's size isn't disclosed.
An 80% interval is a range the true value should fall inside about 80% of the time. A model is "calibrated" when this holds in practice. Across 100 forecasts, roughly 80 actual values should land inside their stated ranges. Far fewer means the model is overconfident.
The tilted, partly illegible receipt photo Clairic read to extract prices, tax, tip and a null discount.1 / 3
Cloudflare has released a preview of Clairic, a model on 1 (Cloudflare's hosted model service). It reads messy input, such as an email, a CSV dump or a receipt photo, and returns typed values instead of generated text. Open weights (downloadable model parameters) are promised later.
The problem
A common production use of 2 is "return the invoice number, vendor and due date as JSON". LLMs write output one token (word fragment) at a time. That is slow, you pay per output token, and the JSON can be malformed or contain misread digits. Teams end up adding validation, retries and spot checks.
Cloudflare already offers 3, a fast "System One" model that picks among options you list and gives each a probability. But an invoice total can't be listed in advance. Clairic covers that case: values rather than choices.
How it works
You send the input plus named fields, each with a type: text, number, date, duration, list, estimate or forecast. Clairic answers all of them in one 4, a single run through the network with no generation. You pay only for input, and responses usually come back in under a second.
What comes back:
Text answers are exact spans with character offsets, so you can trace them to the source.
Every value has a confidence score and p_null, the probability it isn't in the input. Missing fields return null instead of a plausible fake.
Estimates and forecasts come with a median and an 5.
In one example email, Clairic added a $649 price and $52.10 tax to get 701.10, and counted 11 days between two dates. It returned null for a tracking number that wasn't there.
Results
Forecasting: over 21 days of U.S. airport passenger counts, forecasts were off by 3.3% on average. 17 of 21 actual values fell inside the 80% interval. Three of the misses fell on a holiday weekend.
Full pipeline: in a damage-claim pipeline, Clairic extracted the facts, Clef chose refund or repair, and the 6 language model wrote the reply. That took 19.3 seconds against 61.7 seconds with GLM-5.3 doing every step, and cost $6,032 versus $11,740 per million claims. Both versions got the answers right. Clairic's repair estimate was identical on every run; GLM-5.3's varied.
Caveats
Cloudflare says longer chains of arithmetic aren't reliable yet: let code do the math. Estimates need evidence in the input, intervals can be too narrow on unfamiliar data, and forecasting is basic. Every answer is well-formed, not necessarily correct. The benchmark is small and Cloudflare ran it itself.
Cloudflare has released a preview of Clairic, a model on 1 (Cloudflare's hosted model service), under the name @cf/cloudflare/clairic. It also plans to publish the model as 2, meaning the trained parameters will be downloadable. Cloudflare calls Clairic the first "System One extraction model". It reads unstructured input, such as an email, a CSV dump or a photo of a receipt. It then returns typed values: names, numbers, dates, durations, lists, estimates and forecasts. Every answer comes with a confidence score, and it never generates text.
The problem it targets
A large share of production LLM traffic is the same structured question asked over and over. What's the invoice total? When was this delivered? How many days late is it? The usual approach is to prompt an LLM to "return these fields as JSON". The author, Alex Reneau, lists the ways this fails:
The output may not be valid JSON.
A digit can be misread while it's being generated.
A copied snippet can't be traced back to where it appeared in the source.
Teams wrap this in layers of parsing, schema validation, retries and spot checks. They also pay for reasoning and output tokens on every field of every document.
Cloudflare's earlier System One model, 3, doesn't fit these questions either. Clef is a decision model: you list the options and it returns a probability for each. An invoice total can't be listed in advance. Clairic fills that gap. Cloudflare sorts the tasks into three kinds:
Choose from a fixed set: Clef.
Produce a value: Clairic.
Write open-ended text: an LLM.
How a request works
A request has two parts. The first is a state: the text to read, plus up to four optional images. The second is a map of named fields. Each field gets a type and, optionally, plain-language instructions. The types are:
text: an exact span copied from the input, with its character offsets (positions in the input).
number: honors integer and decimal-place rules.
date, time, datetime: returned in ISO format.
duration: in a unit you choose.
list: matching items in input order.
estimate: a distribution inside your min and max, on a step grid you choose, linear or log scale.
forecast: one estimate per future step of a series.
The model answers every field in a single 4, meaning one run through the network with no token-by-token generation. You pay only for input, and responses usually take under a second. Asking for thirty fields costs about the same as asking for three. If the fields don't fit next to the input, the server splits them into parallel passes that each see the whole input, then merges the answers.
Each scalar answer includes confidence and p_null, the probability that the value isn't in the input. If the value is likely missing, the field comes back null, along with the guess the model would otherwise have given. Estimates include a median and an 5 (the 10th to 90th percentiles, called p10 and p90). All 99 percentiles are available on request. Because the output is always a value of the declared type or null, it can't come back malformed. Cloudflare stresses that well-formed is not the same as correct.
Worked examples
Take this support email: "order A-88412 arrived on Sept 28, 2026 with a cracked screen and a dented corner. I paid $649.00 plus $52.10 tax. It's now Oct 9, 2026…" Clairic returned:
The order ID, as a span at offsets 10–17.
The delivery date, 2026-09-28.
amount_paid of 701.10. That number appears nowhere in the email; the model added price and tax.
days_since_delivery of 11, worked out from the two dates.
Both kinds of damage, as exact spans.
A tracking_number field came back null with p_null of 1. Its rejected guess was "$52.10", at confidence 0.42: exactly the kind of plausible wrong answer an LLM might have returned as fact. A tilted, partly illegible receipt photo yielded quantity, unit price, subtotal, tax and tip. It correctly gave a null discount. The confidence scores were lower on the harder fields, such as 0.78 for the order number.
For estimation, Clairic was given five comparable home sales and a subject property. It estimated a sale price with a median of $415,000 and an 80% interval of $370,000–$470,000. The interval's width is the signal: narrow means the input strongly supports the answer, wide means it doesn't.
Forecasting, tested on real data
Each morning for 21 days, Cloudflare gave Clairic the daily count of people screened at U.S. airport checkpoints by the TSA (Transportation Security Administration) and asked for the next day's count. The average error was 3.3%. 17 of the 21 actual values fell inside the 80% interval, close to the expected 80% (about 17 of 21). The model picked up the weekly pattern, with quieter Tuesdays and Saturdays. Three of the four misses fell on Columbus Day weekend, a holiday the data series gave no hint of. Cloudflare says plainly that this does not replace a dedicated time-series model.
The cascade benchmark
The headline comparison is a three-step damage-claim pipeline:
Clairic extracts facts from the email, including an estimated repair cost.
Clef picks "refund" or "repair" from a five-line typed summary and a written policy.
6, an open LLM, writes the reply to the customer.
The baseline used GLM-5.3 for all three steps.
Cascade
All GLM-5.3
End-to-end time
19.3 s
61.7 s
Extract
0.46 s
16.1 s
Decide
0.22 s
21.9 s
Reply
18.6 s
16.4 s
Cost per million claims
$6,032
$11,740
That works out to about 3.2 times faster and 49% cheaper. Nearly all of the cascade's remaining time and cost comes from the one generative step, the reply. Both pipelines got the facts and the decision right in every run. Clairic's repair estimate was identical on every run, while GLM-5.3's varied ($250, $250, $220). In the all-LLM version, Gemma 4 and Qwen failed to work out values like the repair cost. Cloudflare also argues the design narrows 7, where text in the input tries to give the model instructions. Clef never sees the raw email, and the LLM sees only the decision and the facts it needs.
Caveats and open questions
Cloudflare lists its own limits for this early checkpoint:
One- or two-step arithmetic is reliable. Longer chains of dependent steps, and totals across many table rows, are not. Cloudflare's advice is to have the model extract the values and let your code do the math.
Estimates need evidence in the input, such as comparables, history or description.
Intervals are trained to be calibrated but can run too narrow on unfamiliar data.
Forecasting is basic.
Some questions are left open. The benchmark rests on a handful of runs of one pipeline, and the post gives no extraction accuracy across a standard dataset. It doesn't describe the architecture, or how one pass manages to compute sums and dates. The run-to-run consistency is to be expected from a single non-sampling pass, but consistency is not accuracy. Clairic is a preview, so production adoption would mean building your own evaluation around its confidence scores and p_null.
Remember
In one pass, Clairic returns a value of the declared type or null, plus confidence and p_null for review routing.
Nearly all the cascade's remaining latency and cost came from the single text-generating reply step.
Don't trust Clairic with multi-step arithmetic yet: extract the values with it and compute in code.
An embedding is a list of numbers that places an item in a space where similar items sit close together. Quantization rounds such a vector to the nearest entry in a fixed table called a codebook. What comes out is a discrete code, like a word, which a language-model-style network can read as a token.
cold start
The difficulty of recommending a new item that has no engagement history yet to learn from.
Overview: one tokenizer serves both retrieval and ranking for ads, products, pages and pixels.1 / 4
Meta's ads team has changed how its recommendation models represent items. They used to give every ad, product and page its own ID. The system they call 1 instead describes each item with a few tokens drawn from a stable, shared vocabulary. Combined with 2 (Meta's Generative Ads Model) and its Adaptive Ranking Model, Meta says the change raised advertiser conversions by 0.35%, measured in February 2026.
The problem with one ID per item
A classic recommender learns a separate vector for each item ID from how people engage with it. A new ad starts with nothing. This is the cold-start problem. Niche "tail" ads with few impressions never collect much signal either. To the model, a new trail-shoe ad looks no more related to last season's trail shoe than to a toaster.
How the tokenizer works
Language models handle this with a small vocabulary whose tokens combine to describe anything. Meta applies the same idea to items in three steps:
Encode. Pretrained 3 models (models that read both text and images) turn an item into an 4, a vector that captures what the item is. Meta blends this with signals learned from how people engage with the item.
Similar items end up sharing tokens, much as related words share roots. A new shoe ad starts with what the model already knows about tokens like "running footwear."
A cheap way to judge a tokenizer
Meta scores each tokenizer on its own properties:
Cluster purity: items that share a code also share a category.
Balance: items are spread evenly across codes.
Codebook utilization: most codes are actually used.
Across eight tokenizer versions, better scores went with better 7 (click-through rate) prediction. That means a tokenizer can be compared without retraining a full ranking model each time. Meta doesn't publish the correlation figures.
The flywheel claim
Meta argues the gains compound. Better predictions for tail ads produce better training data, and that improves predictions for all ads. It also says tokens improve how performance scales with sequence length, training data and model size, but gives no numbers for this.
Caveats
The post is light on detail. It reports a single headline number with no baseline or confidence interval. Next on Meta's list: wider token coverage, mixing tokens with compacted text descriptions of ads, and possibly one shared vocabulary across text, image, audio and video.
Meta's ads engineering team has published an account of 1, a change in how its recommendation models represent ads, products, pages and pixels. Pixels here are the tracking snippets advertisers put on their websites. Older models give every item its own ID and learn a separate vector for each one. The new system describes every item with a short sequence of tokens drawn from a stable, shared "semantic vocabulary." Meta paired it with 2, its Generative Ads Model, and its Adaptive Ranking Model, and reports a +0.35% lift in conversions across surfaces, from internal data measured in February 2026.
What problem this solves
A conventional recommender is discriminative: it scores each candidate ad separately. It also relies on item IDs. Every ad gets a learned 3 (a vector of numbers), and that vector only becomes useful after enough people have seen, clicked or bought. This causes two familiar problems.
Cold start. A newly created ad has no history, so the model knows almost nothing about it.
Tail sparsity. Niche products and small advertisers' ads may never gather enough impressions for their ID vectors to settle.
IDs are also ephemeral. When an advertiser launches a new campaign, the new ID shares nothing with the old one, even if the product is almost identical. Meta uses this example throughout: a sportswear brand's new trail shoe ought to inherit what the model learned from last season's shoe, but under pure IDs it starts from zero.
Meta's broader project, GenRecs (Generative Recommenders), uses large-language-model (LLM) architectures to reason about ad content and user preferences together, rather than scoring each candidate on its own. For an LLM-style model to do that, items need to look like language: discrete tokens from a fixed vocabulary. The tokenizer provides that.
How the tokenizer works
The analogy Meta draws is to how language models handle words. An LLM doesn't keep a dedicated entry for every concept in the world. It composes a few tens of thousands of tokens into anything. GenRecs Tokenization does the same for items through a three-stage pipeline:
Encode. Pretrained 4 models read the item's text and images and produce an embedding that carries "world knowledge," such as the fact that a trail shoe is footwear for running off-road. The tokenizer combines this content signal with representations learned from how people actually engage with items.
Decompose. The embedding is split into independent subspaces.
Quantize. Each subspace is mapped to its nearest entry in a 5, a learned table of representative vectors. The index of that entry becomes a token.
The result is a short sequence of discrete codes, often called a semantic ID. Similar items share some or all of their codes, much as related words share roots.
The post says that for many ad representations, Meta combines several techniques: a learned encoder, 6, learned vector codebooks, 7, and token parametrization. Here is how the two less obvious ones generally work. This is an illustrative sketch, not Meta's published numbers.
Residual decomposition. Quantize the vector to its nearest first-level code, say code 17, which roughly means "footwear." Subtract that code's vector, then quantize what remains with a second codebook, giving code 203, roughly "running." Repeat for a third level, code 45, "trail, cushioned." The shoe becomes (17, 203, 45). Each level refines the one before it.
Prefix n-grams. Rather than giving the model only the full triple, give it the prefixes too: (17), (17, 203) and (17, 203, 45), each with its own learned embedding. A brand-new trail shoe that lands on (17, 203, 88) still shares its first two prefixes with every well-learned running shoe, so it inherits their signal right away.
Meta says the best combination of these stages depends on the entity type and the features involved.
Measuring token quality without full training
A practical finding in the post is that a tokenizer can be judged by its own properties before anyone trains a ranking model on it. Meta tracks three intrinsic metrics:
Cluster purity: items that share a code tend to share a category, so each token carries a cleaner signal.
Distribution balance: items spread evenly across semantic IDs instead of crowding into a few.
Codebook utilization: a larger share of codes contains at least one item, so the vocabulary can express more distinctions.
Across eight tokenizer versions, Meta found a strong correlation between these metrics and downstream 8 (click-through rate) prediction accuracy. Meta doesn't publish the correlation coefficient or the size of the accuracy differences. If the finding holds, though, it shortens the iteration loop: tokenizers can be compared cheaply instead of retraining a large ranking model for each one.
To improve codebooks, Meta lists three techniques. A diversity loss penalizes the model when it overuses a few codes. Dead-code resets reinitialize codebook entries that no item maps to. Contrastive objectives pull similar items' representations together and push dissimilar ones apart. Meta reports that these gains carry over when the resulting tokens are used to train generative recommenders.
The flywheel argument
Meta argues that the benefit compounds. For high-volume advertisers, tokens expose patterns across large product catalogs. For tail ads, shared tokens let the model generalize from similar content, recovering cold-start performance. Better tail predictions mean those ads are shown to more suitable people. That produces better training examples, which in turn improve predictions for all ads. The post presents this as a self-reinforcing loop but doesn't quantify it.
Meta also claims that tokens "bend the scaling curve" along three axes: sequence length, training data and model architecture. In other words, performance improves faster as each grows. No curves or numbers back this up in the post.
Retrieval and what comes next
The same vocabulary serves retrieval as well as ranking. Meta describes a 9 retrieval model, where one network embeds the user, another embeds the item, and their similarity selects candidates. In this version both towers are built on a shared LLM encoder. Meta says it reaches state-of-the-art retrieval quality, stays efficient to serve, and matches classic two-tower performance with a fraction of the training data. Again, no figures are given.
Planned work includes:
Wider coverage of tail inventory, where Meta expects the biggest advertiser impact.
Tokenizer quality as a first-class objective, with engagement-aligned training and one vocabulary shared across entity types.
Efficient serving. Describing an ad fully in words ("fully verbalized" tokens) gives a model the richest view, but needs context windows too long for ads-scale latency. Meta is mixing compacted verbal tokens with semantic IDs.
Longer term, a single token space across text, image, audio and video for both user interests and advertiser offerings.
What to make of it
The idea is not new to the field. Semantic IDs built by residual quantization have appeared in published research from Google and from Meta itself. What's new is the claim of large-scale production deployment across item types, plus the evidence that intrinsic metrics predict tokenizer quality. The evidence is thin for an engineering audience, though. There's one aggregate number (+0.35%) with no baseline, variance or breakdown by head versus tail ads. The scaling and retrieval claims come without data. The post also doesn't separate the tokenizer's contribution from GEM's and the Adaptive Ranking Model's. A 0.35% conversion gain is still meaningful at Meta's ad volume, but the post doesn't show how much of it the tokenizer is responsible for.
Remember
Shared tokens let a brand-new ad inherit what the model learned from similar items, easing cold start.
Across eight tokenizer versions, cluster purity, balance and codebook use tracked CTR accuracy, so tokenizers can be compared cheaply.
The +0.35% conversion lift has no baseline or breakdown, and the scaling claims come without numbers.
Skipping reviewers on re-reviews. About two-thirds of jobs are repeat reviews.
Dropping an unseen summary turn. This saved about $0.07 per job.
Stopping cancelled jobs. GitLab's 10 doesn't signal a job's script when the job is cancelled, so in a sample of 446 cancelled jobs, 358 ran to completion. Jobs now check their own status every 15 seconds.
Decompose. The embedding is split into independent pieces. One method Meta uses is 5: code the vector coarsely, then code whatever is left over.
Quantize. Each piece is matched to its nearest entry in a learned 6, which yields a short sequence of discrete tokens.
