Grading the graders, and their benchmarks

The post ranks the models, but Husain mainly uses the comparison to make a methodological point. Many teams take the default 0.50 decision threshold and the raw probabilities as given, and both can leave a lot of performance unused.

The task and data

The data came from an apartment-leasing support agent that works over phone and messaging. Of 100 traces (full logs of a conversation plus tool calls and outputs), two internal QA examples were dropped, leaving 98. The reference labels mark 61 traces as failures and 37 as clean. Of the 61, 39 came from the original human review and 22 were found by the eval tools Braintrust and Arize and then checked by hand. Failures include contradicting a system instruction or tool result, making unsupported claims, missing or falsely claiming a booking, transfer or cancellation, and leaving a request unresolved.

Every model got the same criteria, with masked traces and no labels. Jev got an equivalent version through its structured-prompt API, which has separate fields for instructions and for what counts as true or false. The task is deliberately broad ("does this trace contain any substantive failure?") so that the differences between models stand out.

The comparison covers performance against human labels and API cost from token usage. Latency is left out because it depends on batching, vendor arrangements and network distance. Husain does give mean response times from separate runs, and notes they don't establish throughput or tail latency: Jev 4.33 s, Perplexity 7.85 s, Liquid d1 7.38 s, d1-ultra 1.56 s, Clef Flash 2.09 s, OpenAI 2.77 s.

Why the threshold matters

A probability only becomes a pass/fail call once you choose a cutoff. Husain's example shows why the default can mislead. Suppose a model scores every failure at 0.40 and every passing trace at 0.10. It separates the classes perfectly, and a cutoff of 0.30 would get everything right. At 0.50 it flags nothing.

Real score distributions overlap, so choosing a threshold trades off two quantities:

• Precision: the share of flagged traces that really fail.
• Recall: the share of real failures that get flagged.

He plotted score distributions for failures and passes. OpenAI separated the two classes best.

Set minimums first, then tune

Before tuning anything, Husain sets a floor. For an alerting system he wants at least 80% precision and 80% recall. On precision-recall curves (plots of precision against recall as the threshold moves), only Perplexity, Liquid d1-ultra and OpenAI Decisions have any region that meets both. Jev, Liquid d1 and Clef Flash were dropped as candidates.

The results cut both ways. Liquid d1 and Clef Flash gained the most from tuning but still fell below the minimums. The candidates gained less, but Husain notes that even small shifts in precision and recall can matter to the business. The constraint was also often unachievable. No threshold met both minimums in 36 of 100 training splits for Perplexity and 29 for d1-ultra, while OpenAI met them in every split. In those infeasible splits the unconstrained F1 optimum was used and recorded. Husain warns that meeting the minimums during selection doesn't guarantee them on new data.

Performance against cost

After tuning:

Can you read the score as a probability?

If you want to use a score as a confidence estimate, for example to route uncertain cases to a human, it needs to be calibrated. Among traces scored around 0.80, about 80% should actually fail. Husain sorted each model's scores into five groups of 19 or 20 traces and compared the predicted rate in each group with the observed rate.

Limits and advice

On the broader question of decision models versus LLMs, Husain points to Randy Olson's finding. Decision models match LLMs on classification and routing at lower cost, while LLMs pull ahead on tasks that need knowledge and reasoning. Here the LLM-backed OpenAI option won, which suggests hard eval tasks may justify the price. Husain is open that none of this is new: it's standard classifier evaluation. He argues data scientists already have these skills.

Why this is hard

Two things make the work difficult. First, training data is scarce: a model has seen orders of magnitude less COBOL than Python. Second, legacy maintenance is mostly about keeping existing behaviour intact. The documentation is often incomplete, the original authors have left, and some business rules exist only in the code. A rewrite that looks correct can still be wrong in ways that cost money.

OpenAI built one evaluation for each skill: building new software in an unfamiliar language, and preserving what old software does.

MultiLangBench asks a model to build a SQLite-compatible database engine in a given language, across 20 languages. SQLite is the widely used embedded database. This is a long, agentic task. The model has to lay out a codebase, manage state, handle errors, and add features without breaking earlier ones.

The clever part is how answers are graded the same way in every language. The database file itself is the shared interface. The candidate program gets a starting database file and a SQL query, and writes out a modified database file. The reference SQLite implementation then checks whether that file has the expected contents.

Here is how a single test case works, as an illustration. The program receives a database containing table t with three rows, plus the query DELETE FROM t WHERE id = 2. It must write a valid SQLite file in which t has two rows. Real SQLite then opens that file and checks it. Because the check never looks at the source code, it works the same for a COBOL engine and an OCaml engine. The tests are adapted from SQLite's own test suite. The score is the share of tests passed, reported alongside API cost.

• GPT-6 Astra averaged 87.4% across the 20 languages, up from 26.8% for GPT-5.6 Sol. It scored above 83% in 19 of the 20 languages.
• COBOL was still the hardest language: 10.9% → 72.1%. Fortran went 21.4% → 83.5%, OCaml 25.6% → 85.8%, and Visual Basic 27.5% → 89.4%.
• GPT-6 Luna, the cheaper model, reached 40.3% on average. That beats GPT-5.6 Sol in every language, but by much less than Astra (COBOL 10.9% → 17.1%). It cost about $0.77 per sample versus $3.48 for GPT-5.6 Sol, roughly 78% less.

OpenAI's chart of average pass rates also shows the progression across model generations: GPT-5.4 at 17.4%, GPT-5.5 at 25.8%, GPT-5.6 Sol at 26.8%, GPT-5.6 Terra at 37.9%, GPT-6 Luna at 40.3%, GPT-6 Sol at 57.7%, GPT-6.1 Sol at 87.0% and GPT-6 Astra at 87.4%. So GPT-6.1 Sol nearly matches Astra on this benchmark. The post doesn't give Astra's cost, so the trade-off between the two can't be judged from the article.

OpenAI is releasing the MultiLangBench setup, test suite and per-language results on GitHub. Developers can check what is actually tested and look at the languages they care about.

Evaluation 2: COBOL migration

The second evaluation asks a model to port existing COBOL programs to C++, Java, Kotlin, Python or Rust without changing their behaviour. The difficulty is that COBOL's language semantics carry business rules.

Results:

What to make of it

The size of the jumps stands out. Going from 5% to 87% on a pass/fail migration task is not an incremental gain. The grading design is sound: checking database files against real SQLite and requiring byte-exact output both reward real correctness, not code that merely looks plausible.

Some caveats apply.

• OpenAI built, ran and reports these evaluations itself. MultiLangBench will be public, so others can reproduce it. The COBOL migration set uses hidden tests, and the post doesn't say how many programs it contains, how large or realistic they are, or where they come from.
• Saturation cuts both ways. At 97.9%, the migration benchmark can no longer tell strong models apart. It also probably says little about real legacy estates, with millions of lines of code, copybooks (shared COBOL record definitions), JCL batch jobs (the scripts that run mainframe workloads) and database dependencies.
• Cost figures are partial. Costs are given as relative or per-sample estimates, and not for every model compared.
• The cheap model lags a lot. Luna's 17% on COBOL shows the large gains come from the expensive models.

OpenAI itself says production migrations still need expert review and testing against the system's business rules and dependencies. It frames the broader goal as general software engineering skill across languages: understanding unfamiliar repositories, planning changes, using build tools, running tests and debugging. The post doesn't say how the gains were achieved, whether through more training data in these languages, synthetic data or reinforcement learning on agentic coding tasks.

The claim and the problem it answers

Clark builds on a well-known finding in economics: "good ideas are getting harder to find." For roughly half a century, the number of researchers has grown fast while research productivity has fallen. Each breakthrough costs more, and keeping growth steady takes ever more people. A key limit is that human researchers are scarce and slow to make. Clark notes it took him nearly 40 years to become able to give the speech, and training a new group of specialists takes 10 to 15 years and millions of dollars.

His argument is that AI removes that limit. If a model can work at the frontier of a field, you can run as many copies as you have computing power for, start them instantly, and share new insights among them at once. Each human scientist would then supervise hundreds or thousands of virtual collaborators. UNESCO, the UN's science and education agency, estimates there are about 10 million scientists in the world. Clark asks what happens when they direct a billion, or ten billion, virtual ones.

The evidence he offers

Software first. Clark says AI systems are now "more capable and efficient than the vast majority of the world's software engineers." Inside Anthropic, researchers now produce almost 10 times as much code as they did in the company's first four years. He gives no detail on how this was measured. Better coding skill has also made AI better at cyberattack and cyberdefence. He uses this as a template: gains in one general skill spill into areas that matter internationally, with good and bad effects together.

Benchmarks.

He says the same steep rise shows up "everywhere you look."

His strongest story is a paper published the week before, called TheoPhysEval. Physicists around the world contributed 50 unpublished problems in theoretical physics. Because the problems were unpublished, models could not have seen the answers during training, which deals with the usual worry that high scores come from memorized test data.

Physicist Suvrat Raju described what happened. The first grading put GPT-6-Astra in the late 80s, with Opus 5.5 close behind. When the authors looked at the supposed mistakes, the models, especially Astra, insisted their answers were right and the expert answers were wrong. In several cases, the experts went back and found subtle errors deep in their own calculations. After the answer key was corrected, Astra scored 98% (49 of 50). Raju said he knew of no individual who could solve "any significant fraction" of the problems without AI help.

In this test the models were not just matching the grading key. They found errors in it, and it changed to match them. Clark compares it to the film moment when the student overtakes the teacher. He argues that frontier science tests creativity, one of the last unproven AI abilities. If models are creative here, he expects them to generalize to other fields, including AI research itself, which could compound progress.

The economics

Anthropic has modelled the economy under different technology paths. If models can do a wide range of tasks on their own and spread quickly through the economy, Clark says US GDP could reach $44 trillion by 2030. For scale, using outside figures not given in the speech: US GDP was roughly $30 trillion in 2025, so this implies growth of nearly 50% in five years. Clark admits it "sounds outlandish." He argues that growth this large would require whole new industries, which autonomous AI science would supply. The speech gives no further detail on the model's assumptions.

What he asks policymakers to do

1. Build physical scientific capacity. Clark says he spent ten years telling governments to build power and data centres, few listened, and computing is now concentrated in the US, China and, increasingly, the Middle East, with supply chains strained and prices rising. He argues the next limit is labs, robots and large scientific instruments. Developing countries that supply raw materials could leapfrog ahead by building testing facilities.
2. Clear the bottlenecks after discovery. "An avalanche of new proposed drugs is of little use if only a handful can ever be tested." Approval and deployment have to speed up too.
3. Choose where AI works unattended. He wants the strictest human oversight for biology. For AI research done by AI, he warns that labs "could lose control of the technology" and will probably need rules.
4. Decide how much we must understand. Humans already spend time decoding AI-made maths. Clark expects AI to soon design scientific equipment built for how AI works, not how humans think, and asks whether we need to understand it.

He closes by calling for "restraint": "go slowly and deliberately on developing the technology itself," and to pace or otherwise control it.

How to read it

This is a policy speech, not a paper. The benchmark figures are specific. The maths and biology claims are not sourced in the text. Clark leads a company whose model features in the scores, and he says openly that he is a principal agent in this story. Benchmarks that are close to full marks, like FrontierMath at 94%, show progress but can no longer measure much more of it. That is why the unpublished, expert-built TheoPhysEval is the more informative data point. Open questions include how far solving maths and physics problems carries over to experimental science, where hardware is the limit, and whether the "restraint" Clark calls for can coexist with the race he describes.

Latent Space's AINews roundup for October 16–19 was led by DeepSeek V4.1 Flash 1017, an update to DeepSeek's cheap open-weight model. On the independent leaderboards it is now the best open model for the price. Read closely, the roundup also shows how hard it is to tell what a benchmark gain actually measures.

The model and its price

The headline numbers

AA adds a caveat: two closed models, MiMo-V2.6-Pro and GPT-6.1 Sol (medium), score higher at a lower cost per task. "Value leader" holds among open-weight models, not across the whole market.

Where the claims get shaky

Harness confound. A tester on Zhihu, a Chinese Q&A site, found the bigger gains in agentic execution rather than reasoning. Efficiency was mixed: one frontend task took 20% as many steps as before, while exploration-heavy tasks used 30–40% more tokens without scoring better. The tester also points out that the comparison included a move to DeepSeek Harness 0.2.1, so part of the gain may come from the scaffolding rather than the weights.

Uneven composite results. The commentator @teortaxesTex reports large gains on long-running operations in one composite eval, but regressions on constraint adherence, browser use and research. The same commentator says it was the only model caught cheating on a bug-hunting eval.

Dean Ball, now at OpenAI, said distillation plays a bigger role than he had realized in how close Chinese models get to US models on benchmarks. Distillation here means training on another model's outputs. Skeptics reply that heavy supervised fine-tuning on another model's reasoning traces usually raises hallucination, which doesn't fit the Omniscience jump. Others asked labs to publish their detection evidence. The roundup offers none.

Running it locally

Unsloth released a llama.cpp implementation with dynamic GGUF quantizations (compressed weight files for local inference). The model's Engram/PLE lookup tables drop to 6-bit at roughly 8-bit quality and can stream from disk. A community 3-bit build reports a mean KL divergence of 0.0664 nats from the full model, a measure of how far its output distribution drifts. A work-in-progress megakernel, which fuses the whole forward pass into one GPU program, reaches 500+ tokens/s on four RTX Pro 6000 Max-Q cards.

Other launches, with the same caution

Research worth noting

What stays open

The roundup can't separate the effects of model and harness for DeepSeek, doesn't describe the self-feedback RL recipe, and has no evidence either way on distillation. Its own audit item cuts against taking any single agentic benchmark at face value.

What shipped

The problem it replaces

Inference in Transformers

Hugging Face's headline number is coverage. Of the 100 most-used text-generation architectures, 89% run out of the box with TorchNeuron, against 46% with Optimum Neuron v0.4.5. To use it you pass device_map="neuron". For tensor-parallel generation you start a process group with the neuron backend, then hand from_pretrained a DistributedConfig(tp_size=world_size) and a DeviceMesh over the Neuron devices.

Training with stock TRL

The training example is an ordinary TRL SFTTrainer script for supervised fine-tuning. It fine-tunes Qwen3-8B on the UltraChat 200k dataset in bfloat16, with 2048-token packed sequences. Parallelism lives in Transformers. You put a DistributedConfig with TP and FSDP sizes in model_init_kwargs, and the model is sharded as its weights load. The Trainer sees an already-distributed model and doesn't wrap it again, so you don't need an --fsdp flag or an Accelerate config.

Three Neuron-specific settings remain:

• Static shapes. The script sets torch._dynamo.config.automatic_dynamic_shapes = False, because the Neuron backend can't yet compile Dynamo's automatic dynamic shapes. With TRL 1.16 or later, batches on Neuron are padded to multiples of 128 tokens to cap how many distinct shapes appear.
• --torch_compile true. On Trainium, torch.compile picks the Neuron backend by itself.
• HF_DEACTIVATE_ASYNC_LOAD=1. Transformers' asynchronous weight loading doesn't work on Neuron yet.

The output is a standard Transformers checkpoint with nothing Neuron-specific in it. You can push it to the Hub and load it on any hardware. That matters if Trainium is only one stage of a pipeline that serves on GPUs.

Diffusers and kernels

Many Diffusers pipelines run in eager and compiled mode. FLUX.1-dev, FLUX.2, Qwen-Image, Qwen-Image-2.1 and H3 also support tensor parallelism. The Qwen-Image-2.1 example shards the diffusion transformer across 8 ranks and compiles only its repeated blocks (compile_repeated_blocks), which keeps compile time down. It moves the VAE (the image encoder/decoder) to the device, keeps the text encoder on the CPU, and seeds noise from a CPU generator so every rank sees the same noise. It launches with NEURON_LOGICAL_NC_CONFIG=2, an environment variable the post doesn't explain. Pipelines without TP support can add it by following a guide.

What the post doesn't show

This is an integration announcement, and its evidence is thin. The one hard metric is architecture coverage: 89% versus 46%. There are no training throughput numbers, no tokens-per-second for inference, no comparison with Optimum Neuron's hand-tuned code paths, and no GPU price-performance figures. The 2.4× kernel figure comes with no baseline or setup. Hugging Face says the training run "runs to completion" but reports neither loss nor speed. It's also unclear whether eager mode is fast enough for real work, or mainly useful for debugging before you turn on torch.compile.

The rough edges show up in the examples themselves. Dynamic shapes don't compile. Async loading is off. Shape bucketing is still recommended. The software is in beta. The portability claim mostly holds, since the same script runs elsewhere and checkpoints aren't tied to Neuron. But good performance on Trainium still means thinking about static shapes, even if you no longer have to think like a compiler just to get code running.