DeepSeek open-sources its serving stack as both governments tighten open-model rules, while Anthropic and IBM publish rare end-to-end measurements of agents doing real work.
Picked from 24 new articles across 24 sources · arrow keys for other days
Today in 30 seconds
30s left
DeepSeek released its inference stack and V4.1 Flash; Reflection's Beam is now the strongest permissively licensed US open model. story 1 ↓
Licensing and model provenance, not just benchmarks, now decide which open model you can ship, as Washington and Beijing write rules. story 1 ↓
An Anthropic model ran a 64-hour protein design campaign: 318,484 candidates, 211 tested, 17 confirmed binders, zero multi-toxin hits. story 2 ↓
IBM's ALTK-Evolve mines prompt rules from its own practice runs: +10 points retail reliability on τ²-bench, up to 65% more cost per conversation. story 3 ↓
Mollick counts 180 models beating launch-day GPT-4, 19 phone-sized, and METR's task-length benchmark now out of headroom. story 4 ↓
Jane Street and NVIDIA rewrote Mamba-2's forward pass in Python cuTile as three tiled kernels, up to 1.82x over Triton on one B300. story 5 ↓
1/5Interconnects (Nathan Lambert)· Florian Brand· Oct 20
DeepSeek open-sources its inference stack as Washington and Beijing tighten rules on open models
DeepSeek published DeepInfer, the software it uses internally to run its models, plus a cheap new model. Meanwhile the US barred DeepSeek from federal civilian purchasing, and China proposed pre-approval for exporting frontier AI.
Why it matters · If you pick open models for products, you now have to weigh licensing, where a model comes from, and serving cost alongside benchmark scores.
Know this first
Context from the editor, not the article · 27 sec
1 · Open-weight models
Open-weight models publish their trained parameters so anyone can download, run and fine-tune them. Chinese labs such as DeepSeek, Qwen, Kimi and Zhipu have led this space, which has turned licensing and national origin into policy questions.
2 · Total versus active parameters
Names like 552B-A16B describe mixture-of-experts models. The model holds 552 billion parameters in total, but each token passes through only about 16 billion of them, which keeps inference cheap relative to the model's size.
3 · Inference engines
Serving a model efficiently takes specialized software that batches requests, manages GPU memory and runs optimized kernels. vLLM and SGLang are the main open-source engines. Labs usually keep their faster in-house engines private.
2/5Anthropic Research· Oct 20
Claude ran a 64-hour protein design campaign; 17 binders confirmed in lab
Anthropic gave an internal Claude model one prompt plus four follow-ups to design proteins binding scorpion neurotoxins. It ran 64 hours mostly autonomously; outside labs tested 211 designs and confirmed 17 binders.
Why it matters · A concrete, lab-checked measurement of how much of a real scientific campaign a model can run end to end — including where it plainly failed.
Know this first
Context from the editor, not the article · 25 sec
1 · Binders and antivenom
A binder is a molecule that latches onto a specific target. Antibodies are the natural version; today's antivenoms are animal antibodies, slow and costly to make and effective against only a few toxins each.
2 · Designing proteins from scratch
Machine-learning tools can now generate novel protein shapes against a chosen target and predict whether a candidate and target fold together. Most candidates fail, so campaigns generate many and filter computationally before lab testing.
3 · Earlier Claude binder work
In August, Anthropic reported that Claude designed protein minibinders from a single prompt. This campaign reuses that prompt, modified, against a harder and less-studied set of targets.
3/5Hugging Face· Oct 20
IBM's ALTK-Evolve tops τ²-bench retail using GPT-4.1 and learned prompt rules
IBM Research had GPT-4.1 turn its own practice runs into short rules for its system prompt. With those rules, a basic customer-service agent beat every retail entry on the τ²-bench leaderboard. No retraining was involved.
Why it matters · If you run an agent with policies to follow, this is a cheap way to make it more reliable: rules mined automatically from logs, which you can read and audit, with no fine-tuning.
Know this first
Context from the editor, not the article · 29 sec
1 · What τ²-bench measures
τ²-bench, a benchmark from Sierra, puts an agent in a conversation with an LLM playing a customer. The agent has domain tools and a written policy. A task counts only if the final database state is exactly right.
2 · The ReAct agent loop
ReAct is a common, simple agent design. The model reasons about the next step, calls a tool or replies, reads the result, and repeats. τ²-bench ships a stock version that many leaderboard entries use.
3 · Memory instead of fine-tuning
There are two ways to improve an agent. You can update the model's weights by fine-tuning, which is costly and tied to one domain. Or you can give it better context, such as prompts or retrieved notes, at inference time.
4/5One Useful Thing (Ethan Mollick)· Ethan Mollick· Oct 20
Mollick charts 30 months of AI progress since GPT-4 led the field
Ethan Mollick, launching his second AI book, charts the 30 months since his first. AI tasks grew from 4 minutes to 17 hours of human work, and costs fell about 750-fold.
Why it matters · It puts the past 30 months of capability and cost data in one place, and it argues that agents now change how engineers should divide work with AI.
Know this first
Context from the editor, not the article · 30 sec
1 · METR's time-horizon benchmark
METR is an AI evaluation group. Its benchmark asks how long a task, measured in human working time, a model can finish with at least a 50% success rate. A 50% rate means it succeeds about half the time, not that it is reliable.
2 · GPQA
GPQA, the 'Google-Proof Q&A' benchmark, is a set of graduate-level science questions written so that searching the web doesn't help. Expert grad students often get them wrong. It is widely used to measure reasoning.
3 · The jagged frontier
This is Mollick's earlier idea that AI ability is uneven. Models can be superhuman at some tasks and weak at others that look similar, so it is hard to guess in advance where they will fail.
5/5NVIDIA Technical Blog· Tanya Lenz· Oct 19
Mamba-2 forward pass written in NVIDIA's Python cuTile runs up to 1.82x faster than Triton
Jane Street engineers and NVIDIA wrote the forward pass of the Mamba-2 sequence model in cuTile, NVIDIA's Python language for GPU kernels. On a B300 GPU it ran 1.14x to 1.82x faster than the standard Triton version.
Why it matters · The post shows step by step how a recurrent model gets rewritten as matrix multiplies, and it tests whether a Python tile language can beat Triton on new hardware.
Know this first
Context from the editor, not the article · 30 sec
1 · Attention versus recurrence
Softmax attention trains fast because it is mostly matrix multiplies, but the memory it needs at inference grows with sequence length. RNNs (recurrent neural networks) use constant memory per token but train slowly, because each step waits on the one before.
2 · State space models like Mamba
Mamba-2 belongs to a family of state space models. Like an RNN, it carries a fixed-size hidden state forward token by token. The recurrence is linear, so it can be rearranged into matrix multiplies for fast parallel training.
Xiaomi MiMo-V2.6-Pro, which the article places as Artificial Analysis's top open model after scaled-up RL training.1 / 4
DeepSeek has open-sourced DeepInfer, the software it uses internally to serve its models, in the same week the US and Chinese governments both moved toward controls on 1 models. That is the lead of Interconnects' latest open-model roundup, written by Florian Brand.
Two governments, two moves
China's Ministry of Commerce (2) proposed that frontier AI and some other technologies need government pre-approval before being shared abroad. It is a proposal, not a law. The authors call the reaction overblown, because the document promises automatic approval for open source. Others argue it creates a framework that could later be used against China's AI industry, or that it mainly targets deals like the sale of the startup Manus to Western buyers.
In the US, the Department of Homeland Security barred DeepSeek, both the company and its models, from federal civilian procurement. The authors ask whether this reaches fine-tunes or models trained on DeepSeek outputs. NVIDIA's Nemotron models, for example, have long relied heavily on DeepSeek in training. They see the order as reinforcing the need for a domestic US open-model ecosystem.
Why DeepInfer matters
Top labs are known to run inference engines far more efficient than the open-source 3 and SGLang, and none had released one until now. Estimates the article cites put DeepSeek's gross margin on token sales at up to 90% with this stack. The authors expect inference providers to adopt it fast, which would push down DeepSeek model prices across the ecosystem or widen providers' margins. They give no benchmarks for DeepInfer itself.
Models worth knowing
The available text ends at "Read more", and Kimi K3.1 is rumored to be close.
The 25th "Latest open artifacts" roundup from Interconnects, written by Florian Brand, opens with policy before it gets to models. In one week the US and Chinese governments both moved toward controlling 1 models, which are models whose trained parameters are published for anyone to download. DeepSeek answered with a release that may matter more than any single model: its internal inference software.
The regulatory moves
China.2, China's Ministry of Commerce, released a policy proposal. It is not yet law. It would require frontier AI models and some other technologies to get government pre-approval before they can be shared internationally. The authors think the reaction has been overblown, because the document says open-source technologies would be approved automatically. Geopolitics watchers counter that the proposal still sets up a regulatory framework that could later be used against China's AI industry. Other sources the authors trust read it as aimed at deals like the sale of the Chinese startup Manus to Western companies, not at frontier models.
United States. The Department of Homeland Security (DHS) issued an order that bars DeepSeek, both the company and its models as a technology platform, from being a vendor in federal civilian procurement. The order leaves an open question: what happens to fine-tuned derivatives, or to models trained on DeepSeek outputs? The authors point to NVIDIA's Nemotron models, whose training recipes have long relied heavily on DeepSeek models, and ask whether those could be caught too. They call the order expected, and say it strengthens the case for a domestic US open-model ecosystem. They also note that Zhipu AI, maker of the GLM models, is already on the Commerce Department's 3, which restricts its access to American technology.
DeepInfer: a frontier lab's serving stack, made public
Soon after the DHS order, DeepSeek released DeepInfer, the inference stack it uses internally. It is the software that turns model weights into tokens served at scale. The authors expect it to be far more efficient for DeepSeek's models than the open-source engines 4 and SGLang. They cite estimates, without saying whose, that DeepSeek earns up to 90% gross margin selling tokens on this stack.
Why it matters, in the authors' view: everyone knows the top labs run inference tools far more efficient than what is public, and this is the first time a leading lab has shared one. Their prediction is that inference providers will adopt it quickly. That should either push down prices for DeepSeek models across the ecosystem or raise providers' margins. The authors count it among the biggest open-source AI releases of the past few months, even though it is neither a model nor a dataset. The article gives no benchmarks for DeepInfer itself, so the efficiency claim is still an expectation.
The notable models
The roundup also lists:
Ternary-Bonsai-2-27B: a ternary version of Qwen3.8-27B for on-device use, with weights restricted to −1, 0 or +1.
Ornith-1.5-397B: Qwen3.5 and Gemma 4 fine-tunes that beat their strong baselines and are popular with people running models locally.
Ling-3.0-flash-VL: Ant Group's 124B-A5.5B model in FP4.
Muse-Spark-Open-816B-A42B: Meta's open version of Muse.
Kimi K3.1 is rumored to be coming soon, since its metadata has started appearing in public libraries.
What's open
The available text breaks off at "Read more", so the rest of the list is missing. Key questions are unanswered. Does the DHS order reach derivatives and distilled models? Would MOFCOM's automatic approval for open source survive into the final rule? Does DeepInfer actually beat vLLM and SGLang on hardware outside DeepSeek's own setup? The 90% margin figure is an unsourced estimate.
Remember
DeepInfer is the first public release of a leading lab's internal inference engine, though its efficiency claims are untested.
DeepSeek's US federal ban leaves open whether derivative or distilled models, such as NVIDIA's Nemotron, are also covered.
Reflection's Beam stands out for token efficiency, not peak scores, which the authors read as a sign of mature post-training.
How the model picked target structures and the surface positions its designs should aim at.1 / 2
Anthropic has published the results of an experiment in which an internal research model, run inside a system the company calls 1, was pointed at a drug-discovery problem and left largely to itself: design small proteins ("2") that stick tightly to scorpion neurotoxins, as a first step toward a cheaper antivenom. The question came from Citrini Research, a financial research firm, which asked whether motivated nonexperts plus AI could take on a real health problem. Scorpion stings injure 1.2 million people a year and kill more than 3,250, mostly children, and attract little commercial investment.
What the model did
The model got one initial prompt defining the targets, the available design and structure-prediction tools, and the rules for scoring and filtering, plus four follow-ups over 64 hours. Within that frame it chose which published toxin structures to design against, picked the specific surface positions to aim at, set its own quality thresholds, decided how heavily to use each of 12 design tools, and invented its strategies for the hardest variant of the task: one binder that hits several toxins at once.
The scale is the striking part. It generated 318,484 candidate designs, screened 553,052 design-plus-toxin pairs by predicting each pair's joint structure, and used roughly 10,000 GPU hours before narrowing to a final list of 180. The team added 31 more, bringing the total to 211. The model also analyzed the resulting data, produced the figures, drafted the report, and emailed sequences to a manufacturing partner — with the emails flagged as written by Claude.
What the lab found
Two 3, Adaptyv Bio and Twist Bioscience, made and tested all 211 designs independently, in different expression formats, measuring binding strength by 4. Anthropic's bar was strict: binding in every replicate at both labs, a 5 of 1 µM or tighter, and no signal against an unrelated human protein used as a stickiness control.
Seventeen designs cleared it — seven against the Mexican toxin Cn2, four each against two North African toxins, two against a Middle Eastern one. Among the 60 single-toxin designs, 11 hit, an 18% success rate the authors call typical for protein design campaigns. The two strongest, Cn2-B1 and Cn2-B2, bound in the single-digit nanomolar range and, in competition tests, appear to occupy roughly the same site as an antibody already shown to neutralize Cn2 in mice.
The multitoxin ambition failed outright: not one design met the bar against more than a single toxin. Anthropic lists several possible causes — its own prompting, tool limitations, or the biology simply not permitting it.
The honest caveat
Nothing here is an antivenom. Binding was shown only against lab-made toxins, with no test of whether the binders block toxin function in cells or protect animals. Existing antibodies against these toxins already bind more tightly and do protect mice. All 211 designs and their data are being released.
Anthropic has released results from a drug-discovery experiment that is mostly interesting as an agent-capability result. An internal research model, operating inside a setup the company calls 1, was given one prompt and four follow-ups and spent 64 hours largely on its own running a protein-design campaign: produce small proteins that bind tightly to scorpion neurotoxins. Two 2 then made and tested 211 of the resulting designs in the lab. Seventeen met Anthropic's definition of a binder. All 211 designs and their measured data are being published.
Where the problem came from
The prompt was not Anthropic's idea alone. Citrini Research, a financial investment research firm, approached the company with a question: could AI let a group of motivated nonexperts make real progress on a genuine human-health problem? Scorpion envenomation fit. Stings affect at least 1.2 million people a year and cause more than 3,250 deaths, concentrated among children in regions with limited medical access. Existing antivenoms are made by milking venom from thousands of scorpions, immunizing horses, and purifying their antibodies — a process that takes months, covers only a narrow set of toxins, and is expensive enough that the field is described as a neglected tropical disease. There is little commercial reason to develop anything better.
3 — computationally generating a protein from scratch against a chosen target — offers a different route, because a designed protein can in principle be optimized for stability and manufacturability rather than inherited from an animal's immune system. The work follows an earlier academic effort that designed proteins neutralizing snake neurotoxins and protecting mice. Anthropic says no de novo binders against scorpion toxins had been described before.
How much the model decided
The initial prompt defined the targets, the design and structure-prediction tools available, and the rules for novelty, scoring, and filtering. Inside those bounds, the model made the choices a human campaign lead would normally make. It searched the public 4 and the 5 to pick which toxin structures to design against. It chose five specific surface positions per toxin as the places its designs should latch onto. It set the quality thresholds a design had to clear. It decided how heavily to lean on each of 12 different design tools. And it developed the strategies for the campaign's harder variants.
Those variants matter for the stated goal. Anthropic asked for binders against each of five toxins individually, and then for binders that hit two related toxins, three related toxins, or all five at once. A single molecule covering multiple toxins would shrink the number of components in a universal antivenom cocktail — the cheapest, most regionally neutral form such a treatment could take. The five chosen toxins made this a deliberately hard test: they share the same overall fold but are only 26% to 62% identical in sequence.
Scale: 318,484 designs generated, with anything resembling a known protein or a toxin itself filtered out; 553,052 design–toxin pairs screened by predicting each as a single folded complex; roughly 10,000 GPU hours. After 24 hours of autonomous work the first results were good enough that the team told the model to keep going. Most of the final 180 designs — including the eventual best binder — were selected only after the third follow-up. The team then added 31 more with the model's help, for 211 total: 60 single-toxin (12 per toxin), 30 against one toxin family, 30 against the other, and 91 against all five. The model also analyzed the data, generated the figures, drafted the final report, and sent sequences to the manufacturer by email, with the emails marked as written by Claude.
The measurement, and the bar
Adaptyv Bio and Twist Bioscience each produced the designs in a different format — one cell-free, one in mammalian cells — as deliberately complementary readouts of the same underlying question. Both measured binding by 6 (SPR), which tracks in real time how fast a target attaches to and falls off a candidate, yielding a 7: lower means tighter.
Anthropic's criteria were strict and worth noting, because loose criteria are how design campaigns flatter themselves. A design counted as a binder only if it hit at both labs, showed binding in every replicate, had every KD at 1 µM or tighter, and showed no signal against CLEC12A, an unrelated human protein used as an off-target "stickiness" control.
Results
Seventeen designs passed, each against a single toxin: seven against Cn2 (Mexico), four against AaH II and four against AaH I (North Africa), two against LqhIII (Middle East). Of the 60 single-toxin designs, 11 were binders — an 18% hit rate the authors describe as consistent with typical protein design campaigns, including their own earlier work.
The two best, Cn2-B1 and Cn2-B2, bound in the single-digit nanomolar range at both labs. A competition experiment — testing whether two molecules can occupy a target simultaneously — suggests both sit on or near the same 8 as LR, an antibody reported to neutralize Cn2 in mice. That epitope covers the region the toxin uses to act on nerve-cell sodium channels, which is the mechanistic reason the authors think neutralization is plausible.
The multitoxin effort failed cleanly: none of the 151 multitoxin designs met the bar against more than one toxin. Anthropic offers no single explanation and lists candidates — the model's current capabilities, its own prompting, the limits of available design tools, or the biological feasibility of the idea at all.
What it doesn't show
Binding was measured against recombinant toxins grown from genes in the lab, not toxin purified from venom. Nothing was tested for neutralization in cells or for protection in animals, and Anthropic explicitly does not claim an antivenom. For calibration, three existing antibodies and nanobodies against Cn2 or AaH II report sub-nanomolar affinities and already protect mice — so the designed binders are not yet better than the incumbents on either axis.
Two outside reviewers, Susana Vázquez Torres (CNIO) and Timothy P. Jenkins (Technical University of Denmark), called it a promising first step. Vázquez Torres' framing is the sharpest line in the piece: binding itself is no longer the main bottleneck, and the real work is everything downstream — safety, stability, affordability, and getting molecules to the places where stings happen.
Remember
Strict criteria: hits required at two independent labs, every replicate, and a clean negative control against an unrelated protein.
The 18% single-target hit rate matches ordinary human-run campaigns — the novelty is autonomy and speed, not better odds.
Every multi-target design failed, and Anthropic says it cannot tell whether that's the model, the tools, or the biology.
Mean accuracy and Pass^4 by rule configuration for retail and airline, GPT-4.1 agent on held-out tasks.1 / 3
IBM Research reports that a basic customer-service agent on GPT-4.1, an older model that doesn't do extended reasoning, scored 67.5% 1 on the retail domain of 2. Pass^4 is the share of tasks the agent solves on all four of four attempts. That beats every retail entry on the October 12, 2026 leaderboard snapshot, including models fine-tuned for the benchmark. The model's weights never changed. The only addition was a list of rules appended to its system prompt.
The benchmark
τ²-bench, from Sierra, pairs an agent with an LLM-simulated customer, a set of tools (look up an order, issue a refund, rebook a flight) and a written policy. A task counts as solved only if the final database state matches the expected one. IBM used the retail and airline domains and the benchmark's stock 3 agent, which runs a think-then-act loop.
How the playbook is built
4, IBM's open-source agent-memory library, runs the agent on training tasks. It then has the same model, GPT-4.1, condense the successes and failures into short behavioural rules. Examples: authenticate the user before any order action, and if email isn't available, look them up by name and zip code. Summarise every change and get explicit approval before writing to the database. Never combine a tool call and a message to the user in one turn. No humans annotate anything. The rules are then applied unchanged to held-out test tasks.
Each rule has a 5: how many distinct training tasks it was extracted from. IBM tested two settings. "sup3" keeps only rules with support of 3 or more (31 retail rules, 14 airline). "Lossless" keeps everything (292 retail, 101 airline).
Results
Retail (40 test tasks): lossless raised mean accuracy from 78.1% to 82.5% and Pass^4 from 57.5% to 67.5%.
Airline (20 tasks): lossless gave the best accuracy (65.0%, up from 57.5%). The smaller sup3 set was more reliable: Pass^4 rose from 35% to 45%, versus 40% for lossless. How many rules to keep is a setting to tune per domain.
GPT-5.5 on airline: with rules mined from its own runs, Pass^4 rose from 55% to 70%. On the leaderboard's full task set it scored 82.0%, against 66–72% for frontier entries.
The cost is in input tokens. Cost per conversation rose 65% in retail ($0.084 to $0.139) and 32% in airline. Conversations got slightly shorter, by about 0.3–0.4 turns. IBM notes that 6 (reusing an unchanged prompt prefix so it isn't billed in full on every call) could recover much of the extra cost.
Caveats
The test sets are small. In airline, one task moves Pass^4 by 5 points. The leaderboard figures are scored on the full task set, which includes the training tasks the rules were mined from. The post doesn't say whether its GPT-5.5 customer simulator matches what other leaderboard entries used.
IBM Research has published results for 1, its open-source library for giving agents memory, on 2, a customer-service agent benchmark. The headline: a basic agent running GPT-4.1, an older model that doesn't do extended reasoning, scored 67.5% 3 on the retail domain. That puts it above every retail entry in IBM's October 12, 2026 leaderboard snapshot. It also beats RAFT-30B-A3B (61.4%), a model fine-tuned on synthetic τ²-bench retail data. No weights were updated. The only change was a block of learned rules appended to the system prompt.
The problem
τ²-bench, from Sierra, tests agents in policy-bound conversations. Each task gives the agent three things:
an LLM-simulated customer with a hidden goal
domain tools, such as looking up orders, issuing refunds or rebooking flights
a written policy document
The agent has to draw out the right details, stay within policy and take the correct actions, many of which can't be undone. The reward is strictly binary: the task counts only if the final database state matches the expected outcome.
This is hard because one mistake sinks the whole task. Examples include skipping authentication, acting before the user confirms, or bundling a tool call with a reply. These are the kinds of mistakes models make inconsistently, so an agent can succeed on a task one run and fail the next.
IBM tracks two metrics over four trials per task:
Mean accuracy: the average success rate across trials. This measures capability.
Pass^4: the share of tasks solved in all four trials. This measures reliability, and it is what the leaderboard reports.
IBM's framing: a support agent that gets a refund right three times out of four still fails a quarter of customers.
How ALTK-Evolve works
The setup is deliberately stock. IBM uses τ²-bench's own 4 agent, a two-phase THINK → ACT loop. GPT-4.1 is the agent, GPT-5.5 plays the customer, and policy, tools and scaffold are all unchanged.
ALTK-Evolve runs the agent on the training tasks and collects the 5 (full logs of each conversation and its tool calls). The same GPT-4.1 model then distils successes and failures into short rules. Duplicates are merged, and the result is appended after the policy as a markdown list headed "Guidelines learned from past task attempts." Real examples from the retail playbook:
Authenticate before any order action. If email is unavailable, use find_user_id_by_name_zip with full name and zip code.
Present a structured summary of every change and require explicit approval before any action that writes to the database.
When handling a return or exchange, give an itemised refund breakdown and explain refund timing without waiting to be asked.
Take one action per step. Never pair a tool call with a user-facing message, and wait for the tool's result before replying.
Clean up names before a lookup. For example, split a handle like daiki_johnson_9523 into first and last name. After a couple of failed lookups, state the exact format needed.
There is no human annotation. The rules are applied unchanged to held-out test tasks.
Each rule has a 6, the number of distinct training tasks it was extracted from. High support suggests a rule that generalises rather than a one-off. IBM compares three configurations:
Variant
What's included
Retail rules
Airline rules
baseline
none
0
0
sup3
support ≥ 3
31
14
lossless
all rules
292
101
Results on held-out tasks (GPT-4.1)
Domain
Config
Mean accuracy
Pass^4
Retail (40)
baseline
78.1%
57.5%
sup3
78.8%
60.0%
lossless
82.5%
67.5%
Airline (20)
baseline
57.5%
35.0%
sup3
60.0%
45.0%
lossless
65.0%
40.0%
The two domains disagree on how many rules to keep. In retail, more is better: the small sup3 core barely moves accuracy, and the full 292-rule set delivers the jump. In airline, lossless wins on accuracy but the compact 14-rule core wins on reliability. IBM's conclusion is that support count is a setting to tune per domain, not hard-code. This echoes the team's earlier post, which found that different models and tasks need different amounts of guidance.
Cost and turns
The rules cost money because they make the prompt longer. Total cost per conversation, agent plus simulator:
Retail: $0.084 baseline, $0.095 with sup3 (+12%), $0.139 with lossless (+65%)
Airline: $0.081 baseline, $0.097 with sup3 (+19%), $0.107 with lossless (+32%)
To check whether the agent wins by talking longer, IBM compared only matched runs: the same scenario and seed, solved under both baseline and the rule-augmented setting. Lossless rules cut about 0.3–0.4 assistant turns (retail 10.59 → 10.16 over 111 runs, airline 9.15 → 8.82 over 40). No variant made conversations longer. Each turn is a sequential LLM call, so fewer turns also means lower latency. IBM notes that because the rule block is identical on every call, 7 could recover much of the extra input cost.
Does it help a frontier model?
One objection is that learned rules only patch the gaps of a weaker model. IBM reran airline with GPT-5.5 as both agent and rule extractor, re-mining rules from its own runs. No rule reached support 3, so the compact core became sup2 (103 rules), alongside lossless (146).
Airline · GPT-5.5 (20 test tasks)
Accuracy
Pass^4
baseline
72.5%
55.0%
sup2
80.0%
70.0%
lossless
82.5%
70.0%
The reliability gain (+15 points) was larger than GPT-4.1's on the same tasks, starting from a higher baseline. IBM also reports that the baseline's Pass^4 varied between reruns, while lossless reproduced exactly.
Leaderboard comparison
The leaderboard scores submissions on the full "base" split (training plus test tasks) at Pass^4, so these numbers differ from the held-out figures above:
Retail: 67.5% with GPT-4.1, ranked #1.
Airline with GPT-4.1: the sup3 core scores 44%, 8th place. That beats GPT-5.2 without reasoning and nearly matches Claude Sonnet 4.5.
Airline with GPT-5.5: 82.0%, above every frontier entry (66–72%).
Caveats
The evidence is real but thinner than the leaderboard framing suggests:
Small samples. There are 40 retail and 20 airline test tasks. In airline, a single task is worth 5 points of Pass^4, so the 10- and 15-point gains are two or three tasks.
Training tasks inside the leaderboard score. The base split includes the training tasks the rules were mined from. IBM says the held-out evaluation has no test-set leakage, which holds, but the leaderboard figure is partly scored on tasks the rules were written from. The held-out numbers are the cleaner evidence.
Simulator comparability. The post doesn't say whether its GPT-5.5 customer simulator matches what other leaderboard entries used. That matters because the simulator affects how hard a conversation is.
Unclear mechanism details. The post doesn't explain how rules are merged, or why lossless and sup3 swap places on airline.
The practical lesson holds regardless. Rules mined automatically from an agent's own logs, which a person can read and audit, raised reliability on a stock scaffold in both domains and for two different models. The cost is a longer prompt, not retraining.
Remember
Pass^4, which requires success on every attempt, rose more than average accuracy: rules mainly cut inconsistent mistakes.
The best number of rules varied by domain: 292 rules won in retail, a 14-rule core won airline reliability.
Held-out test sets were small (20–40 tasks), and leaderboard scores include the training tasks the rules came from.
Which company held the best AI model over time: 20 different leaders since GPT-4, all from three labs.1 / 4
Ethan Mollick, the Wharton professor behind the One Useful Thing newsletter, published his second AI book, Co-Existence, today. It comes 30 months after his first, Co-Intelligence (early 2024). He marked the gap with charts of AI progress, and an AI agent built most of them.
How the charts were made
Mollick used Opus 5.5 in 1. There, one "2" agent starts sub-agents, directs them and reports back. From a couple of sentences of direction, the agents did the research, graphs, layouts and citations in minutes. Mollick estimates the same work would have taken him hours. He asked for many options, said "make it better" a lot, and the AI caught his arithmetic error: he had planned to title the post "28 months."
The numbers
Leadership churn: OpenAI's GPT-4 was clearly the best model from early 2023 to early 2024. Since then, 20 models have held the top spot, all from Anthropic, Google or OpenAI.
Breadth: 180 models now beat GPT-4 as it was at launch. Many are 3 models, whose parameters anyone can download and modify, and 19 run on a smartphone. The post doesn't say what benchmark defines "better."
Task length:4, an AI evaluation group, measures how long a human task a model can finish with at least 50% success. That length doubled eight times: from about 4 minutes in 2024 to about 17 hours in April 2026. Then METR said its methods could no longer measure accurately. Unnamed "other studies" suggest 17 weeks or more.
Cost: On 5, a set of graduate-level science questions that web search doesn't help with, one correct answer cost about 30 cents. It now costs about 0.04 cents, roughly 750 times cheaper.
Still uneven
Mollick says the "6" remains. AI is still a poor long-form writer, while math went from terrible to superhuman. He argues that small gaps, even where AI excels, still create big needs for humans.
A new way to work
His first book described two working styles. Centaurs split tasks between human and AI; cyborgs blend the two within each task. Both keep a human in every loop, which doesn't scale. Mollick adds the 7: you call up agents, assign work and judge the results. He calls it the most powerful mode, and warns that you lose control over how the AI gets its results.
As a demo, a casual prompt had Opus 5.5 produce a full "otter on an airplane using wifi" video, with script, images and music. It used Google tools and 8, a voice and audio generator, with no further help. The video had flaws.
Mollick argues that people keep an edge in deep knowledge, wide knowledge, taste and agency.
Ethan Mollick is a Wharton professor who writes the One Useful Thing newsletter. His second AI book, Co-Existence, is out today, 30 months after Co-Intelligence (early 2024). To mark it, he published a set of charts on what changed between the two books. An AI agent built most of those charts.
How the charts were made
Mollick used Opus 5.5 in 1. There he talks to one "2" agent, which starts sub-agents as needed, gives them instructions and passes their results back to him. He wrote a couple of sentences of direction. The agents then did what he estimates would have taken him hours: research, graphing, layout, images and citations, all in minutes. Because the work cost him almost nothing, he asked for many versions and gave vague feedback like "make it better." The AI also corrected his arithmetic. He had planned to call the post "28 months on the exponential." Keep in mind that the agent researched and drew these charts, and the post doesn't say how the data behind each one was checked.
Who leads, and how crowded the field is
From early 2023 to early 2024, OpenAI's GPT-4 was clearly the best model in the world. Since then, Mollick counts 20 different models that have held the top spot at some point, all from three labs: Anthropic, Google and OpenAI.
He argues that the bigger change sits below the frontier. 180 models now beat GPT-4 as it was at release. Many of them are 3 models, whose trained parameters anyone can download, run and modify. Nineteen of them run on a modern smartphone. The post doesn't say which benchmark or index defines "better than GPT-4."
The task-length curve, until it broke
The main chart uses 4's long-horizon benchmark. It measures how much human work, in time, a model can complete from a single prompt with at least a 50% chance of success. Mollick says the gains have been "startlingly regular." From GPT-4's release to the end of the measurement, task length doubled eight times:
2024: about a 4-minute task
April 2026 (when he submitted his draft): about 17 hours
As a check, 17 hours is 1,020 minutes, roughly 255 times 4 minutes, and eight doublings is a factor of 256. So the numbers agree.
Then, Mollick writes, "the benchmark ran out." METR said its current methods could no longer accurately measure how much work a model can do. Mollick cites "other studies" suggesting the figure may be 17 weeks of work or more, but doesn't name them. Treat that number as much weaker evidence than the METR series. Also remember the 50% threshold: a 17-hour horizon means the model finishes such tasks about half the time, not every time.
The cost curve
Prices fell while ability rose. On 5, the graduate-level science questions designed so that web search doesn't help, one correct answer used to cost about 30 cents in model usage. It now costs about 0.04 cents, roughly 750 times cheaper. The post doesn't say which models or dates the two ends of that comparison use.
Still jagged
Mollick stresses that progress has been uneven. Long-form writing is "stubbornly" weak: he calls AI "still quite a bad long-form writer." Math went from terrible to superhuman over the same period. He also argues that even where AI is excellent, small remaining gaps create a large need for humans, "at least for now."
Centaurs, cyborgs, and 6
In Co-Intelligence, Mollick described two ways of working with AI:
Centaur: the work is split, with the human doing some tasks and the AI doing others.
Cyborg: human and AI work are blended within each task.
Both keep a human in the loop, and Mollick notes that "you can only be in so many loops at once." Now that agents work, he adds a third mode, the conjurer. You call up agents, assign them work and judge what comes back. The chart-making was done this way. He calls it the most powerful way to use AI, and he names its risk: you increasingly lose control over what, exactly, the AI did to produce the result.
The otter test
For years, Mollick has tested image and video tools with the prompt "Otter on an Airplane Using Wifi." Thirty months ago, the best result was a single Midjourney still image. This time he gave Opus 5.5 a casual, typo-filled request for "the ultimate" otter video, noting that prompting matters much less now. Working through his subscriptions to Google's tools and 7 (a voice and audio generation service), the model wrote the script, generated images and music, and assembled the video with no further help. Mollick concedes the video "has a few flaws." His point is that the AI handled the whole production pipeline by itself.
What the book argues
Mollick says Co-Existence deliberately doesn't track model releases. It focuses on two trends: AI is improving exponentially but unevenly, and use is shifting from chatbots to agents doing real work. He argues that humans keep an advantage in four areas: deep knowledge, wide knowledge, taste and agency. The book also covers when to hand work to AI and when to avoid it.
What to make of it
This is a book-launch post, not a study. Its strongest evidence is the METR series and the GPQA cost drop, both drawn from outside sources. The weakest numbers are the unnamed "17 weeks" estimate and the unspecified "better than GPT-4" count. For engineers, the open question is the one Mollick raises himself. If measurement saturates and work moves to the conjurer mode, how do you check what agents did when you can no longer reliably measure or watch it?
Remember
METR's time-horizon benchmark saturated around 17 hours of work at 50% success, so its main trend line has ended.
Below the frontier, 180 models now beat launch-day GPT-4, and 19 of them run on phones.
Mollick's 'conjurer' mode delegates to agents for big gains but makes it hard to see how results were produced.
Triton and NVIDIA's cuTile let you write GPU kernels in Python. You work on blocks ('tiles') of data, and the compiler handles threads, memory movement and Tensor Cores, the GPU units built for matrix multiplication.
Jumping the hidden state across a whole chunk with one matrix multiply, B times X-transpose.1 / 4
What was shipped
NVIDIA published a 1 forward pass written in 2 (CUDA Tile), its Python language for GPU kernels. In cuTile you operate on tiles, which are blocks of data, and the compiler takes care of threads and 3. Engineers at Jane Street, a trading firm, wrote the first version, and NVIDIA helped tune it. On one B300 GPU it ran 1.14x to 1.82x faster than the open-source 4 kernels in mamba_ssm, with bigger gains at longer sequences and larger state sizes.
The problem
Mamba-2 carries a fixed-size hidden state, a matrix rather than an RNN's vector, and updates it token by token: h_t = exp(a_t)·h_{t-1} + b_t x_tᵀ. The decay a_t is always negative. Running this as a loop is fine when generating text, but training needs parallelism. A 5 would have to store every intermediate state, which makes it limited by memory bandwidth and unable to use Tensor Cores.
The trick
Cut the sequence into chunks. Stack a chunk's inputs into matrices, and the state at the end of the chunk becomes one matrix multiply: h_{t+T} = h_t + BXᵀ. The outputs inside a chunk come from the chunk's starting state plus a causally masked CBᵀ term. Without the decay, this is 6, meaning attention without the softmax.
Adding the decay back requires log space: running sums G of a_t turn products into differences, exp(g_i − g_j). Some of those differences are positive and could overflow. But the causal mask zeroes out exactly those pairs, so every exponential that survives is at most 1.
Three kernels
chunk_cumsum computes the running decay sums once per chunk. It scales them so later kernels can call the cheaper exp2.
fwd_pass_state loops over chunks one at a time and applies the jump formula on Tensor Cores. Separate batch, head and tile slices run in parallel.
compute_out fuses all the output terms into one kernel. It keeps the T×T CBᵀ matrix in registers and builds the mask on the fly, where plain PyTorch would write the matrix to global memory.
Tuning
The speedups came from four changes:
latency hints on memory operations
swapping matrix-multiply operands, since Blackwell Tensor Cores require operand B to sit in shared memory
changing the memory layout so the GPU's 7 bulk-copy unit could load the data
an autotuner that searches settings separately for each input shape
Caveats
These are kernel-level numbers for the forward pass only. NVIDIA says end-to-end training speed also depends on the backward pass, which it hasn't published yet. The comparison is NVIDIA's own tuned stack on its newest GPU against a general-purpose Triton kernel, and the post shows no accuracy check.
The news
NVIDIA has published the kernels behind a 1 forward pass written in 2 (CUDA Tile), its Python language for GPU programming. In cuTile you write operations on tiles, which are blocks of data, and the compiler handles thread scheduling and Tensor Core use. Machine learning engineers at the trading firm Jane Street, an early cuTile user, wrote the first implementation, and NVIDIA helped tune it. On a single B300 (Blackwell Ultra) GPU, the three kernels ran 1.14x to 1.82x faster than the open-source 3 kernels in mamba_ssm. The gap was widest at longer sequences and larger state sizes. The code ships in TileGym, NVIDIA's kernel collection. A post on the backward pass is promised but not out yet.
The recurrence and why a for-loop won't do
Each Mamba-2 layer carries a hidden state h. Unlike an RNN's state, which is a vector, this one is a matrix of shape (S, H). With per-token inputs x_t (size H, similar to attention's values), B_t (size S, similar to keys) and c_t (size S, similar to queries), plus a per-head step size dt_t, the update is:
h_t = exp(a_t)·h_{t-1} + b_t x_tᵀ and y_t = c_tᵀ h_t + D x_t
Here a_t = A·dt_t is a decay that is always negative, b_t = dt_t·B_t, and A and D are learned parameters. Run as a loop, this is ideal for generating one token at a time. Training and prefill (processing the whole prompt) need parallelism. A 4 would work because the update is linear, but it has to store an enormous tensor of every intermediate state. That makes it limited by memory bandwidth, and it can't use 5.
Turning it into matrix multiplies
The trick is to cut the sequence into chunks of T tokens and jump over each chunk in one step. Ignore the decay for a moment. Then the state after T tokens is the starting state plus a sum of outer products, and stacking the chunk's b vectors into B (T×S) and its x vectors into X (T×H) gives one matrix multiply:
h_{t+T} = h_t + BXᵀ
The outputs inside the chunk can also be computed from the chunk's starting state alone, without building the intermediate states:
Y = C·h_t + ((CBᵀ) ⊙ M)·X + D·X
M is a lower-triangular causal mask, so each token only sees b values from itself and earlier tokens. Without decay, this is just 6, meaning attention with the softmax removed.
To put the decay back, the method works in log space. A running product of decays becomes a running sum G of the a_t values within the chunk. The jump then becomes h_{t+T} = exp(G_last)·h_t + (exp(G_last − G) ⊙ B)ᵀ·X. Inside the chunk, each pair of tokens (i, j) gets weighted by exp(g_i − g_j). That looks dangerous: if i comes before j, the difference is positive and the exponential can overflow. But the causal mask throws away exactly those pairs. Every surviving entry has i ≥ j, and because every a_t is negative, g_i − g_j ≤ 0. Every exponential that is kept is therefore at most 1, so the result stays numerically stable.
Illustrative example: take a chunk with decays a = (−0.1, −0.2, −0.3). The running sums are g = (−0.1, −0.3, −0.6). Token 3 weights token 1's contribution by exp(−0.6 − (−0.1)) = exp(−0.5), about 0.61. The reverse pair, exp(+0.5), gets computed but is masked to zero.
The three kernels
The tuning that mattered
Results and limits
The benchmark ran on one B300 SXM6 (1,920 MHz, 1,100 W) against mamba_ssm 2.3.2.post1 on Triton 3.7.1, using cuda-tile 1.5.0, CUDA 13.3 and PyTorch 2.13. The sample setup uses batch 4, 16,384 tokens, 8 heads, state size 128, head size 64, chunk size 128, and bfloat16 inputs with an FP32 state. Speedups ranged from 1.14x to 1.82x. Per-configuration numbers are only in Figure 5, which the text doesn't break down.
Some caveats. NVIDIA tuned its own stack on its own newest GPU, while the comparison point is a general-purpose Triton kernel, and the post doesn't say whether that kernel was tuned for Blackwell. The figures are for kernels only. NVIDIA itself says end-to-end training speed depends on the backward pass, which hasn't been published. The post also gives no accuracy comparison against the reference beyond the stability argument. Still, the step-by-step derivation, from recurrence to chunked matrix multiplies to a causal mask that makes the exponentials safe, is a clear account of how chunked linear-attention kernels work in general.
Remember
Chunking turns Mamba-2's sequential recurrence into masked matrix multiplies, the same structure as linear attention.
The causal mask also makes the math stable: it discards exactly the exponentials that could overflow.
Hardware details like Blackwell's operand placement and memory layout for TMA drove much of the speedup.
DeepSeek-V4.1-Flash: 552B total parameters, 16B active per token, now accepts images, MIT licensed. It leads the price-versus-performance frontier on DeepSWE, an agentic coding benchmark.
Xiaomi MiMo-V2.6-Pro: the same base models as V2.5 with much more 4. It comes with a tech report and some training environments, and is currently the top open model on Artificial Analysis's rankings.
Reflection Beam-1.0 (501B): Apache 2.0 licensed and, by the authors' account, the strongest permissively licensed American open model. It is not the smartest overall, but it is very token-efficient.
NVIDIA Nemotron-3.5-Super (120B, 4-bit): runs on a strong laptop, with most of its training data released.
Cloudflare clef: a "5", a classifier that outputs a few tokens with probabilities.
DeepSeek-V4.1-Flash-1017. DeepSeek has shipped two iterations since the last roundup, in a new size: 552B total parameters with 16B active per token. It now accepts image input. It ships under the MIT license, which the authors note at a time when licenses are getting more bespoke and exclusionary. On DeepSWE, an agentic software-engineering benchmark, DeepSeek dominates the 5 of performance versus price. That means no other model scores higher at the same or lower cost.
MiMo-V2.6-Pro-RL (Xiaomi). It reuses the MiMo V2.5 base models but spends far more compute on 6 (RL) post-training. Xiaomi published an in-depth tech report and some of its RL environments, which the authors call rare at this scale, along with a live training dashboard. V2.6-Pro is currently the top open model on Artificial Analysis, an independent benchmarking site.
Beam-1.0-501B-A23B (Reflection). It is Apache 2.0 licensed and, by the authors' account, the strongest open American model under a permissive license. It is not the smartest open model overall, but it is very token-efficient: it uses few output tokens to reach its scores, second only to GPT-6-Astra on the chart the article shows. The authors take that as a sign of a mature post-training stack.
Nemotron-3.5-Super-VL-120B-A12B-NVFP4 (NVIDIA). This is the mid-sized Nemotron update. It comes in 7, NVIDIA's 4-bit floating-point format, and is small enough to run on a strong laptop or an RTX Spark. As usual, NVIDIA released most of the training data and a detailed report.
clef (Cloudflare). This is a "8", a category the authors credit to the startup Typesafe. These are general classifiers that output a few tokens together with their probabilities. They have caught on quickly, and clef is one of many open fine-tunes of the idea.
chunk_cumsum computes a[h]·dt[t,h] and its running sum inside each chunk, then stores the result so the next two kernels don't redo it. It multiplies by log₂(e) so later kernels can call the cheaper exp2 instead of exp. The whole kernel is a handful of lines: ct.bid gives the block's position in the launch grid, ct.load and ct.store move tiles with zero padding, and ct.cumsum does the running sum.
fwd_pass_state walks the chunks one after another and applies the jump formula. It stores each chunk's starting state and, at the end, the final state, which can seed the next call during inference. The chunk-to-chunk dependency stays inside one program instance. Different (batch, head, head-tile, state-tile) slices run in parallel across the grid. ct.extract pulls out G_last, and ct.mma runs the multiply-add on the Tensor Cores.
compute_out fuses the three output terms into one kernel. A plain PyTorch version would write the full [batch, chunks, T, T] matrix CBᵀ to global memory. Here it stays in registers. Phase 1 loops over state tiles and builds both CBᵀ and C·hᵀ with ct.mma. Phase 2 applies the pairwise decay, builds the causal mask on the fly with ct.arange and ct.where (no mask tensor is ever stored), adds the decayed carried-state term, and does the last multiply in 7. TF32 is a Tensor Core format with near-FP32 accuracy. The skip term D·x is added at the end.
Latency hints on loads and stores tell the compiler how long DRAM traffic will take, so it can schedule instructions and size its pipeline buffers.
Swapping A and B: on Blackwell's fifth-generation Tensor Cores, the two operands of a matrix multiply are treated differently. Operand A can sit in Tensor Memory, but operand B must sit in shared memory. In fwd_pass_state, the fused operand b_dt_d (B scaled by dt and decay) would otherwise compete for shared memory. The kernel uses the identity BᵀAᵀ = (AB)ᵀ to move it into the A slot.
Memory layout:decay_cumsum and dt were read with strided access, which compiles to cache-unfriendly LDG (plain global-memory load) instructions. Storing them as [batch, head, token] makes the reads contiguous, so the kernel can use 8, the hardware unit that copies bulk data asynchronously.
Autotuning: the bottleneck shifts with input shape. These choices, together with block size and occupancy, form one search space that cuTile's exhaustive_search sweeps separately for each shape.
