Worms, Flows and a Faster Taste Curve · Morning Gist
Monday, October 125 stories · 16 min
Worms, Flows and a Faster Taste Curve
Small models get closer to copying themselves, Claude starts building its own agent teams, and a new benchmark makes AI-research timelines look shorter.
Picked from 39 new articles across 24 sources · arrow keys for other days
Today in 30 seconds
30s left
Anthropic tested whether small models can spread themselves: a 9B model got inference running on 8 of 9 hardware types. story 1 ↓
Only GLM-5.2 finished the full copy-and-spread loop, and only in a simplified test. Running on odd hardware was the hard part. story 1 ↓
NVIDIA and Dao AI Lab's ReplaySSM caches inputs instead of state to speed up hybrid decoding. It's already in vLLM, SGLang and TensorRT-LLM. story 2 ↓
A new benchmark finds AI 'research taste' doubling every three months. Jack Clark now expects recursive self-improvement in 2027, not 2028. story 3 ↓
Anthropic's 'flows' let Claude write its own multi-agent orchestration program. Anthropic says this costs 'much more' than a single agent. story 4 ↓
Microsoft's StudentSim models individual learners. Used as a reward signal, it raised AI chess tutor accuracy to 90.5%. story 5 ↓
1/5Anthropic Research· Oct 12
Anthropic finds small AI models are close to spreading themselves across hacked smart devices
Anthropic researchers ran a 26B model across 15 cheap TV boxes and tested whether small models can copy and run themselves on hijacked devices. They can do most steps now and may manage the whole loop by late 2026.
Why it matters · The hard part of an AI worm turns out to be ordinary agentic engineering work, getting a model running on strange hardware, and small open models are quickly getting good at that.
Know this first
Context from the editor, not the article · 28 sec
1 · Worms and botnets
A worm is malware that copies itself from machine to machine with no human involved. Botnets like Mirai and Kimwolf took over millions of poorly secured smart devices, often just by trying factory-default passwords.
2 · Autonomous replication (ARA)
Autonomous replication and adaptation is a long-standing AI safety worry: a model that deploys copies of itself, keeps them running and improves them, all outside human control. Labs track it as a dangerous-capability threshold.
3 · Splitting a model across machines
A model too big for one device can be quantized, meaning its weights are stored at low precision such as 4 bits, and split by layers across several machines. Each machine computes its layers and passes the result to the next.
2/5NVIDIA Technical Blog· Michelle Horton· Oct 12
ReplaySSM speeds up hybrid-model decoding by caching inputs, not state
NVIDIA and Princeton store state-space-layer memory as a checkpoint plus a short buffer of recent inputs. This roughly halves memory traffic, makes rollback in speculative decoding cheap, and allows 4-bit state reads with little accuracy loss.
Why it matters · Most new open frontier models are hybrids, and this shows that their "cheap" recurrent layers can be the serving bottleneck. That changes where you look for inference speedups.
Know this first
Context from the editor, not the article · 29 sec
1 · Hybrid SSM models
Many recent models mix many state-space model (SSM) layers with a few attention layers. An SSM layer keeps a fixed-size state matrix per sequence, updated at each token, in place of a key-value cache that grows with context length.
2 · Memory-bound decoding
When generating one token at a time, GPUs often spend more time moving data to and from high-bandwidth memory (HBM) than doing arithmetic. Cutting the bytes moved is then the main way to cut latency.
3 · Speculative decoding
A small drafter guesses several next tokens, the big model checks them all in one pass, and generation keeps the accepted prefix. Rejected guesses must be undone. For attention, that just means moving a pointer back in the cache.
3/5Import AI (Jack Clark)· Jack Clark· Oct 12
New benchmark says AI 'research taste' is doubling every three months
P-Zero Research's TasteVal benchmark finds that frontier models' skill at designing and interpreting AI experiments has doubled every three months since December 2025. Its best model beats 24 human experts. Jack Clark has moved his self-improvement forecast a year earlier.
Why it matters · If models are getting better at the experimental side of AI research this fast, timelines for AI that speeds up its own development get shorter. Engineers should watch this curve closely.
Know this first
Context from the editor, not the article · 31 sec
1 · Recursive self-improvement
RSI is the idea that AI systems could do enough of the work of AI research to substantially build their own successors. Each generation would then speed up the next. Many forecasters treat it as the key threshold for very rapid AI progress.
2 · Pass@1 scoring
Pass@1 is the share of questions a model answers correctly on its first attempt, with no retries or picking the best of several samples. It is the strictest common way to report benchmark accuracy.
3 · Mixture-of-experts models
A mixture-of-experts (MoE) model splits its weights into many 'expert' subnetworks and sends each token through only a few of them. Total parameter count can be huge while the compute per token stays near that of a much smaller model.
4/5Latent Space· Richard MacManus· Oct 12
Anthropic's "flows" let Claude design and run its own team of agents
Anthropic's new "dynamic workflows" feature lets a lead Claude agent plan a task and write a program that launches and coordinates its own subagents. In a demo it found more than a single agent did, at much higher cost.
Why it matters · Multi-agent setup is moving from hand-built rosters to topologies the model designs itself, so cost limits, evals and observability now decide whether you can control it.
Know this first
Context from the editor, not the article · 24 sec
1 · Multi-agent orchestration
Splitting a large task across several LLM agents, each with its own context window, then combining their outputs. Usually a coordinator agent hands out work. Developers have typically wired up who does what by hand.
2 · Claude Managed Agents
Anthropic's hosted agent service, used through its API and the Claude Console developer interface. Until now it required developers to configure a coordinator agent and a fixed roster of subagents.
3 · Test-time compute
Spending more computation when the model runs (more reasoning, more attempts, more parallel agents) rather than during training, in order to get better answers on hard problems.
5/5Microsoft Research· Ke Yang, Chenglong Wang, Michel Galley, Jeevana Priya Inala, ChengXiang Zhai, Jianfeng Gao· Oct 12
Microsoft trains per-student simulators to give AI tutors a feedback signal
Microsoft researchers trained a separate AI model to imitate each real student, using de-identified learning records. A chess tutor trained against these simulated students outscored one trained against a role-playing GPT-5.4.
Why it matters · Training against a simulated user is a general technique, and this work shows that a fine-tuned simulator built from real data beats a prompted frontier model playing the user.
Know this first
Context from the editor, not the article · 28 sec
1 · The tutor feedback problem
You judge a tutor by what the student does after the hint, not by the hint alone. Measuring that takes studies with real learners, which are slow and expensive, so they can't keep up with fast model iteration.
2 · Simulated users as reward
Reinforcement learning needs a reward for every attempt. When real humans can't score each attempt, one option is to train a model that predicts how a human would react and then score the agent on that predicted reaction.
The four replication stages; the study skips the exploit stages and focuses on inference setup.1 / 2
Anthropic researchers William Zhou and Daniel Freeman report that small open-weight AI models may be months away from being able to self-replicate on consumer smart devices. They mean things like off-brand Android TV boxes, smart ovens and security cameras. Their concern is a "1": a worm that carries its own model weights and runs them on the hardware it hijacks, so there is no central server anyone can switch off.
Can cheap devices even run a model?
Yes, though very slowly. The team bought 15 off-brand TV boxes (about $55 each, 4 GB of RAM) and ran 2 across them with 3: each box holds two of the model's layers and passes its output to the next box. It generated 0.39 tokens per second.
The hard step: inference setup
The researchers split replication into four stages: find targets, break in, get the model running, then repeat from the new host. Earlier studies already show small models can do the break-in part, so this work focuses on inference setup. That means choosing a compatible software stack, getting package versions right and debugging problems specific to each machine. Using 4, their harness that periodically restarts the agent with notes left by the previous run, Qwen3.5-9B got a model serving on 8 of 9 hardware types, from an Nvidia Jetson Nano to a Mac.
The full loop
On a sealed network of 8 emulated Android cloud machines with an unprotected debug port, only GLM-5.2 completed the whole cycle: it scanned for targets, copied over the payload, stood up a server for Llama3.1-8B and spread again. It reached all 8 nodes in 4 of 5 trials and 7 of 8 in the fifth. Every other model failed consistently. Because breaking in was trivial in this setup, the result is an upper bound. GLM-5.2 is too large to fit on a TV box. But small models have improved by 13.7 points a year on the 5, a general capability score, and that trend puts a small model at GLM-5.2's level by the end of 2026.
How fast, and how bad
On TV boxes, a single inference setup would take about 1–2 days with the 9B model and 3–6 days with the 26B model. A botnet the size of Kimwolf (over 2 million devices) could host about 133,000 copies, which together equal only about 40 H100 GPUs. The authors compare it to a slime mold: slow but hard to wipe out. Copies that inherited notes from earlier copies raised setup success from 45% to between 61% and 84%. Giving the 9B model access to a frontier model for advice did not help.
Defenses
The defenses are familiar ones. Multi-gigabyte weight transfers and steady device-to-device traffic are visible to internet providers. When a provider quarantined Mirai-infected customers, 92% of infections were cleared within 14 days. Banning default passwords shrinks the pool of easy targets.
Anthropic researchers William Zhou and Daniel Freeman have published experiments suggesting that small open-weight models, 30B parameters or fewer, may be months away from being able to self-replicate on commercial Internet of Things (IoT) hardware. That includes smart ovens, off-brand Android TV boxes and AI-equipped cameras. Their specific worry is what they call a 1: a worm whose payload includes model weights, which it runs directly on the devices it compromises. A replicator like this has no central point to shut down. In their evals, a 9B model got inference running on 8 of 9 hardware types.
Why this is a distinct question
Earlier work has shown AI agents behaving like worms on GPU clusters: scanning a network, running an exploit and copying weights across. Datacenter GPUs are the obvious target, but they are guarded. The authors ask whether cheap, poorly defended consumer devices could supply enough compute instead.
To show the compute exists, they bought 15 off-brand Android TV boxes (about $55 each, 4 GB of RAM). They ran 2, a 26B-parameter mixture-of-experts model with about 4B parameters active per token, across the boxes at 0.39 tokens per second. They used 3, where each box holds two layers and hands its activations to the next box, like stations on an assembly line. They chose it over tensor or expert parallelism for fault tolerance. In those schemes, one failed node takes down the whole replica. In a pipeline, the failed box can be swapped out and its cached state recomputed from the box before it. These ARM devices are limited by memory bandwidth, not compute. And during generation only one box works at a time, an effect called pipeline bubbling.
Four stages, one focus
The authors break replication into network enumeration, initial access and payload delivery, inference setup, and onward replication. They largely set aside the first two, because prior benchmarks (CyberGym, ExploitBench and work on adaptive worms) already show small models can exploit systems when given tools. Their focus is the least-studied step: inference setup. It is a long chain of decisions specific to each operating system and device: choosing an inference stack, finding compatible package versions, debugging quirks. Each step depends on the last, and mistakes are hard to recover from.
All runs use 4, a harness the team built on the open-source Pi coding agent. It targets a failure they saw often: an agent fixating on a wrong theory. In one case, a Qwen agent spent hours debugging package versions when the real bug was an endpoint that didn't match the grader's API. Memento lets an agent work for up to 1.5 hours, has it write short notes on what it tried and what failed, then starts a fresh agent with the same workspace and the notes. Total time is capped at 6 hours. The authors say Memento improves reliability on long tasks without adding new abilities.
Tasked with serving Llama3.1-8B on unfamiliar hardware, Qwen3.5-9B succeeded on devices from a Jetson Nano to Macs and an RTX 4090. Larger models such as GLM-5.2 were more reliable and got faster serving speeds.
Putting the loop together
Next, the team built a firewalled virtual network of 8 Google Cloud nodes running 5, an Android 15 emulator. Each node exposed 6, Android's debug interface, with no authentication. The worm starts on one node. It must scan, get a shell on a new node, deliver its weights and harness, and stand up a Llama3.1-8B server that passes correctness and anti-cheating checks. When it passes, a fresh copy of the worm is spawned there. The worm's own "brain" ran on a separate inference server, because the nodes were too small to run the larger models.
Only GLM-5.2 completed the full cycle. It reached 8 of 8 nodes in 4 of 5 trials and 7 of 8 in the other. The other models failed reliably. The test assumes GLM-5.2's ability could be packed into an 8B model; the post's explanation for using Llama3.1-8B rather than self-replication isn't in the text provided. The authors call the result an upper bound because the exploit step was trivial. They note, though, that Mirai spread by trying default manufacturer passwords, which is not much harder.
In an ablation on a 17-node network, two 8-node subnets joined by one pivot node, the team scripted the inference-setup step away. Here many models, including small ones, handled the remaining orchestration, and even a 3B model reached about half the nodes. Inference setup, not the command-line busywork, is what holds models back.
Floor meets ceiling
The authors frame the problem as two limits. A capability floor: the worm must be smart enough, and GLM-5.2 clears it. A size ceiling: the model must fit on its targets, and Qwen3.5-9B and Gemma4-26B-A4B fit on 4 GB TV boxes, with 4 or more boxes for 9B and 10 or more for 26B at 4-bit precision. Today no model clears both. The best open models of 30B or fewer have gained 13.7 points a year on the 7, a general benchmark composite (fit R² = 0.95). That trend reaches GLM-5.2's level by the end of 2026. The authors caution that the index is only a rough stand-in for replication ability.
Speed and scale
The spread would be slow. On the real TV boxes, Gemma averages about 0.3 tokens/s over a 32k-token context window, counting both reading input and generating output. A setup run takes 2.6–4.7 windows. That works out to:
Gemma4-26B-A4B: 27.3–31.8 hours per window, 71–150 hours per setup
Qwen3.5-9B: 10.9 hours per window, 28–51 hours per setup
The authors scale this up to a Kimwolf-sized botnet of over 2 million devices. With one Gemma copy per 15 boxes, that gives about 133,000 copies producing about 80,000 tokens/s in total. One H100 serves this model at about 1,950 tokens/s, so the whole swarm equals about 40 H100s. The authors compare it to a slime mold. Edge accelerators widen the ceiling: there are about 9.5 million Nvidia-silicon IoT units in the field, worth roughly 9,000–35,000 H100-equivalents. But most are not reachable from the internet, and none has ever been mass-compromised.
Adaptation, and advice that doesn't help
Each generation passed its notes on to the next, in 20 lineages of Qwen3.5-9B. Setup success rose from 45% in the first generation to between 61% and 84% in later ones. The authors call this Lamarckian evolution: lessons learned are inherited.
Access to a frontier advisor (Opus 4.8) made no significant difference. That held whether the advisor was offered as a tool or stepped in unprompted. It held even after reinforcement learning taught the small model to call the advisor on coding tasks. Small models "don't know the right questions to ask." The authors expect coordination to run the other way, with a large model handling strategy and starting small models as subagents. Small models also scored "very poor" to "poor" on Meta's CyberSecEval 3 spear-phishing test, so social engineering likely stays with large models.
Defense and open questions
The goal, the authors say, is to push 8, the number of new devices each infected device goes on to infect, below 1. Moving multi-gigabyte weights and keeping up steady device-to-device traffic leave signals that internet providers can spot. A replicator must trade speed against stealth, and decoy devices (honeypots) can capture its payload. When an internet provider quarantined Mirai-infected customers, 92% of infections were cleared in 14 days, against 74% with no notice. Rules like the UK's ban on default passwords shrink the pool of targets.
Gaps remain. Exploits were trivial in these tests. The forecast leans on a general benchmark. And the authors did not test whether an AI botnet could actively evade defenders.
Remember
The bottleneck for an AI worm is getting a model running on strange hardware, not breaking in or orchestration.
Passing notes between agent generations raised a 9B model's setup success from 45% to as high as 84%.
A frontier advisor didn't help small models: they didn't know what to ask, even after training.
Conventional decoding stores state every step; ReplaySSM buffers inputs and writes the checkpoint only on flush.1 / 4
NVIDIA and Princeton's Dao AI Lab have published ReplaySSM, a new way to store the recurrent state of hybrid language models during decoding. It is merged into vLLM, TensorRT-LLM, SGLang and FlashInfer, and GLM-5.3-Flash uses it in production.
Why the "fast" layers are slow
Hybrid models such as Nemotron 3, Qwen3.5 and Kimi-K3 mix a few attention layers with many state-space model (1) layers. These are variants like 2 or 3 that keep a fixed-size state matrix instead of a growing key-value (KV) cache. Each decode step loads that matrix from GPU memory, applies a small update, and writes it back. The work is limited by memory bandwidth, not compute.
The states are large. On Nemotron-3-Super, one layer's FP32 state is 4 MiB per sequence, as much as 8K tokens of FP8 4, and there are five times as many SSM layers as attention layers. At batch size 256 on a B300 GPU, SSM layers take up most of the decode step for contexts up to 100K tokens.
The fused update-and-store step also causes two more problems. 5 (draft several tokens, verify them in one pass, discard the rejects) needs a state snapshot saved per draft token so it can roll back. And a quantized state compounds its rounding error, because each step's output state becomes the next step's input.
The fix: checkpoint plus buffer
ReplaySSM stores a checkpoint plus a buffer of the last L tokens' inputs, a few vectors each. On most steps it appends the new inputs and computes the output from checkpoint plus buffer, writing no state. When the buffer fills, a "6" folds the inputs into the checkpoint and writes it once. With L = 16, state transfers drop from 32 per window to 17. That figure is illustrative arithmetic.
Rollback becomes a pointer move, as with a KV cache. Because most steps need only the output, the kernel can compute it without ever building the state matrix, using the chunked matrix maths prefill already uses. It can also keep a high-precision checkpoint for the recurrence and a separate 4-bit copy just for reads, so read errors never feed back.
Results
Batch 256: the kernel ran 1.46–1.80× faster and the full decode step 1.20–1.47× faster across four models. At batch 1 there was no gain.
Speculative decoding at batch 512: 1.85× and 2.10× the throughput of vLLM's speculative decoding on two 120B-class models. Eight draft tokens verify in about the time of one normal step.
4-bit reads with a high-precision recurrence cut long-generation error drift about 36× on Qwen3.5-4B. On the GPQA-Diamond benchmark, three of four models stayed within about 2 points of full precision; a single shared 4-bit state lost 3–31 points.
The gains need large batches, and the small Nemotron model still lost 5–7 points under 4-bit reads.
NVIDIA and the Dao AI Lab at Princeton have released ReplaySSM, a different way to store the recurrent state in hybrid language models while decoding. The paper is titled ReplaySSM: Cache 1 Inputs, Not State. Code is on GitHub, and the technique is merged into TensorRT-LLM, vLLM, SGLang and FlashInfer. GLM-5.3-Flash uses it in production, and Kimi-K3 uses a closely related concurrent approach.
The problem: the "cheap" layers are the slow ones
Hybrid models such as Nemotron 3, Qwen3.5 through 3.8, Kimi-K3 and GLM-5.3-Flash interleave many state-space model (SSM) layers with a few attention layers. The SSM layers are variants of 2, 3 or Kimi Delta Attention (KDA). For Mamba-2, one decode step is:
S_t = α_t·S_{t−1} + v_t·k_tᵀ, o_t = S_t·q_t
Here S is a state matrix, α is a decay that depends on the input, and v, k, q are the token's input vectors. GDN and KDA add a delta-rule correction, but their updates can be written in the same outer-product form.
Each step loads the full state from high-bandwidth memory (HBM), does a little arithmetic, and writes it back. That makes the kernel bound by memory bandwidth. Two things make it costly on Nemotron-3-Super:
It has five times as many SSM layers as attention layers.
One layer's FP32 state is 4 MiB per sequence. An FP8 KV entry is 0.5 KiB per token, so one SSM state weighs as much as 8K tokens of 4.
At batch size 256 on one Blackwell Ultra B300 GPU, the SSM layers are the largest part of the decode step for contexts up to 100K tokens.
The authors trace three problems to one design choice. The state does two jobs, producing this token's output and becoming the next token's state, and conventional kernels do both in a single fused step:
Traffic. Every step reads and writes the full matrix.
Speculative rollback. Once draft tokens are folded into the state, the update can't easily be reversed. vLLM's workaround saves a state snapshot after each draft token: one read and T writes per step, and those snapshots use memory that could hold more requests.
Quantization. A rounded state is fed back into the next update, so its error compounds. A KV cache never writes an entry back into itself, which is why it tolerates FP8 while SSM states stay in FP32.
The mechanism: checkpoint plus replay buffer
The recurrence can be unrolled over a window of h tokens. The current state is the decayed checkpoint plus a decayed sum of the recent outer products. ReplaySSM stores exactly that: a checkpoint and a buffer of up to L input records (α, v, k).
On a non-flush step, the kernel appends the new record, a few vectors, and computes the output from the checkpoint plus the buffer. It writes no state. When the buffer fills, a 5 step folds the records into the checkpoint, writes it once, and clears the buffer. Outputs match conventional decoding up to floating-point differences.
A worked count, with illustrative arithmetic for L = 16. Conventional decoding moves the state 32 times per window (16 reads, 16 writes). ReplaySSM moves it 17 times (16 checkpoint reads, 1 write). That roughly halves the dominant traffic.
Speculation becomes pointer arithmetic. All T draft records are appended before verification. Committing the accepted prefix means advancing a buffer pointer, and the next append overwrites rejected records. No snapshots are saved or restored, and a flush only folds committed records.
Separating the two jobs also allows output-only evaluation. If you need only the output, (v·kᵀ)·q can be computed as v·(kᵀ·q): form a scalar and scale a vector, without ever building the d_v×d_k matrix. For T drafts against the buffered records, this generalizes to one checkpoint product shared by all queries, plus two small matrix products over cached keys, values and draft queries, with decay masks. This is the same chunkwise computation prefill kernels already use, it runs on Tensor Cores, and it shares kᵀ·q across value heads in a group.
Separate precisions follow naturally. The flush keeps a high-precision checkpoint for the recurrence and also writes a low-precision copy used only for non-flush reads. The low-precision copy is never fed back into the recurrence, so its rounding error stays inside one window.
Results
The team tested four models: Nemotron-3-Nano-4B and Qwen3.5-4B in BF16 on one H100, and Nemotron-3-Super-120B-A12B and Qwen3.5-122B-A10B in 6 (NVIDIA's 4-bit float format) on one B300. They used FP32 state, L = 16, and vLLM's own kernels at their best launch settings as the baseline.
Caveats and open questions
The gains depend on batch size: they come from memory bandwidth, so latency-sensitive single-stream serving sees nothing. Precision splitting is not free for every model. Nemotron-3-Nano-4B still drops 5.6–6.7 points on GPQA-Diamond. A residual read error remains, and the authors suggest finer-grained quantization scales could reduce it. Keeping two checkpoint copies plus a buffer adds memory, and the post doesn't quantify that or the cost of replaying a full buffer on later steps in the window. The baseline is vLLM's kernels, and this is a vendor post. The broader point is credible, though: for hybrid models, keeping the recurrent state out of the per-step critical path is now an inference-engineering problem in its own right.
Remember
In hybrid models at large batch sizes, SSM state memory traffic, not attention, can dominate decode latency up to 100K tokens.
Storing recent inputs instead of updated state makes speculative rollback a pointer move, like a KV cache.
Use one precision for recurrent updates and another for reads: 4-bit reads become safe because their error never feeds back.
The lead item in Jack Clark's Import AI 476 is a benchmark that bears directly on how soon AI might speed up its own development. 1, from P-Zero Research, finds that frontier models' "experimental research taste" has doubled every 3.0 months since December 2025 (95% confidence interval 1.7–5.0 months).
What it measures
P-Zero defines research taste as picking good problems, designing experiments and interpreting results. TasteVal tests the last two. Given a fixed problem, the model designs experiments step by step and draws conclusions from the results. The eight tasks mirror frontier AI work: pretraining data curation, pretraining and fine-tuning, preference modeling, and robustness to adversarial prompts. The human baseline comes from 24 experts recently at OpenAI, Google DeepMind, NVIDIA and Microsoft Research.
The best model, Opus 5.5, beats that baseline with a "2" of 2.30x (CI 1.15–4.37x), at about 1/30 of the experts' cost per attempt. The newsletter doesn't explain how the multiplier is calculated. Progress was much slower before the turn of 2026.
Clark has argued that creativity is the missing ingredient for 3, meaning AI systems that substantially build their own successors. He now thinks full RSI is more likely to become possible in 2027 than 2028. Some caveats apply:
The confidence intervals are wide.
This is a single benchmark with a small expert pool.
It skips problem selection, arguably the hardest part of taste.
Models still miss what a scene implies
4, from Elorian and Scale AI, asks what images and video imply but don't show, such as whether a bookcase can fit more books. It has 522 questions over 288 images and 234 clips. Humans score 93.1 on first-try accuracy. The best model, GPT 6 Astra, scores 53.6. Chinese open models score around 26. Clark argues that even 53.6 undercuts the claim that visual reasoning requires explicit 3D 5.
Western open-weight models trail China
Two Western labs announced open-weight models, meaning models whose weights are published, and both trail Chinese releases on most benchmarks:
Reflection's Beam is a 501B-parameter 6 model with 23B active per token. It scores 44.4 on the 7 coding benchmark, against 61 for Zhipu's GLM-5.3. Its selling point is efficiency: Reflection says it matches GLM-5.2 on reasoning with 3–4× less inference compute, after RL training across a million environments.
Mistral Large 4 is tuned for cybersecurity work with fewer refusals. It scores 62 on DeepSWE, against 68 for Kimi K3, but 28.3 on Terminal-Bench 4.0, against 41.9 for GLM-5.3.
Also in the issue
US national labs used 8, where each site trains on its own data and only model updates are shared, across several supercomputers. One run used about 1,700 GPUs to fine-tune a 7B model. Clark notes this is roughly 100x short of frontier scale.
The lead item in Jack Clark's Import AI 476 is a benchmark that bears directly on how soon AI might speed up its own development. 1, from P-Zero Research, finds that frontier models' "experimental research taste" has doubled every 3.0 months since December 2025 (95% confidence interval 1.7–5.0 months). The best model it tested beats a panel of human AI researchers. Clark says this moves his own forecast: he now thinks full 2 is more likely to become possible in 2027 than in 2028. RSI means AI systems doing enough AI research to substantially build their successors.
Why "taste" is the bottleneck
Clark had argued earlier this year, in Import AI 455, that creativity was the key variable for RSI. Models could already write code and run experiments. What they hadn't clearly shown was the judgment that tells good research apart from busywork. At the start of 2026 he wrote that AI systems "haven't yet displayed this [creativity] in a transformative and major way," though math results were suggestive. Since then, he says, there have been big advances in creative-seeming math breakthroughs, and TasteVal suggests similar progress in AI research itself.
Taste is hard to measure because good research choices only show their value later, and experts disagree about them. P-Zero breaks the concept into three parts: picking interesting problems, designing experiments, and interpreting results. TasteVal tests only the second and third. Given a fixed research problem, the benchmark scores how well a model iteratively designs experiments and draws conclusions from what comes back.
How the benchmark works
TasteVal has eight tasks meant to represent frontier AI development:
pretraining data curation
pretraining and fine-tuning language models
preference modeling (training models that predict which outputs people prefer)
robustness to adversarial prompts
The model under test proposes experiments, sees the results and decides what to try next. Its performance is compared with a human baseline drawn from 24 experts, at least two per task, who recently worked at organizations including OpenAI, Google DeepMind, NVIDIA and Microsoft Research.
The numbers
Trend: experimental taste doubled every 3.0 months from December 2025 (CI 1.7–5.0). Growth was much slower in earlier years, with the slope changing around the turn of 2026.
Best model: Opus 5.5 "significantly exceeds" the human baseline, with a "3" of 2.30x (CI 1.15–4.37x). The newsletter doesn't say how this multiplier is defined. A reasonable reading is how much more effective the model's experimental choices are than the experts', measured in compute terms, but that is an inference.
Cost: about 1/30 of the experts' average cost per attempt.
Caveats
The confidence intervals are wide. The doubling time could be anywhere from under two months to five, and the multiplier's lower bound is barely above parity. This is one benchmark from one group, with eight tasks and a small expert pool. It also deliberately leaves out problem selection, arguably the hardest part of taste. Clark's revised forecast is a judgment call built on this and on the math results, not a measured quantity.
Visual intuition still lags far behind
A second benchmark points the other way. 4 (HSS), from Elorian and Scale AI, tests what a scene implies but doesn't show. Can this bookcase fit more books? Do the chains on these horses bear the plough's drag weight? It has 522 questions over 288 images and 234 short video clips (17.6 hours in total). The questions fall into four areas: temporal and causal dynamics, physical and spatial logic, social understanding, and abstract inference.
Humans score 93.1 5, meaning correct on the first attempt. The best AI, GPT 6 Astra, scores 53.6. Chinese open-weight models do much worse: GLM-5.3-flash gets 26.8 and Kimi K3 gets 25.5. The most common failures are missing a detail that decides the answer (21% of failures) and misidentifying an object, person or role (20%).
Clark reads the result optimistically. Many researchers have argued that real visual reasoning needs explicit 2D or 3D "6" (internal simulations of physical space). That general-purpose language models score as well as they do counts somewhat against that view.
Western open-weight models trail China
Two Western labs announced open-weight models, meaning models whose weights are published, and both trail Chinese releases on most of the benchmarks cited.
Reflection's Beam is a 501-billion-parameter 7 (MoE) model with 23B parameters active per token. It was pretrained on 23.8 trillion tokens using 6,144 Nvidia GB300 NVL72 GPUs for four weeks. Reinforcement learning (RL) then ran on about 10.5K of the same GPUs for another four weeks, across one million coding, agentic and STEM environments, which Reflection believes is one of the largest RL runs by any open lab. Its pitch is token efficiency: it matches GLM-5.2 on advanced reasoning while using 3–4× less inference compute. Head to head with GLM-5.3, it trails:
Weights and a technical report are promised for later this month.
Mistral Large 4, trained on 3,800 Grace Blackwell GPUs in Mistral's European datacenters, is tuned for cybersecurity work with fewer refusals than Anthropic's or OpenAI's proprietary models. It scores 62 on DeepSWE 1.1, against 68 for Kimi K3 and 61 for GLM-5.3. On Terminal-Bench 4.0 it scores 28.3 to GLM-5.3's 41.9, and on Finance Agent V2 it scores 54.7 to 55.8. In Mistral's internal preference test, users preferred it in CAD and STEM and rated it on par in finance and coding. Clark expects it to be the strongest Western open model at release, but still behind China's best. He frames Beam as a chance for the US to return to the open frontier, since Meta has stopped shipping strong LLaMA models.
Government compute, federated
US national labs reported two projects that train models across multiple supercomputers. In 9, each site trains on its own data and only model updates are shared, so restricted data such as nuclear-weapons material never leaves its site.
Three nuclear-security labs (Sandia, Lawrence Livermore and Los Alamos) trained LLaMA-based models at 1B, 8B and 70B across AMD MI300A and Nvidia H100 machines. They compared the federated frameworks NVFlare, Flower and APPFL. The workflow is now in production on classified machines.
Four Department of Energy supercomputers (Aurora, Polaris, Frontier and Perlmutter) fine-tuned LLaMA-2 7B on a chemistry instruction dataset using about 1,700 mixed Intel, Nvidia and AMD GPUs.
Clark notes this is roughly 100x off frontier scale. To match industry, governments would need to tie together about 100,000 chips.
The issue also covers an AI Frontiers essay arguing that AI could give first movers a compounding advantage in space, plus a short story.
Remember
TasteVal tests only experiment design and interpretation, not choosing which research problems to pursue.
Visual intuition still lags far behind: humans score 93.1 on Humanity's Sixth Sense, the best model 53.6.
Reflection's Beam and Mistral Large 4 both trail China's GLM-5.3 or Kimi K3 on key coding benchmarks.
Anthropic's diagram: the lead agent plans, then writes an orchestration program that launches and coordinates subagents.1 / 4
Anthropic has added 1, or "flows," to 2, its hosted service for running Claude agents through an API. Until now, a developer had to set up a coordinator agent and a fixed roster of 3. With flows, Claude decides how to divide the work. CJ Avilla and Harrison Stall of Anthropic presented the feature at AI Engineer New York 2026.
Why this exists
Avilla pointed to a common pain: once many agents are producing "tons and tons of PRs," the human reviewing them becomes "the bottleneck." The presenters compared agents to Linux, which grew from one programmer into layers of trusted maintainers. Their argument is that agents need that kind of structure to take on big projects.
How it works
A lead agent plans the task. Claude then writes an 4, code that launches subagents, gathers their results and controls how information moves between them, across several phases. Stall called it letting Claude "fully orchestrate the entirety of its own multi-agent topology." Developers can still supply pre-configured specialist subagents. Subagents share a filesystem and can exchange information through it, but there is no API for one to message another directly.
The demo was a stock watchlist scanner. In the scan phase, each ticker went to its own subagent with an isolated context. A review phase then checked the findings before writing a report.
Cost and control
Avilla said flows cost "much more" than a single agent but were "much more effective at finding all of the details." The article gives no figures.
The audience questions brought out the controls:
Agent count is held back by "evals and lower defaults," so ambitious tasks need explicit encouragement to run more agents in parallel.
Budget is a session cap set by the developer. The model isn't told its budget, because Stall said that makes agents "a little bit anxious."
Visibility: 5, Anthropic's developer interface, shows each run as a graph, including the exact payload passed to every subagent.
Flows don't improve themselves. To make repeated runs cheaper, Avilla suggested 6, a research-preview feature that reorganises an agent's memory store using past session transcripts. It merges duplicates and replaces stale entries. Hill-climbing (repeatedly tweaking the setup and keeping what scores better) could also help.
The pitch
Stall described flows as "a way to throw 7 at hard problems," meaning extra computation spent while the model runs. He said the goal is to make large scientific work doable by ordinary API users. So far, the only evidence is a small demo.
What was announced
Anthropic has released a feature called 1, which its staff mostly call "flows". It belongs to 2, the company's hosted service for running Claude agents through its API. Anthropic describes flows as "a new type of multiagent orchestration, built for your most ambitious workloads." CJ Avilla and Harrison Stall, Members of Technical Staff at Anthropic, walked through it at a workshop at AI Engineer New York 2026, three days after the announcement. Richard MacManus reported on the session for Latent Space.
The problem it targets
Avilla described a problem that many people running agents will recognise. Once you have many agents producing "tons and tons of PRs," the human becomes "the bottleneck and… the breaking point," stuck in GitHub approving changes and checking that everything works. The presenters compared this to the history of operating systems. Unix started with Ken Thompson working largely alone. Linux grew from Linus Torvalds alone into layers of trusted maintainers. Their argument is that agents need a similar organisational structure before they can take on large projects.
Until now, Managed Agents made the developer design that structure: you configured a coordinator agent and a roster of 3. That works when the shape of the task is known in advance. It works less well when the right split depends on what the task turns out to be.
How flows work
According to the presenters, the idea began in Claude Code, Anthropic's coding agent, as "a harness for every task," meaning a custom control structure for each job. With flows, Claude decides how its agents are organised instead of the developer. Stall called it the point "where you can let Claude fully orchestrate the entirety of its own multi-agent topology."
The process has three steps:
A lead agent writes a plan for the task.
Claude writes an 4, code that launches subagents, collects their results and controls how information passes between them.
That program runs through several phases. Stall said each phase produces "discrete, verifiable units of work" without the developer managing how the pieces connect.
Developers can still pass pre-configured specialist subagents in the agent config if they want specific roles. Subagents share a filesystem and can exchange information through it. The API does not let you instruct one subagent to message another directly.
A worked example. The demo was a stock watchlist scanner, a deliberately small task. In the scan phase, each ticker went to its own subagent with an isolated context, so no agent's analysis was cluttered by the others. In the review phase, the gathered findings were checked before a final report was written. The pattern is a parallel fan-out followed by a review step that brings the results back together.
Results and cost
The article gives no quantitative results. The one comparison is Avilla's statement that, on the watchlist task, flows cost "much more" than a single agent but were "much more effective at finding all of the details." There are no token counts, dollar figures or eval scores. Treat the trade-off as real but of unknown size.
Controls: budgets, evals, observability
Most of the useful material came from the audience Q&A.
How many agents? Someone asked whether a simple ticker scan could spin up hundreds of subagents. Stall said the answer is "evals and lower defaults": the model is tuned toward modest parallelism, and ambitious tasks usually need you to explicitly encourage it to spread out more.
Budget. The lead agent does not know its budget while it runs. Developers can set a session budget as a hard cap. Stall said telling models their budget can make them "a little bit anxious," which suggests Anthropic chose to keep cost awareness out of the model's context on purpose.
Visibility.5, Anthropic's developer interface, shows each run as a graph. For every transition you can inspect the exact payload the lead agent passed to a subagent.
Getting cheaper over time
On making repeated flows more efficient, Avilla suggested asking Claude to review the costs it incurred, and using 6, a research-preview feature. According to Anthropic's documentation, a dream "reads an existing memory store alongside past session transcripts, then produces a new, reorganized memory store." It merges duplicates, replaces "stale or contradicted entries" and surfaces new insights. Avilla said dreaming could "crystallize" the memory associated with a recurring flow.
Self-improvement is not built in. A flow cannot notice that its scanning approach is weak and rewrite its next scheduled run. Stall said this wasn't built in. He and Avilla suggested combining 7 (repeatedly tweaking the setup and keeping what scores better) with dreaming to get some of that effect.
Where Anthropic says this is heading
An audience member mentioned OpenAI's recently announced proof concerning the Navier–Stokes Millennium Prize problem and asked whether flows could support work at that scale. Stall would not discuss how Anthropic itself tackles large scientific problems. He described flows as "a way to throw 8 at hard problems and get more capability faster out of the models," and said Anthropic's goal as an API provider is to make that kind of work "broadly doable by the general public."
What's still unclear
The article gives no numbers on cost or quality compared with single-agent or hand-built rosters.
Since the model doesn't see its budget, it's not clear how flows behave when they hit the session cap. Do they fail, truncate or summarise?
"Lower defaults" means the parallelism default is set by Anthropic's evals, not by your workload, so you will probably need to tune it.
The only demo was a small task, so there is no evidence yet for the "most ambitious workloads" pitch.
Remember
Claude now writes the code that coordinates its subagents, replacing the coordinator-plus-roster setup that developers had to configure by hand.
The model never sees its budget; developers cap spending with a session budget, and default parallelism is kept low on purpose.
Anthropic reports flows find more but cost "much more", and gave no figures for either.
Knowledge tracing has long been used in education software to estimate what a student knows from their past answers. Most of these models predict correctness and cannot read a tutor's free-text explanation.
knowledge tracing
A long-standing education technique that estimates a student's current mastery of skills from their history of answers.
The two evaluation criteria, illustrated with a student who drops a minus sign and then corrects it after guidance.1 / 4
Microsoft Research has built 1, a system that trains a separate AI simulator for each real student. Each simulator models how that student answers questions and how they revise after a tutor's hint. The team also released 2, a benchmark covering chess, second-language English writing, and math. The authors describe it as a research tool for building AI tutors, not a classroom product.
The problem
The real test of a tutor is whether the student fixes their mistake after the hint, and finding that out normally takes slow, expensive studies with real learners. Prompting a large language model to "act like a struggling student" doesn't solve this. The model changes its tone but keeps its frontier-level reasoning, so tutors look more effective than they are. Older knowledge-tracing models, which estimate what a student knows from past answers, are grounded in real data but usually can't read a tutor's free-text guidance.
What a simulator has to do
Drawing on Vygotsky's 3 (what a learner can do alone versus with help), the authors score simulators on two measures:
4: does it match a specific student's answers on held-out problems?
5: does it change its answer after reading the tutor's guidance?
How it's trained
Individual records are sparse, so training happens in two stages. First, a model trains on records pooled from many students in one domain and learns common errors and common revision patterns. Then it is fine-tuned on one student's own records. All data comes from public, de-identified research datasets.
Results
The post gives numbers only for chess. StudentSim scored 0.515 on fidelity and 0.907 on responsiveness. GPT-5.4 scored 0.232 and 0.719: it follows guidance but doesn't behave like the specific player. 6, a model that predicts human chess moves, scored 0.454 and 0.272: it predicts moves well but can't take in coaching.
The authors then used a frozen StudentSim as the reward for reinforcement learning (RL) of a chess tutor. The tutor sees a student's wrong move and writes guidance, the simulator plays a revised move, and the 7 chess engine scores how much the move improved. Human experts rated the tutors without knowing which training they had. The StudentSim-trained tutor scored 90.5% accuracy, against 75.7% with no RL and 71.6% when GPT-5.4 played the student. Training against the overly capable GPT-5.4 student made the tutor worse than doing no RL at all.
Caveats
The RL result covers only chess, where Stockfish gives a cheap, objective score. Nothing yet shows that real students learn more from the improved tutor. The authors call this a proof of concept and say human studies remain essential. The paper and code are public.
Microsoft Research has released 1, a system that builds a personalized simulator for each individual student. The simulator models how that student answers questions and how their answers change after a tutor gives guidance. The team also released 2, a benchmark and evaluation protocol covering chess, second-language English writing, and mathematics. The authors present StudentSim as a research tool for developing AI tutors before testing them with real learners. They say plainly that it is not a product, is not deployed in classrooms, and does not replace human studies. The paper and code are public.
Why simulating a student is hard
The obvious shortcut is to prompt a large language model: "You are an elementary-school student who struggles with math." The model will change its tone and sound unsure, but it still reasons at the level of a frontier model. The authors argue this is the real danger for tutor training. If the simulated student is secretly too capable, almost any hint appears to work, so the tutor looks effective when it isn't.
Classic student modeling fails the other way. Knowledge tracing, which estimates a learner's current skills from their past answers, and similar behavior-prediction models are grounded in real records. Many of them, though, take no natural-language tutor guidance as input. They can describe what a student did, but not how that student would react to a particular hint or Socratic question.
Two things a simulator must get right
The authors draw on Vygotsky's 3, the gap between what a learner can do alone and what they can do with help. From it they define two evaluation targets:
4: does the simulator match a specific student's recorded responses on held-out problems?
5: does the simulator update its response after reading tutor guidance?
The paper's own illustration uses the problem −7.3 × 5. A real student first answers 36.5, which drops the minus sign, then fixes it to −36.5 after guidance. A good simulator should make the same first mistake and the same correction.
You need both measures. A simulator that only predicts past behavior can't tell you whether teaching helped. A simulator that always follows the tutor can't represent one student's particular strengths and mistakes. The blog post doesn't give the exact formulas for either score.
How StudentSim is trained
Individual records are sparse: a learner may have only a few essays or answers. Training one model from scratch per student would overfit, and each model would have to relearn the same domain patterns. StudentSim therefore trains in two stages:
Pooled stage. Train on records from many students in one domain. The model learns common answer formats, common errors, and the usual ways students revise after guidance.
Personalization stage. Fine-tune on one student's own records, producing a simulator for that student.
As an analogy, this resembles a teacher who has seen hundreds of students make the usual errors, then spends a week getting to know one particular child. The post doesn't name the base model.
What the simulator outputs depends on the domain:
Chess: the player's next move, and how they might revise it after coaching.
English writing: essays carrying the learner's characteristic error patterns, and the corrections they would make after teacher feedback.
Math: the student's multiple-choice answer, and whether they change it after an explanation.
Every method is compared on the same per-student training records and the same held-out evaluation records. All data comes from public, de-identified research datasets, and each simulator sees only the records of the student it represents.
Results
The authors say StudentSim beats leading baselines in all three domains, but the post reports numbers only for chess:
Maia2 is a chess model trained to predict the moves real human players make. The two baselines fail in opposite ways. GPT-5.4 responds to guidance but doesn't behave like the specific player. Maia2 predicts human moves fairly well but has no way to take in natural-language coaching. StudentSim does better on both axes. Even so, a fidelity of about 0.52 means it is far from a perfect copy of a student.
Using the simulator as a reward
The more consequential experiment uses the simulator to train a tutor with reinforcement learning (RL):
Each episode starts with a chess position and a move a real student previously got wrong.
The tutor writes guidance.
A frozen StudentSim, whose weights don't change during tutor training, reads the guidance and plays a revised move.
7, an open-source chess engine used here as an independent judge, scores both moves. The reward is how much the revised move improves on the original.
The study compares three tutors built from the same policy, supervised starting point and RL setup:
a supervised fine-tuned (SFT) tutor with no RL;
an RL tutor rewarded by GPT-5.4 playing the student;
an RL tutor rewarded by StudentSim.
Human chess experts rated the tutors' responses without knowing which condition produced them. The StudentSim-trained tutor reached 90.5% accuracy, against 75.7% for the no-RL baseline and 71.6% for the GPT-5.4-reward tutor. It also scored higher on guidance quality and personalization.
The GPT-5.4 tutor scored below the untrained baseline. That fits the paper's argument: a simulated student that is too capable gives a misleading reward, and RL optimizes toward it anyway. The post doesn't define what "accuracy" measures for tutor responses.
Caveats and open questions
Only one RL domain. The tutor-training result comes from chess, where Stockfish gives a cheap, objective score for moves. Writing and math have no equivalent judge, and the post doesn't say how a reward would work there.
No learning outcomes yet. Nothing here shows that real students learn more from the improved tutor. The authors list connecting simulated feedback to real learning outcomes as future work, along with longer learning trajectories, more subjects, and richer student profiles.
Limited reporting in the post. Figures for math and writing, sample sizes, and metric definitions are presumably in the paper but not in the blog.
A single interaction. Each test covers one hint followed by one revision. Whether a simulator can model a student over a whole tutoring session is still untested.
The authors' own framing is modest: StudentSim is a proof of concept for an early-stage feedback signal, and human studies still decide what actually works for learners.
Remember
A prompted frontier model playing a weak student is too capable, and using it as a reward made the tutor worse than no RL.
Pooled training followed by per-student fine-tuning works around sparse individual data, as pretraining followed by personalization does.
The tutor-training result relies on chess's objective scorer, Stockfish; math and writing have no equivalent judge.
Standard decoding, batch 256: the kernel ran 1.46–1.80× faster, flush costs included, and the full decode step 1.20–1.47× faster. At batch size 1 there was no speedup.
7 (multi-token-prediction drafter, GSM8K prompts, T = 4, batch 512): 1.85× and 2.10× the throughput of vLLM's speculative decoding on the two 120B-class models, with matched acceptance lengths. At T = 8, vLLM's verification kernel takes almost 5× a standard step. ReplaySSM verifies eight tokens in about one standard step. Conventional speculation stops improving past batch 128, and on Qwen3.5-122B at batch 512 it is slower than not speculating at all.
Precision: reading NVFP4 instead of FP32 state makes the Mamba-2 non-flush kernel 3.8× faster. On Qwen3.5-4B at 128K generated tokens, measured by KL divergence from FP32 decoding (a measure of how far the output distributions drift), NVFP4 reads with an FP32 recurrence cut drift from 0.136 to 0.0038. FP16 with 8 tracks FP32 closely.
Accuracy: on GPQA-Diamond, which has the longest generations of the three benchmarks, a shared NVFP4 checkpoint loses 3 to 31 points. Separate precisions keep three of four models within about 2 points of FP32. On MMLU-Pro and Math-L5, separate checkpoints stay within 0.5 points on average.
