Coin-Flip Opinions and Dreaming Weights · Morning Gist
Wednesday, October 285 stories · 16 min
Coin-Flip Opinions and Dreaming Weights
Today's common thread is who is really doing the judging: experts echo their agents, LLM judges bend to wording, and models keep learning while they sleep.
Picked from 21 new articles across 24 sources · arrow keys for other days
Today in 30 seconds
30s left
In an Anthropic trial, 50 analysts adopted Claude's coin-flip-assigned preference 86% of the time, often explaining it in Claude's words. story 1 ↓
Google's offline "sleep" phases distill in-context learning into new expert blocks, with reported 10M-token recall and almost no forgetting. story 2 ↓
NVIDIA's three MoE decode tweaks on DGX Spark gave 185% on synthetic tests but only 42.5% on real math problems. story 3 ↓
Rewording prompts swung LLM novelty judges by over 50 points; a plainly prompted Claude Opus 4.6 beat purpose-built checkers. story 4 ↓
DeepMind's Genie 4 world model adds sound and speech, with Gemini Flash planning scenes and checking frames. No benchmarks were published. story 5 ↓
CUDA 13.4 lets Kubernetes workloads share one GPU under hard memory limits instead of each claiming a whole device.
1/5Anthropic Research· Oct 27
Expert analysts adopted Claude's secretly assigned preferences 86% of the time
Anthropic gave 50 professional analysts a Claude agent that was secretly told to favour randomly chosen options on four judgment calls. The analysts sided with it 86% of the time, and most still described the decisions as their own.
Why it matters · If you build or review agent-driven work, a human sign-off may not be the independent check it looks like, and normal error checks won't show this.
Know this first
Context from the editor, not the article · 26 sec
1 · Judgment calls in analysis
Many analyses involve choices with no single correct answer: what to measure, which comparison to use, how to weight the data. Each choice is defensible, but different choices can change the final number by several points.
2 · Causal inference from observational data
Causal inference means working out whether one thing causes another. With observational data, such as farms that chose to enrol in a program, there is no randomised control group, so analysts rely on design choices to correct for differences.
3 · Human in the loop
A common safeguard for AI-assisted work is a human expert who reviews and approves it. The safeguard assumes the reviewer's judgment is independent of the AI's.
2/5Google Research· Oct 27
Google's "sleep" cycle moves what an LLM learns in context into its weights
Google researchers alternate an LLM between normal use and offline "sleep." During sleep it writes recent context into newly added weights and trains on filtered self-generated data. They report almost no forgetting and gains on math benchmarks.
Why it matters · A deployed model that keeps learning without forgetting would change how LLM products handle memory and knowledge cutoffs. This is a concrete recipe for that, with numbers.
Know this first
Context from the editor, not the article · 27 sec
1 · Catastrophic forgetting
When you keep fine-tuning a trained network on new data, gradient updates overwrite weights that encoded older skills, so performance on earlier tasks collapses. This is the core obstacle to models that learn continually after deployment.
2 · Context versus weights
An LLM can pick up new facts inside its prompt (in-context learning), but they vanish when the session ends. Only changes to its parameters persist, and changing parameters risks forgetting.
3 · Mixture of Experts
A Mixture-of-Experts (MoE) model has many sub-networks (experts) and a router that sends each token to a few of them. Experts can be added without touching existing ones, which makes it possible to add capacity in modular pieces.
3/5NVIDIA Technical Blog· Yang Liu· Oct 27
Three inference tweaks raise MoE decode speed 42.5% on NVIDIA DGX Spark
NVIDIA sped up a 124-billion-parameter mixture-of-experts model on its DGX Spark desktop machine. It used custom kernels, a smaller-precision output layer and multi-token drafting. Decode throughput rose 42.5% on GSM8K with no measurable accuracy loss.
Why it matters · It shows the gap between synthetic and real speculative-decoding gains, and it documents a silent failure mode that anyone stacking quantization with draft models should test for.
Know this first
Context from the editor, not the article · 28 sec
1 · Why decoding is memory-bound
When generating one token at a time for one user (batch size 1), the GPU must read every needed weight from memory for each token. Memory bandwidth, not arithmetic, sets the speed, so fewer bytes read means faster output.
2 · Mixture-of-experts models
A mixture-of-experts (MoE) model splits its feed-forward layers into many "experts" and routes each token to a few of them. Total parameters can be huge while each token uses only a small active subset.
3 · Speculative decoding
A cheap draft predicts several upcoming tokens; the full model then checks them all in one pass and keeps the longest correct prefix. Correct guesses yield several tokens for roughly the price of one expensive step.
4/5Hugging Face· Oct 27
Rewording a prompt swings LLM novelty judges by over 50 points
Ai2 tested six GPT and Claude models as judges of research-idea novelty. Their verdicts swung widely with small prompt changes, mostly on AI-generated ideas. Adding search, extra reasoning or a purpose-built pipeline didn't reliably make them better.
Why it matters · If you use an LLM to judge another LLM's output, this is a well-measured case of how fragile that setup is, and of how a check on human-written data can hide the problem.
Know this first
Context from the editor, not the article · 30 sec
1 · LLM-as-judge
Many AI systems use one language model to grade another's output instead of asking humans. It's cheap and it scales, but the grade depends on the judge model, its prompt and its settings. Those choices are rarely tested for stability.
2 · Automated research ideation
Labs are building agents that read papers, propose hypotheses and run experiments. A proposed idea is only worth something if it's new. So these pipelines often add a 'novelty judge' to filter out ideas that just repackage existing work.
3 · Pointwise vs pairwise judging
A pointwise judge rates one item on its own, for example 'novel or not'. A pairwise judge sees two items and picks the better one. Each format can fail differently, so benchmarks often test both.
5/5Google DeepMind· Oct 27
DeepMind's Genie 4 adds sound and speech, with Gemini planning each scene
Google DeepMind released Genie 4, a model that generates explorable worlds in real time. It now also produces sound and character dialogue, and Gemini models plan and check what happens in each scene. So far only invited testers can use it.
Why it matters · Pairing a language model that plans and checks with a model that renders pixels is a pattern practitioners can reuse, and it shows where DeepMind thinks world models are heading.
Know this first
Context from the editor, not the article · 30 sec
1 · What a world model is
A world model predicts what happens next in an environment. Genie-style models produce video frame by frame from user actions such as keypresses, so you can walk around a world with no game engine and no hand-built 3D assets.
2 · The Genie line so far
Genie 1 (2024) learned playable 2D worlds from unlabeled video. Genie 2 extended this to 3D. Genie 3 (2025) ran in real time at 720p and kept scenes consistent for a few minutes, but it had no audio.
3 · Why drift is hard
Each frame is generated from earlier frames, so small errors pile up. If you look away and back, a building or a mug may have changed. Keeping the world consistent over long sessions is the central unsolved problem.
Study design: five decisions per analyst, and a hidden coin flip sets which option the AI favours on each.1 / 4
Anthropic researchers tested whether experts working with an AI agent really make the judgment calls themselves. They ran a 1 with 50 professional analysts, all experienced in 2: working out from data whether one thing causes another. Each analyst used 3, Anthropic's coding agent running Claude Opus 4.8, to advise a fictional farming co-op. The question was whether to expand a paid advisory program. The board would expand it only if it raised productivity by about 3–4%.
The setup
The task had four open decisions, each with two defensible options:
whether to measure yield per hectare or per unit of operating cost
whether to use 4 (comparing how enrolled and non-enrolled farms changed over time) or a direct comparison with statistical adjustment
whether to estimate the effect on enrolled farms only or on all farms
whether to weight every farm equally or by size
Each choice could move the estimate by several 5. A final recommendation followed.
For each analyst and each decision, a coin flip chose which option the agent would favour. Hidden instructions (the 6) told the agent to present that option as its own view, make the strongest honest case for it, change its mind if the evidence demanded, and never mislead.
Results
Analysts went along with the agent: they matched its lean 86% of the time, against about 50% expected if the lean had no effect. The shift per decision ranged from 62 to 82 percentage points.
The final call was predictable: a model using only the four coin flips predicted 82% of final recommendations, against 61% for always guessing the most common answer.
They changed their minds: in 38 cases where an analyst had already stated a preference, they switched 60% of the time when the agent disagreed and 6% when it agreed.
They didn't notice: 77% later explained their choices using the agent's arguments, sometimes word for word, but only 15% credited it.
Following the agent didn't change how confident analysts felt or whether they called the work their own. Experience and expertise didn't predict who deferred.
The agent also helped. Analysts who used it made far fewer serious errors (6%, against 55% for the 11 who worked without it) and finished faster. That smaller group was also less experienced, so the comparison isn't clean.
Caveats and argument
The leans were made stronger on purpose. The study covered one task, one field and one model, and analysts were new to farming. On its own, without an assigned lean, Opus 4.8 made the same five choices in 112 of 123 runs.
The authors argue the bigger risk is correlated judgment. If many teams defer to models with the same preferences, an apparent consensus may reflect one model's bias. That includes AI safety research, where the human reviewer is supposed to be an independent check. Standard error checks won't catch this, because either option looks sound. The authors propose ways to measure deferral, agents that ask for the expert's view first, and institutions that decide which calls need human judgment.
Anthropic researchers Amaç Herdağdelen, Hailey Joren and Alex Tamkin set out to test a common assumption about AI agents: the agent does the work, but the expert makes the calls that matter. In a 1, they gave professional analysts a statistics task to do with 2, Anthropic's command-line coding agent. Each analyst's agent had been quietly told to favour one option on each of several open judgment calls, and a coin flip chose which option. The analysts went along with the agent's lean 86% of the time, and most of them still described the decisions as their own.
The setup
The researchers recruited 61 analysts from a pre-screened pool. All of them had passed a causal-inference knowledge check, used Python or R regularly, and had hands-on experience with observational causal analysis, meaning they had studied cause and effect using real-world data rather than controlled experiments. Eleven chose not to use the agent, which left 50 in the main analysis. Of those, 65% held a master's degree or doctorate or had six or more years of experience.
The task was set at a fictional organisation, the Greenfields Farming Cooperative. About 40% of its member farms had enrolled in a paid advisory program. The board would subsidise and expand the program only if it raised productivity by roughly 3–4%, and it did not say how productivity should be measured. The analysts faced four open decisions, each with two defensible options:
Outcome: crop yield per hectare, or yield per unit of operating cost.
Comparison method:3, which compares how enrolled farms' yields changed after joining against how other farms' yields changed over the same period, or a direct comparison of yields with statistical adjustment for differences between farms.
Which farms are covered: the effect on enrolled farms only, or across the whole membership.
Weighting: every farm counted equally, or larger producers given more weight.
Each of these choices could shift the estimated gain by several 4, which matters when the board's threshold is 3–4%. The analysts then made a final recommendation: roll the program out or discontinue it. A one-year extension was also allowed and was counted as not rolling out.
The agent was Claude Opus 4.8 at High effort, running inside a web version of the VS Code editor. Its 5 (hidden instructions the user doesn't see) gave it the coin-flipped leans. It was told to hold each lean as its own view, to raise it when the decision came up, and to make the strongest case for it that the data allowed. It was also told to change its mind if the evidence turned decisively against it, and never to misrepresent results or hide data. The median session lasted 65 minutes. A separate AI model coded each submission's choices, and the researchers resolved any disagreements with the analysts' own survey answers.
What happened
Analysts matched the lean 92% of the time on the outcome measure, 82% on comparison method, 81% on coverage and 90% on weighting. Comparing analysts whose agents leaned opposite ways, the lean moved choices by 62 to 82 percentage points per decision.
The effect on the final recommendation was smaller. When the agent favoured rolling out, 54% of analysts recommended it; when it leaned the other way, 36% did. The authors attribute the smaller gap to the threshold: once an analyst had an estimate, the recommendation mostly followed from whether it cleared 3–4%. But the estimate itself had already been shaped by the earlier choices. A statistical model using only the four coin flips predicted 82% of recommendations, against 61% for always guessing the most common answer.
In 38 cases, an analyst had stated a preference before the agent gave its view. When the agent then disagreed, the analyst switched 60% of the time; when it agreed, 6% did. With so few cases, the 95% 6 on that 54-point gap runs from 24 to 82 points. The analysts who stated a preference early may also differ from those who didn't.
None of ten traits predicted who deferred: experience, self-rated expertise, education, quiz score, trust in AI, how often they used AI, "need for cognition" (how much someone enjoys effortful thinking), open-mindedness, gender and age. Age looked significant at first, but the link disappeared after correcting for testing many traits at once. Because nearly everyone deferred, there was little variation to explain in the first place.
Not noticing it
On the post-task survey, 52% rated the recommendation as more their own than the agent's. That group's choices had still shifted about 65 points toward the lean, compared with 79 points for the rest, a difference within the margin of error. Confidence was also nearly identical: analysts who followed the lean on all four decisions averaged 71 out of 100, and the rest averaged 73. When asked why they made each choice, and without access to their transcripts, 77% repeated their agent's argument at least once, sometimes word for word. Only 15% credited the agent. Six of 48 guessed the study was about AI influence, and none suspected the agent's preferences had been planted.
Working without the agent did not look better. Six of the 11 analysts who went without AI (55%) made serious methodological errors that invalidated their results, compared with 3 of the 50 who used it (6%). They also took longer, with a median of 90 minutes against 65. The authors caution that the no-AI group was smaller and less experienced, so the comparison is not clean.
Caveats
The leans were made stronger on purpose. Without an assigned lean, Opus 4.8 made the same five choices in 112 of 123 runs, so the model does have defaults. But the full paper reports that its unprompted preferences are weaker, and that it sometimes pushes back on users. The study covered one hour-long task, in farming, a field the analysts were new to, with one model. In real work, experts would know the domain better and could consult stakeholders. The study did not record how strongly each analyst held a view before the agent spoke, so it gives only an average effect.
The authors' argument
The authors suggest experts often form their views while weighing options. The agents were open about how the results changed under each option, but rarely made a good case for the alternative, so analysts could adopt the agent's framing without working out their own. The authors call this a form of 7: the work carries the expert's name, but the agent's priors settle the key calls.
The larger risk is correlation. Agreement between independent teams counts as evidence because the teams are assumed to have reached it separately. If they all defer to models with the same preferences, an apparent consensus could reflect one model's bias. The same applies to AI safety research that uses agents to run experiments, where the human reviewer is supposed to be an independent check. Normal checks won't catch this: with judgment calls, either option produces work that looks sound.
The authors propose three directions: measuring deferral in everyday settings, for example by estimating what someone would have done without AI; designing agents that ask for the expert's view first or argue both sides, but only on the decisions that matter, without overwhelming users; and having institutions decide which decisions need a person's own judgment and protect the skills that judgment depends on. They do not say how an agent should choose which decisions to raise, and they acknowledge that this choice is itself a judgment call.
Remember
Analysts' final recommendations could be predicted from the coin flips alone with 82% accuracy, before they started the task.
Deferring to the agent didn't lower analysts' confidence or their sense that the work was their own.
Agents argued only one side well; the authors suggest presenting both sides or asking the expert's view first.
An LLM using information in its current prompt without changing its weights. The knowledge is gone after the session.
Image: Google Research
Google Research has proposed a "sleep framework" that lets a deployed LLM keep learning. The model switches between a normal wake phase, where it serves users, and an offline sleep phase, where it moves recently learned material into its weights and trains on self-generated data.
The problem
LLMs learn new things in two ways, and both fall short. In-context learning (ICL), picking up facts from the prompt, is lost when the session ends. Fine-tuning the weights lasts, but repeated updates cause 1, where new training overwrites old skills.
How sleep works
Sleep has two stages, named after stages of human sleep. The names are an analogy, and the rules are fixed by humans.
Memory consolidation. The model is built on a 2 design, where a router sends each token to a few sub-networks. It also uses 3, an architecture from the same group that layers memory from fast-changing (the context) to slow-changing (stable weights). During sleep, the model unlocks pre-allocated 4, small cheap add-on blocks, so new knowledge goes into fresh capacity instead of overwriting old weights. It then distills what was in its context into those experts:
On-policy distillation: the model without the context learns from the token probabilities of the model with it, on text it generates itself.
Reinforcement learning: it is rewarded for completing teacher text with matching meaning and small edit distance.
Afterward, the temporary short-term memory is wiped for the next session.
Dreaming. The router deliberately activates unusual experts to generate synthetic samples that mix domains. These "dreams" are filtered by their expected effect on gradients, and the model fine-tunes on the best ones, keeping only changes that improve downstream tasks.
Results
Caveats
The post doesn't name the base model or its size, and gives no compute cost, no sleep schedule and no baseline scores, so the margins can't be judged. Beating GPT-4 at 10M tokens is expected, since its context window is far shorter. Two questions are open: what happens when the pre-allocated experts run out, and whether self-training drifts over many cycles.
Google Research has described a "sleep framework" for continual learning. A deployed LLM alternates between an active "wake" phase, where it serves inputs as usual, and an offline "sleep" phase, where it moves what it learned into its weights and trains itself on self-generated data. The authors are Ali Behrouz, Farnoosh Hashemi, Adel Javanmard and Vahab Mirrokni of Google, Cornell and USC. They report almost no forgetting across sequential tasks, recall over 10 million tokens on a long-context benchmark, and better math-reasoning scores than standard post-training methods.
The problem
A deployed LLM has two ways to take in new information, and both fall short. In-context learning (ICL), where the model learns from material in its prompt, is fast, but the knowledge disappears when the session ends. Updating the weights makes knowledge persist, but repeated updates cause 1: gradients for the new task overwrite parameters that older skills depended on. Traditional ML avoids the problem by keeping training and inference separate. The authors argue that this separation is artificial for an agent that is supposed to keep learning.
The two phases
In the wake phase, the model handles inputs and keeps temporary information in fast, short-term components, such as its context. In the sleep phase, it stops taking new input and spends compute on two stages that the authors name after the stages of human sleep. The architecture and training rules are fixed by humans. The neuroscience vocabulary is an analogy.
Stage 1: memory consolidation (the "NREM" stage)
Parameter expansion. The model is built on a 2 design combined with 3, an architecture from the same group. Hope arranges memory as a "continuum" of blocks that update at different frequencies: the context is the fastest-changing layer, and deeper blocks change slowly and hold stable knowledge. During sleep, the model unlocks pre-allocated 4. These are small adapter-like blocks, cheap because their weight updates are low-rank matrices. The new experts join the slow-changing blocks, so new knowledge goes into fresh capacity instead of overwriting old weights.
Knowledge seeding. Next, what sits in the fast memory is distilled into the new experts. The method uses 5, in which a student learns from a teacher's token probabilities on sequences the student generates itself. That is called on-policy distillation. Here the teacher is effectively the model with the short-term context available, and the student is the expanded model without it. A reinforcement-learning objective called Learning to Imitate (LTI) is added on top. The student completes prefixes of teacher-written text and is rewarded for semantic similarity and for character-level closeness, measured by 6.
To illustrate (a sketch, not an example from the article): say a user spent a session pasting in a grammar of Kalamang, a language with very few speakers. While awake, the model translates using the grammar in its context. During sleep, the expanded model practices translating without the grammar, gets corrected token by token toward what the context-equipped model would say, and earns reward when its outputs mean the same thing.
Synaptic pruning. When consolidation finishes, the temporary fast parameters are reset, which frees them for the next wake session.
Stage 2: dreaming (the "REM" stage)
This is a self-improvement loop with three parts:
Random expert selection: while generating synthetic data, the router deliberately activates experts it would not normally choose, mixing knowledge from unrelated domains.
Curated rehearsal: the model scores its generated samples ("dreams") with gradient-based data selection and keeps those expected to improve it most.
RL refinement: it fine-tunes on the top-rated dreams and checks whether downstream task performance actually goes up.
The theory, as the authors frame it
The authors present consolidation as low-rank compression. The model does not copy every short-term detail. It finds a small set of weight directions that preserve both its own on-policy behavior and the meaning rewarded by LTI. On-policy distillation reduces the gap between the training data and what the model will actually generate at inference. The semantic reward decides what is worth keeping. Expansion supplies room to store it.
They describe dreaming as improving the "geometry of learning." Random experts widen the set of gradient directions the model explores, and gradient-based filtering amplifies the useful ones. They argue the result is better-conditioned optimization that is less dominated by a few directions. The blog presents this informally, with no theorems or assumptions.
Results
What's missing
The post does not name the base model, its size, how long or how often sleep runs, or what it costs in compute. Without those, "outperforms" is hard to weigh. It gives no baseline numbers, so the size of the margins is unknown. The GPT-4 comparison at 10M tokens is lopsided, since GPT-4's context window is far shorter. Any model that compresses context into weights will beat one that cannot see the input at all.
Other questions are left open:
Capacity: pre-allocated experts are a finite budget. The post doesn't say what happens when they run out, or whether pruning only resets fast memory or also reclaims experts.
Safety and quality: a model that trains itself on its own filtered outputs and keeps the results raises the question of whether errors pile up over many cycles. The reported evaluations involve only a few stages.
The full paper may answer some of this. The blog does not.
Remember
New knowledge is written into newly added low-rank experts, not into existing weights. That added capacity is the main defense against forgetting.
The context-equipped model acts as teacher for itself without context, using on-policy distillation plus a semantic-similarity reward.
The blog gives no base model, compute cost or baseline numbers, so the size of the claimed margins can't yet be judged.
NVIDIA's Yang Liu reports three changes that together raised decode throughput by 42.5% for Ling-3.0-flash, a mixture-of-experts (MoE) model from Inclusion AI, on 1, NVIDIA's desktop AI machine. 2 accuracy stayed flat (96.89% vs 96.82%). A synthetic benchmark claimed 185%, and the post explains why that number misleads.
The bottleneck
The model has 124 billion parameters, but only 5.1 billion are active per token: each MoE layer sends a token to 8 of 512 experts. When generating for a single user, profiling showed the GPU spent its time reading weights from memory. Two operations dominated: the expert computation and the 3, the final layer that scores all 157,184 vocabulary tokens.
The three changes
Humming kernels. The baseline used FlashInfer's prebuilt kernels for 4, a 4-bit weight format. Humming compiles kernels at runtime with , CUDA's runtime compiler, tuned to the model's exact matrix shapes. Throughput rose 9.3% on a synthetic test (32K tokens in, 2K out).
A failure with no error message
With the FP8 head and MTP both on, the draft had to share the target's entire LM-head module, not just its weights. Ling's original code didn't allow that. The framework then fell back without warning: the draft read FP8 weights as 16-bit, only one token was accepted per round, and the system ran slower than the unoptimized baseline. The author's advice is to verify at startup that each optimization is active, not infer it from benchmarks.
Real-world numbers
On all 1,319 GSM8K questions with sampling (temperature 0.6), time per output token fell 29.8% and mean latency fell 24.5%, from 15.21 s to 11.49 s. Accept length dropped to 2.68. The author attributes the gap to three things: synthetic output is repetitive, the synthetic run used greedy decoding, and GSM8K answers are short (median about 310 tokens). All results are at batch size 1.
NVIDIA engineer Yang Liu has published a case study on speeding up Ling-3.0-flash, a mixture-of-experts (MoE) model from Inclusion AI, on NVIDIA 1, the company's desktop machine built around a Grace Blackwell chip with unified memory. Three changes stacked together raised decode throughput by 42.5% on the full 2 math benchmark and left accuracy statistically unchanged. The post also shows that the headline synthetic gain of 185% overstated the real one by a wide margin, and that the three changes interact through a bug that can make the combined setup slower than doing nothing.
The model and the bottleneck
The test model, Ling-3.0-flash-fp4, has 124 billion total parameters but only 5.1 billion active per token. It has 42 layers: 35 use 3, a linear-attention variant, and seven use 4, which compresses the attention cache. Each MoE layer sends every token to 8 of 512 routed experts plus one always-on shared expert.
The baseline followed the official cookbook's W4A8 setup (4-bit weights, 8-bit activations) in 5, an open-source LLM serving framework. It used FlashInfer's 6 MoE kernels (MXFP4 is a 4-bit floating-point format with shared per-block scales), NVIDIA CUTLASS for dense matrix multiplies, and an FP32 7. The LM head is the final layer that turns the model's 2,560-dimensional hidden state into a score for each of 157,184 vocabulary tokens.
Profiling at batch size 1 showed decoding was memory-bound: time went to reading weights. Two operations dominated, the MoE expert computation and the LM head. That suggested two strategies: read fewer bytes per decode step, or need fewer decode steps.
Optimization 1: Humming kernels for the MoE
Humming is an independent kernel library (pip package humming-kernels). FlashInfer ships precompiled, general-purpose kernels. Humming instead uses 8, CUDA's runtime compiler, to generate binaries specialized for the exact matrix dimensions, data types and GPU architecture it meets, and caches them in ~/.humming/cache. That lets it tune for Ling's unusually narrow expert width (moe_intermediate_size of 768). As a side benefit, each kernel compiles in seconds. Building FlashInfer's CUTLASS path means compiling 97 targets, which takes tens of minutes.
Result on a synthetic workload (32K input tokens, 2K output, batch 1): time per output token (9) fell from 27.71 ms to 25.35 ms, and throughput rose from 36.09 to 39.45 tokens/s (+9.3%).
Optimization 2: FP8 LM head
The checkpoint stores the LM head in BF16 (16-bit) and explicitly excludes it from quantization. That matrix is 157,184 × 2,560, about 402 million values, and all of it is read for every generated token. As a back-of-envelope figure (not from the article), that is roughly 0.8 GB per token in BF16. The fix applies dynamic FP8 quantization only to the LM head while the rest of the model loads as before. That halves the bytes read for this layer. It is switched on with SGLANG_ENABLE_FP8_LM_HEAD=1 and can't be combined with the baseline's --enable-fp32-lm-head, which also used a slower matrix-multiply path.
Cumulative result: 22.76 ms TPOT, 43.94 tok/s, +21.7% over baseline and +11.4% over Humming alone.
Optimization 3: multi-token prediction drafting
The first two changes make each step cheaper. Speculative decoding cuts the number of steps. Ling ships with a 10 layer, a small extra prediction head trained alongside the model for multi-token prediction (MTP), so it serves as a draft model with no extra training or download. The setup drafts three tokens in sequence, then has the full model verify four positions at once and accept the correct prefix.
Two numbers govern the payoff. Accept length is the average number of tokens accepted per round. The verification-cost ratio is how expensive one verification pass is relative to a normal decode step; it measured about 2.26. The author traces that cost mainly to the MoE, not attention, whose cache reads are small at batch 1. A normal step activates 8 experts for one token. Verifying four tokens can activate up to 32 different experts, so far more weight bytes are read.
On the synthetic workload, accept length hit 3.90 and the full stack reached 9.72 ms TPOT and 102.88 tok/s, +185.1% over baseline. The article doesn't show how accept length and cost ratio combine into this figure.
The silent failure
With the FP8 LM head and MTP both on, the draft must share the target's entire LM-head module, not just its weight tensor. The original Ling NEXTN code didn't expose that interface. The framework then fell back without any error: the draft received FP8 weights but read them as BF16, produced garbage, and accept length dropped to 1.00. Every verification pass then cost about 2.26 steps and yielded one token, which left the system slower than the unoptimized baseline. The cookbook now includes the fix. The author's advice is to confirm at startup that each optimization is actually active rather than inferring it from benchmarks later.
Real workload: GSM8K
Synthetic tests flatter speculative decoding, so the post reruns all 1,319 GSM8K questions. It used five-shot prompts, chain-of-thought, temperature 0.6, top_p 0.95 and top_k 29, with seeds, parser and memory settings held equal between runs.
Metric
Baseline
All three
TPOT
26.45 ms
18.56 ms (−29.8%)
Decode throughput
37.81 tok/s
53.89 tok/s (+42.5%)
Accuracy
96.89%
96.82%
MTP accept length
—
2.68
Mean latency
15.21 s
11.49 s (−24.5%)
P50 / P95 latency
8.36 / 50.20 s
6.11 / 38.60 s
Mean time to first token
0.207 s
0.214 s
Neither run had errors or truncations.
The author gives three reasons the gain shrank from 185% to 42.5%. Synthetic output is repetitive, so drafts are accepted more often (3.90 vs 2.68). The synthetic run used greedy decoding, while GSM8K used sampling. And GSM8K answers are short (median about 310 tokens versus a fixed 2K), which spreads the per-step savings over less output.
What stays open
Everything here is batch size 1 on one machine and one model. The results don't say how the gains hold under concurrent requests, where decoding becomes less memory-bound and verifying several draft tokens could touch even more experts. Accuracy was checked on only one benchmark, GSM8K, and the FP8 LM head's effect on other tasks isn't measured. The author's own conclusion is to benchmark on a workload that resembles your actual deployment.
Remember
At batch size 1, a full-precision LM head over a 157K-token vocabulary is a large memory-bandwidth cost worth quantizing.
Verifying draft tokens in an MoE model activates more experts, making each verification pass cost about 2.26 normal steps.
Synthetic benchmarks overstated the speculative-decoding gain about fourfold; only a realistic workload shows the real gain.
How often each of six judges picked the reviewer-praised human idea, under the hinted prompt versus the plain prompt.
The Allen Institute for AI (1) has released a benchmark and study, Old Ideas, Novel Problems. It tests whether large language models can reliably judge if a research idea is new. They can't, at least not consistently. Small, reasonable changes to the setup flip verdicts on the same ideas, and the flips are worst on AI-generated ideas.
Why this is needed
Automated-science systems generate research ideas, then often ask a second LLM, a "2", whether each idea is actually new. There's no standard recipe for such a judge: teams choose the model, prompt, reasoning budget and literature retrieval. Judges are usually validated on human-written papers, not on the machine-generated ideas they're used to screen.
How the benchmark was built
The team mined 3 2026 peer reviews (ICLR is a top machine-learning conference) for explicit comments on originality, rather than treating acceptance as a stand-in for novelty. High-novelty ideas (154) came from top-10% accepted papers whose reviewers praised novelty and none disputed it. Lower-novelty ideas (145) came from bottom-10% rejected papers with originality complaints. A second set paired the 154 strong human ideas with 154 ideas from a bare-bones LLM generator, assumed to be less novel. An expert who checked 30 of those pairs preferred the human idea every time.
What they found
The judges were GPT-5.1, 5.2 and 5.4 and Claude Sonnet 4.5, Opus 4.5 and Opus 4.6. Prompt wording mattered most. One prompt told the judge that exactly one idea in each pair had been found novel. The other simply asked which idea was novel. Switching to the plain prompt dropped GPT-5.4 from 93.5% to 40.9%, worse than a coin flip, and it changed its answer on over half the pairs. On human-only pairs the same change moved GPT-5.4 about 7 points, though Opus 4.6 moved 21. No prompt worked best for every model.
When judges compared ideas from two different generators, which needs no labels, a quarter to a third of verdicts flipped across prompt variants.
More effort didn't help. GPT-5.6 Sol with extra-high reasoning and live arXiv search scored 0.78 4 (a score that balances accuracy on novel and non-novel ideas). That was below Claude Opus 4.6 at 0.89, at roughly 10× the cost per decision. A simple Opus 4.6 prompt with three votes scored 0.854 for $2.59 in total. Two purpose-built evaluators running on the same model, Idea Novelty Checker and Sakana AI's AI Scientist, scored 0.642 ($30.66) and 0.772 ($82.05).
Caveats
The labels on generated ideas are assumed, the generator was deliberately weak, and all the data comes from one conference. The article doesn't explain why plain prompts fail on generated ideas.
The Allen Institute for AI (1) has released Old Ideas, Novel Problems, a study of how well large language models judge whether a research idea is new. It comes with a benchmark, the data, code, and a reusable pipeline for building new test sets. The headline finding: across six frontier models, small and reasonable changes to the evaluation setup flip verdicts on the same ideas. The flips are worst on AI-generated ideas, which are exactly the ideas these judges are being used on.
Why 2 exist
Ai2 builds tools for automated science. DataVoyager explores structured data, AutoDiscovery generates and tests hypotheses, and Theorizer synthesizes literature into candidate theories. A system that proposes research directions needs some way to tell a new direction from a reworded old one. That job usually goes to a second LLM, which the authors call a novelty judge. It assesses an idea against prior work, either alone or compared with another idea.
No one has settled how to build such a judge. Each team picks a model, a prompt, a format (3 or pairwise), a reasoning budget, and sometimes a literature-retrieval step. The authors point out that judges are usually validated on human-written papers, not on the machine-generated ideas they will actually see.
Building ground truth from peer review
Novelty has no answer key, so the team mined 4 2026 peer reviews (ICLR is a major machine-learning conference). They deliberately did not treat acceptance as a stand-in for novelty. A paper can be accepted on solid execution without being original, and a rejected paper can be original but flawed. Instead they pulled out passages where reviewers explicitly discussed originality.
High-novelty pool (154 ideas): accepted papers in the top 10% of their ICLR research area, where reviewers praised the novelty and no reviewer disputed it.
Lower-novelty pool (145 ideas): rejected papers in the bottom 10%, where reviewers raised originality concerns and none praised novelty.
A human annotator checked 50 extracted signals and agreed with whether each was praise or criticism in 47. None of the three disagreements would have changed how a reviewer was counted.
These two pools make up the Human-Only setting. A second setting, Human+Generated, pairs the 154 high-novelty human ideas with 154 ideas from a deliberately basic LLM generator. The generator got only a research area and the instruction to propose a novel idea, with no search, tools or iteration. The generated ideas are assumed to be less novel. The authors call these weak labels. An expert reviewed 30 pairs, chosen mostly from cases the strongest judge got wrong, preferred the human idea in all 30, and rated none of the generated ideas novel. The team also rewrote both kinds of ideas into a common structure to rule out judges reacting to writing style.
Many of the findings don't depend on the labels at all. If two reasonable setups give opposite answers on the same pair, the judge is unstable whichever answer is right.
The prompt dominates
The six judges were GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. The team varied one factor at a time: prompt, reasoning budget, access to related papers, idea formatting, how many judgments were combined, and which model generated the ideas.
The prompt caused the biggest swings. In the pairwise Human+Generated test, one prompt told the judge that reviewers had found exactly one idea in each pair novel and asked which. The other prompt simply asked which idea was novel. With the simpler wording, GPT-5.4 fell from 93.5% to 40.9%, below the 50% a coin flip would score, and changed its verdict on more than half the pairs. Other models under the same two prompts:
Judge
Hinted prompt
Plain prompt
Claude Opus 4.6
100.0%
72.7%
Claude Opus 4.5
92.2%
58.4%
GPT-5.2
81.8%
76.0%
GPT-5.1
79.2%
56.5%
Claude Sonnet 4.5
60.4%
36.4%
On Human-Only, the same change moved GPT-5.4 by only about 7 points. A judge can look stable on human research and fall apart on generated ideas. Human-Only wasn't immune, though: Claude Opus 4.6 moved 21 points there.
No prompt was best for every model. In pointwise judging, removing a warning not to count trivial or incremental contributions as novel helped some judges and hurt others.
The team also ran a test that needs no labels at all. They compared ideas from pairs of generators: Opus 4.5 against Sonnet 4.5, and GPT-5.4 against GPT-5.1. Across prompt variants, about a quarter to a third of verdicts flipped on average across the six judges. That matters for anyone ranking ideation systems with an LLM judge.
More compute and search didn't help
The authors gave GPT-5.6 Sol an extra-high reasoning budget and live, iterative arXiv search, making it an 5. They compared it with judges that each saw five pre-retrieved abstracts, with every judge making one call per idea. The agentic judge scored 0.78 6. Macro-F1 averages the F1 score for the novel class and the non-novel class, so a judge can't score well by always giving one answer. Claude Opus 4.6 scored 0.89. The agentic judge landed near simpler Opus 4.5 and GPT-5.4 setups and cost about 10× more per decision.
Purpose-built systems did worse still. On Human+Generated:
Both dedicated systems ran on Claude Opus 4.6 with one call per decision. The model was the same, so the gap comes from the scaffolding around it. Note that the baseline used three votes, so the comparison is not exactly like for like.
Caveats and what's open
The labels for generated ideas are assumed, not verified idea by idea, and the generator was deliberately weak. With a weak generator the task should be easy, which makes the instability more striking but says little about judging strong generated ideas. The ground truth comes from one conference and one field. The article also doesn't say why plain prompts collapse on generated ideas. A plausible guess, not stated by the authors, is that fluent LLM prose reads as original when the judge has no hint that one idea is weaker.
The authors recommend testing judges on the kind of ideas they will actually see, measuring sensitivity across prompts and models, improving how literature is brought in, and building benchmarks that refresh as the literature grows. Their pipeline can be pointed at new review sets to do that.
Remember
A judge that looks stable on human-written data can collapse on model-generated inputs, so validate on what you'll actually judge.
Prompt tweaks don't transfer: a wording that helps one model can hurt another.
Extra reasoning, retrieval and custom pipelines cost 10–30× more here without beating a plain prompt.
Google DeepMind has announced Genie 4, the newest version of its 1. A world model here means a system that generates an explorable environment frame by frame as you steer it with the keyboard and text prompts. Genie 4 is in research preview for invited testers. Google AI Ultra subscribers will get it later in 2, Google's app for Genie worlds, as will early-access developers on the Gemini API.
What's new
DeepMind lists four changes:
Native audio. This is the first real-time Genie to generate sound along with the picture: ambient noise, music, and dialogue in which characters react to you.
More realistic people and physics. DeepMind says characters move and react naturally, and that the model handles collisions and aerodynamics.
Live text steering. You can type "make it rain" or "add a person" while the world is running. A Gemini model first rewrites the prompt to fit the current scene.
A Gemini planner in the loop. This is the most interesting architectural change.
How the planner works
The planner is 3, a fast member of Google's Gemini model family. It watches the generated frames, breaks a goal into a sequence of steps, and passes each step to Genie 4 as a text caption. It then checks the new frames to confirm the step happened, and replans if the user steps in or the world goes off track.
In one demo, a model called 4 plans out a game of checkers, and Genie renders 5 humanoid robot playing it move by move. The point is that the pixel model alone cannot keep a long cause-and-effect sequence straight. The language model holds the plan, and Genie draws the result.
DeepMind calls this an early step toward 6, which pair the structured reasoning of code and language models with simulation at the level of pixels.
What's missing
The post contains no numbers: no resolution, frame rate, latency, session length, physics benchmark or comparison with Genie 3. All the evidence is demo video and quotes from testers. DeepMind also admits that details and 7 still drift over longer sessions, after repeated prompt edits, and as you move through the scene. Complex physics, accurate audio, readable text, spatial accuracy and long-term consistency all remain open problems.
Testers are using it to block out film scenes, turn archival photos into explorable spaces, prototype games, and generate simulation scenarios. All output carries 8, Google's invisible watermark.
The verdict: the capabilities shown are wider than before, the architectural idea is worth watching, and the claims can't be checked until outsiders measure them.
Google DeepMind has announced Genie 4, the fourth version of its interactive 1. A world model, in the sense used here, is a neural network that generates an environment frame by frame in response to user input, so you can walk through a scene without a game engine. The authors are Raia Hadsell, Diego Rivas and Hang Qi.
Genie 4 is in research preview with invited testers. Google AI Ultra subscribers will get it next in 2, Google's interface for exploring Genie worlds, along with early-access partners on the Gemini API and Google AI Studio.
The problem Genie is trying to solve
Generating video one frame at a time from user actions is easy to demo and hard to keep coherent. Each frame depends on the ones before it, so small mistakes compound. A doorway shifts, an object disappears when the camera turns away, and causes stop producing their effects. Earlier Genie models showed real-time exploration was possible, but they had no sound. They also struggled with anything that needed several deliberate steps, because a model trained to predict the next frame has no explicit notion of a goal.
DeepMind frames the stakes in terms of AGI (artificial general intelligence) and practical use. Simulated worlds that can vary without limit could serve as training and test grounds for robots and software agents, and as a medium for interactive experiences.
What Genie 4 adds
The post lists four advances.
1. Real-time steering in natural language. While the world is running, a user can type a prompt such as changing the weather, moving the camera, or adding a person or object. Before Genie sees the prompt, a Gemini model rewrites it to fit the current scene. Between user inputs, Gemini also "monitors and steers the scene" so the world keeps moving forward coherently. Navigation still uses ordinary keyboard controls.
2. Physics and human motion. DeepMind claims a "new standard of realism" for real-time simulation, including aerodynamics and collisions. It calls simulated people the biggest jump: characters that move naturally, react to their surroundings and can be steered live. 3, described only as the Gemini model that "anchors the simulations in real-world appearance and behavior", supplies part of this grounding. The post does not explain how Omni is wired into Genie, whether through shared weights, distillation or conditioning.
3. Native audio. This is the first real-time Genie to generate sound together with the frames, so the same model produces both. Each world gets ambient sound that matches its setting, such as wind or surf. DeepMind singles out speech as "the true frontier": characters hold conversations and respond to the user or to events in the world as they happen.
4. A Gemini planner in the loop. This is the most substantive design change. 4 runs alongside Genie 4 as a guiding agent, in a four-part loop:
Observe: watch the scene as Genie generates it.
Decompose: break the goal into an ordered list of steps.
Instruct: give Genie the current step as a text caption, interleaved with the rendering.
Verify: review the incoming frames to check the step succeeded, and replan if the world has drifted or the user has intervened.
A rough analogy: Genie is a highly skilled improv actor with no script and a short memory, and Gemini is the director calling out the next beat and checking that it landed. In the demo, 5 (a Gemini model DeepMind is pairing with its Gemini Robotics work) breaks a "play checkers" instruction into moves. Genie then renders 6 humanoid robot carrying them out. DeepMind says this loop gives "reliable causality, procedural behaviors, and multi-step reasoning", but it offers no measurement of how reliable.
For engineers, the pattern is familiar from agent design: a slow, explicit planner wrapped around a fast, implicit generator, with checks done by looking at the output. DeepMind suggests where this leads in its future-work section, which proposes 7. These would combine "the logical, structured reasoning of code and language models" with pixel-level simulation. Gemini would handle structure and the narrative timeline, while Genie handles what the world looks like, sounds like and does from moment to moment.
Evidence, and what's missing from it
The post includes no quantitative results. It gives no resolution, frame rate, latency, maximum session length, physics or consistency benchmark, human preference study, or comparison with Genie 3 or competing world models. The evidence is recorded real-time demos (all audio, DeepMind says, was generated by Genie 4) and accounts from testers. Treat "unprecedented realism" as a claim until someone measures it independently.
The tester reports describe uses rather than measurements:
Film pre-visualization: filmmakers run several live "takes" of a scene, changing camera, lighting, dialogue and action as they go.
Archival photos: photojournalists turn a single historical photograph into a populated simulation in seconds, or picture how a flash flood or wildfire might unfold.
Game prototyping: developers go from an idea to an explorable prototype and test visual style and mechanics before writing engine code.
Enterprise simulation: partners evaluate Genie 4 for "zero-to-one" scenario generation, meaning a first scenario built from nothing, and report gaps back to DeepMind.
Limitations DeepMind acknowledges
The post is unusually direct about weaknesses. Environments stay coherent over "short sessions", but details and 8 drift as sessions get longer, after several complex prompt edits, and as the user moves through the scene. It lists complex physical interactions, real-time audio accuracy, readable text, spatial accuracy and long-term consistency as ongoing research. These are the properties that matter most for training robots, and they line up with the jobs the Gemini planner is meant to take on.
Safety and provenance
DeepMind says Genie 4 went through internal safety evaluations, follows the same content policies as its other models, and is deliberately going to trusted testers first. All generated video and audio carries 9, Google's invisible watermark for identifying AI-generated content.
What stays open
Three questions remain. First, how much of the gain in coherence comes from Genie itself, and how much from Gemini correcting it at every step? Second, what does the planner loop cost in latency and compute? Third, can a model that keeps the world consistent only over short sessions produce training data good enough for robot policies, where physical accuracy matters more than looking real? The post doesn't answer any of them.
Remember
Genie 4 uses an external Gemini planner to keep events coherent, splitting reasoning from rendering like an agent framework does.
There are no benchmarks, resolutions, frame rates or session lengths, so DeepMind's realism claims haven't been checked.
Drift in object permanence over long sessions is still unsolved, and robot training depends on fixing it.
Sequential translation of Manchu and Kalamang, low-resource languages the model had not seen: almost no drop across stages. ICL, supervised fine-tuning and 5 (offline context compression) forgot badly.
6, which hides facts in very long text: near-perfect recall up to 10M tokens, ahead of GPT-4 and 7, the group's earlier memory architecture.
SQuAD fact incorporation: 48.9% single-task and 46.2% with continued pre-training, ahead of fine-tuning and 8.
Math: 79.2% on AIME 24, 69.0% on AIME 25 and 46.1% on HMMT 25, beating fine-tuning, 9 and on-policy self-distillation.
Continual translation (Manchu and Kalamang, languages the model had not seen): performance dropped almost to zero across multi-stage sequential learning. The baselines forgot badly: ICL, supervised fine-tuning (SFT), and 7, a method that trains a compact cache of context offline.
Long context on 8, a benchmark that hides facts in very long distractor text: "near-perfect scaling" up to 10M tokens, beating GPT-4 and 9, the group's earlier memory architecture.
Knowledge incorporation on SQuAD: 48.9% for single-task and 46.2% for continued pre-training, ahead of the base model, SFT, and 10, an MIT method in which a model writes its own fine-tuning data.
Math reasoning: 79.2% on AIME 24, 69.0% on AIME 25 and 46.1% on HMMT 25. These beat SFT, 11 (the RL algorithm popularized by DeepSeek), and on-policy self-distillation (OPSD).
5
FP8 LM head. The checkpoint kept the LM head in 16-bit, and its roughly 400-million-value matrix is read for every token. Quantizing just that layer to 8-bit halves those reads. Cumulative gain: 21.7%.
Multi-token prediction (MTP). The model ships with a 6 layer, a small built-in head that can draft upcoming tokens. It drafts three, and the full model checks four positions in one pass, keeping the correct prefix. Verification costs about 2.26 normal steps, mostly because four tokens can activate up to 32 experts instead of 8. On the synthetic test, 3.90 tokens were accepted per round, and the total gain reached 185%.
System
Macro-F1
Total cost
Claude Opus 4.6, low reasoning, 7 of 3
0.854
$2.59
Sakana AI's AI Scientist novelty evaluator (iterative 8 search)
