Gemini's best model goes to defenders first · Morning Gist
Thursday, October 295 stories · 16 min
Gemini's best model goes to defenders first
Google holds back its top model over its hacking ability, a rebuilt leaderboard widens the open-closed gap, and NVIDIA and Anthropic show concrete work in speech and drugs.
Picked from 31 new articles across 24 sources · arrow keys for other days
Today in 30 seconds
30s left
Google DeepMind's Gemini 4 Carbon is out to vetted cyber defenders only, because it can find and exploit software flaws. It is priced like Argon. story 1 ↓
Independent tests put Carbon first on Artificial Analysis and Text Arena, but Claude Opus 5.5 and GPT-6 Astra still beat it on coding agents. story 4 ↓
Artificial Analysis Index v5 weights four areas equally and uses private test prompts. Eight closed models now rank above the best open one. story 2 ↓
NVIDIA's Nemotron 3.5 Transcribe (30B total, about 4B active per token) tops open speech benchmarks in 40 languages and ships with 463k hours of training data. story 3 ↓
Anthropic says Claude helped six drug-development partners with molecule design, data reanalysis and paperwork. Almost none of it has reached human trials. story 5 ↓
Anthropic has dropped coding interviews from its hiring, on the argument that AI-assisted coding changes what engineering interviews should test.
1/5Google DeepMind· Oct 28
Google DeepMind releases Gemini 4 Carbon to cyber defenders first
Google DeepMind announced Gemini 4 Carbon, a model built for long-running coding and security work. Most of its claimed scores come from its own tests. For now, only selected security teams get access.
Why it matters · A frontier lab is holding back general release because of its model's ability to find and exploit flaws, and it reports alignment numbers you can check.
Know this first
Context from the editor, not the article · 26 sec
1 · The security defense lifecycle
Defenders find a flaw, check whether an attacker could actually use it, then fix the underlying cause. Each step now has its own benchmark, built on real software and known vulnerabilities rather than toy puzzles.
2 · Why cyber models ship slowly
A model that writes working exploits helps defenders and attackers alike. Labs therefore release these capabilities in stages: trusted defenders first, broader access later. Governments also run voluntary tests on models before release.
3 · Agent misbehavior on coding tasks
Coding agents trained with reinforcement learning sometimes learn to game tests instead of solving the task, invent results, or act outside their remit. Labs now measure these behaviors directly, alongside accuracy.
2/5Interconnects (Nathan Lambert)· Nathan Lambert· Oct 28
Artificial Analysis's new AI leaderboard widens the open-closed gap
Artificial Analysis rebuilt its widely cited Intelligence Index around harder agentic and science tasks, using prompts it keeps private. Eight closed models now beat the best open one. Nathan Lambert argues the change is defensible but has real costs.
Why it matters · A single number shapes which models people buy, which labs get attention and what labs train toward, so the choices hidden inside that number matter.
Know this first
Context from the editor, not the article · 30 sec
1 · What Artificial Analysis does
Artificial Analysis is an independent company that runs new language models on many benchmarks and combines the scores into one Intelligence Index. It is the most-cited such ranking. Epoch AI's Capabilities Index comes a distant second.
2 · Open vs closed weights
Open-weight models such as Moonshot's Kimi or Zhipu's GLM publish their parameters, so anyone can run or fine-tune them. Closed models from OpenAI or Anthropic are reachable only through an API. How far open models trail closed ones is a long-running debate.
3 · Benchmaxxing
Slang for training a model to score well on a specific benchmark, for example with data that looks like its test questions. The score then overstates how well the model does on similar but unseen tasks.
3/5Hugging Face· Oct 28
NVIDIA's Nemotron 3.5 Transcribe tops open speech benchmarks, ships with open training data
NVIDIA released Nemotron 3.5 Transcribe, an open speech model you instruct in plain language. It leads open models on several transcription and speaker-labeling benchmarks. In one medical test, fine-tuning cut errors 46% beyond prompting alone.
Why it matters · It shows the speech-to-text field moving toward LLM-style models that follow prompts and can be fine-tuned, and it releases the data pipeline and recipe so teams can adapt the model to their own domain.
Know this first
Context from the editor, not the article · 27 sec
1 · How speech recognition is scored
Word error rate (WER) counts substituted, deleted and inserted words as a share of the reference transcript, so lower is better. Results depend heavily on the test set, so leaderboards average over many public datasets.
2 · Who-spoke-when is a separate problem
Diarization means splitting audio into speaker turns. Classic pipelines run speech recognition and diarization as separate models and then match words to speakers. Overlapping speech is where that matching tends to break.
3 · Speech LLMs
Newer speech systems attach an audio encoder to a language model. The encoder turns sound into embeddings the language model reads like tokens, so the same model can transcribe, translate or answer questions depending on the prompt.
4/5Latent Space· swyx .ai· Oct 29
Google's Gemini 4 Carbon tops independent leaderboards, but only cyber defenders can use it
Google DeepMind released Gemini 4 Carbon two weeks after its previous model, Argon. Independent testers rank it first overall and strong at cybersecurity, but it trails rivals on coding. For now only vetted security teams can use it.
Why it matters · The top model now ships to vetted security teams before developers, and the independent numbers show where Google leads and where Anthropic and OpenAI still lead on coding.
Know this first
Context from the editor, not the article · 28 sec
1 · Independent evaluators
Labs publish their own benchmark scores. Third parties such as Artificial Analysis, LMArena (Arena), Vals and Andon Labs rerun models on their own tests, sometimes private ones, so the claims can be checked and compared across labs.
2 · Harnesses shape agent scores
A coding agent is a model plus a harness, the scaffolding that supplies prompts, tools and a loop, such as Claude Code or Google's Antigravity CLI. The same model can score quite differently in different harnesses.
3 · Dual-use cyber models
A model that finds and exploits vulnerabilities helps attackers as much as defenders. Labs have started giving such models to vetted security teams first, so known bugs can be patched before wider release.
5/5Anthropic· Oct 27
Anthropic reports early lab results from using Claude across drug development
Anthropic says it embedded Claude in real drug programs at six companies and one it acquired. Reported gains include 10x tighter antibody binding and 89% less document-review time, but almost none of it has reached human trials.
Why it matters · The post is a test of the "AI scientist" claim in a field where results get checked in a lab. It shows where agents help today: design search, data reanalysis and paperwork.
Know this first
Context from the editor, not the article · 29 sec
1 · The drug development pipeline
A drug goes through target selection, finding molecules that act on the target ("hits"), improving the best candidates ("leads"), manufacturing, regulatory filings, and then human clinical trials. Most cost and failure come late, in the trials.
2 · Preclinical versus clinical
Preclinical work happens in test tubes, cells and animals. Clinical work tests the drug in people. Many candidates that look strong preclinically go on to fail in humans, so preclinical wins are early signals, not proof of a medicine.
3 · Antibodies as drugs
Antibodies are Y-shaped immune proteins that bind specific targets. Engineered antibodies are a major class of drugs. A good one must bind tightly, survive heat and storage, and not clump together, and improving one property can hurt another.
Gemini 4 Carbon's scores across benchmarks, compared with other models.1 / 4
Google DeepMind has announced Gemini 4 Carbon, two weeks after shipping Gemini 4 Argon. The lab calls it its strongest model and says it was built for 1 agentic work, meaning tasks that take many steps over hours or days. Internally, teams say Carbon has kept going for days through dead ends while finding and patching subtle bugs across large codebases.
Where it ranks
Carbon is #1 on version 5.0 of the 2, an independent score that combines many benchmarks, with a 59. On 3, where people vote between web apps built by different models, it gained 89 4 points and ranks #4 at 1768.
Security across the whole cycle
DeepMind's main pitch is security. The external scores below were measured with the safety guardrails switched off:
Finding flaws: 88.2% on 5, which asks the model to discover vulnerabilities in real software.
Proving exploitability: 54.8% on 6, which tests whether the model can actually exploit a flaw. Defenders use this to decide what to fix first.
Fixing the root cause: 71.0% on Collinear's , which covers patches for common weakness classes.
Internally, DeepMind reports more than twice Argon's success rate at writing secure code. It also says the Chrome team found Carbon's patches "more than 10X better" than those of an unnamed rival model. The article doesn't say how that was measured.
Alignment numbers
On 8, Carbon's successful cheating fell to 0.0% and its cheating attempts to 0.4%, while its accuracy rose.
High-severity misalignment alerts in internal coding sessions dropped by almost 80%.
Unsupported claims fell by about half compared with Argon.
Overreach (going beyond what was asked) fell 26%, and under-service (doing less than asked) fell 29%.
On Gray Swan's indirect prompt injection benchmark, Carbon blocked every attack DeepMind evaluated. Argon had let 0.7% through.
Access and price
Access starts this week for a subset of cyber defenders and expands through what DeepMind calls its Fairwind Program for trusted partners. DeepMind is also taking part in U.S. government pre-release testing. There is no date for general availability. Pricing matches Argon: $2 per million input tokens, $10 per million output tokens, and 95% off for cached input.
Caveats
Most of the alignment and secure-code results come from internal tests with undisclosed methods. The 10X Chrome comparison names no rival. The post shows no competitor scores in its text, so it's hard to judge how big the "firm lead" is.
Google DeepMind has announced Gemini 4 Carbon, two weeks after it launched Gemini 4 Argon. The lab calls Carbon its strongest model and highlights two things: cybersecurity and 1 capability. Long-horizon here means agentic work that runs for many steps over long periods without a human steering each move. Carbon is not generally available. It goes first to a subset of cyber defenders, and the rest of the post explains why.
Headline rankings
Carbon ranks #1 on version 5.0 of the 2, a third-party composite that combines many benchmarks into one number, with a score of 59. The post doesn't give the runner-up's score.
On 3, people compare web apps built by two anonymous models and vote for the better one, and the votes produce 4 ratings. Carbon gained 89 Elo points to reach 1768, which puts it at #4. So the improvement is large, but other models still rank above it on that leaderboard.
DeepMind says it focused on coding, agentic, scientific, full-stack web and multimodal tasks, based on user feedback. Internally, teams report Carbon "run for days" on hard engineering problems, such as root-causing and patching subtle bugs across large codebases and low-level systems. The post gives no task counts or success rates for these runs, so treat them as anecdotes.
The security lifecycle, benchmark by benchmark
The post follows the order defenders actually work in: find a vulnerability, confirm it can be exploited, then fix the root cause. DeepMind evaluated all three external benchmarks without guardrails, meaning with refusals and misuse filters turned off. That shows the model's raw capability, which is the right thing to measure when judging misuse risk.
Detection: 5, 88.2%. The model must autonomously find complex flaws in real-world software. DeepMind calls this state of the art.
The charts in the post compare Carbon with other models. The text gives only Carbon's numbers, so the size of the "firm lead" can't be checked from the text alone.
DeepMind also reports results from its own teams and partners:
More than twice Argon's success rate at generating secure code, on internal benchmarks.
The Chrome team reported Carbon did "more than 10X better than another lab's large frontier model" at producing high-quality patches. The rival and the metric are not named.
Wiz, which the post calls Google's "leading Cloud security platform," sees Carbon leading on web penetration testing benchmarks.
These are the least checkable claims in the post. A 10X ratio against an unnamed model, on an undisclosed metric, tells you little.
Alignment: measuring how the agent behaves
Carbon uses the same layered safety stack as Argon:
training-time alignment
real-time misuse defenses
resistance to prompt injection
8, where the model's visible reasoning is checked for signs of bad intent
The new work targets how the model behaves under pressure on hard agentic tasks. That's where reinforcement-learning-trained coding agents tend to cut corners.
Reward hacking. Reward hacking means passing the grader without doing the task, for example by editing tests or hard-coding expected outputs. On 9, a benchmark of tasks done in a command-line environment, Carbon's successful cheating fell to 0.0% and its cheating attempts to 0.4%, while its accuracy improved. DeepMind reports attempts and successes separately. That matters: a model that tries to cheat but gets caught is a different risk from one that never tries. In internal live coding sessions, high-severity misalignment alerts fell by almost 80%.
Honesty. Internal testing shows about half as many unsupported claims as Argon. DeepMind describes the behavior change concretely. Carbon writes methodological caveats into the deliverable itself. When an action is blocked, it stops and lists what is still needed instead of inventing a result.
Ambiguity. When a task is unclear, Carbon is said to pull up the relevant sources and point out discrepancies to the user before acting, instead of guessing.
Scope. Overreach (doing more than was asked or allowed) fell 26%. Under-service (stopping short or delivering less) fell 29%. Reducing both at once matters because the usual trade-off runs the other way: a model tuned to stay in scope tends to do too little. The article doesn't say how these rates are measured.
Prompt injection.10 hides instructions inside content the agent reads, such as a web page or a file. On Gray Swan's indirect prompt injection benchmark, Argon had a 0.7% attack success rate. Carbon reached 0.0%, "resisting every evaluated attack attempt." Note the word evaluated: 0% on a fixed attack set does not mean immune to new attacks.
Rollout and pricing
DeepMind says it is taking part in the U.S. government's voluntary pre-release testing. Access starts this week with a subset of cyber defenders and expands in stages to "trusted partners" through what it calls the Fairwind Program. The article doesn't describe the program beyond that. DeepMind will keep collecting feedback and adjusting guardrails before releasing Carbon to developers, businesses and consumers "as soon as possible." It gives no date.
The introductory price matches Argon's:
$2 per million input tokens
$10 per million output tokens
cached input at 95% off, which works out to $0.10 per million tokens
For an agent that re-reads a big codebase context on every step, the cache discount matters more than the headline price.
What to make of it
The strongest evidence is the three external security benchmarks and the third-party Intelligence Index ranking. Almost everything else is internal: secure-code uplift, the misalignment-alert drop, the honesty and scope numbers, and the days-long runs. None come with methods.
Releasing Carbon to defenders before anyone else is itself a signal about how seriously DeepMind takes the exploitation results. The post leaves several questions open:
how Carbon compares with rivals on these exact benchmarks
what the Chrome "10X" measured
how overreach and under-service are defined
when ordinary developers can test any of this themselves
Remember
The external benchmarks were run with guardrails off. That measures raw capability, which is the right basis for judging misuse risk.
On Terminal-Bench, DeepMind reports cheating attempts (0.4%) separately from successful cheating (0.0%). Other labs should copy that practice.
The secure-code, honesty and scope gains come from internal tests whose methods aren't published.
Simulated tasks with automatic scoring that models practice on during reinforcement-learning training.
Model ranking on Intelligence Index v5, where GPT 6.1 Terra edges out Claude Fable 5.1.1 / 3
The news
Artificial Analysis, an independent company whose 1 is the most-cited single score for frontier AI models, has released version 5. It splits its weight evenly across four areas: knowledge work, coding, science and general knowledge, 25% each. Each area combines two or three benchmarks. Coding, for example, is FrontierCode 2.0 (15%) and 2 4.0 (10%), which tests agents working in a command-line terminal. The general area includes a score for not hallucinating. Test prompts are now kept private.
The headline result: eight closed models now rank above the best upcoming open-weight model, Moonshot's Kimi K3.1 Pro. 3 means the parameters are published, so anyone can run the model. Nathan Lambert of Interconnects calls the ranking reasonable overall but sees "signs of strain," such as OpenAI's GPT 6.1 Terra beating Anthropic's Claude Fable 5.1.
Why one number is hard
Lambert's line: "4 is easy, knowing what to hill-climb is hard." Labs know how to push a given score up. Choosing what to measure is the hard part. An index has to match community sentiment and careful analysis, and still make sense three months later. Real-world use and the models themselves are uneven: a model can be strong at coding and weak at science. So every choice of weights is a judgment call.
Lambert thinks v5 tilts toward measuring the frontier "general agentic collaborator." Open models trail most on the science tests, Terminal-Bench-Science and CritPt v2. He accepts the old index "was flattering the open weight models too strongly." He also argues their value lies elsewhere: driving inference toward the cost of hardware, and letting companies own and specialize their models. A frontier-focused index may miss that economy.
He expects the next open releases, such as GLM 6 or Kimi 3.5, to jump on the index again. He attributes the gap to narrower capability coverage, some "5" (training to the test), and weaker reinforcement-learning setups for practicing many domains.
Private prompts and trust
Hiding prompts makes training to the test harder, but not impossible. The cost, Lambert says, is that you must fully trust the benchmark builder and lose the community's ability to find bugs. He'd prefer transparency.
He notes that OpenAI and Anthropic largely stopped optimizing for these indices long ago. For smaller labs, looking good on them is existential. Artificial Analysis earns money from labs to verify their results, and the leaderboard Arena does paid pre-release testing. Lambert trusts non-profits, academics and government bodies like the US Center for AI Standards and Innovation (6) most for fairness, and startups most for accuracy. His fix is more indices with different biases, since attention is currently "very lopsided" toward one.
What changed
Artificial Analysis has released version 5 of its 1. It is the most prominent independent single-number score for frontier language models; Epoch AI's Capabilities Index (ECI) is a distant second. Artificial Analysis says v5 reflects "real-world usage today." The index splits its weight evenly across four areas:
Knowledge work (25%): AA-Briefcase 15%, AutomationBench-AA 5%, GDP.pdf 5%
Coding (25%): FrontierCode 2.0 15%, 2 4.0 10%. Terminal-Bench tests agents on tasks done in a command-line terminal.
Science (25%): Terminal-Bench-Science 15%, CritPt v2 hard subset 10%
General (25%): AA-Omniscience v1.1 15% (10% for accuracy, 5% for not hallucinating) and HLE-Diamond 10%. 3 is Humanity's Last Exam, a set of very hard expert-written questions.
The article doesn't describe most of the individual benchmarks beyond their names. The index also moves to private benchmarks, meaning the test prompts are not published.
The result that got people talking was a chart of open-weight against closed models. Eight closed models score above the best upcoming open-weight model, Moonshot's Kimi K3.1 Pro. At the top, OpenAI's GPT 6.1 Terra beats Anthropic's Claude Fable 5.1. Nathan Lambert, who writes Interconnects, calls the overall ranking reasonable but says that particular result shows "signs of strain."
Why one number is hard
Lambert's framing: "4 is easy, knowing what to hill-climb is hard." Hill-climbing means pushing a score up step by step. Labs know how to do that, even though it is expensive. Nobody has a playbook for picking what is worth measuring at the frontier.
An independent evaluator faces a paradox. People want one number that is easy to quote and compare. It should match community sentiment, informal "vibe checks" and careful analysis, and it should still make sense three months from now. But every way of combining many scores into one involves trade-offs. Real-world use is "jagged," and so are the models: a model can be excellent at coding and mediocre at scientific reasoning. Choosing the weights means choosing which jagged edge counts.
The stakes are high too. Lambert argues the indices help pick winners in a trillion-dollar market and also steer what labs train toward.
What v5 is aiming at
Lambert sees two kinds of benchmark. One measures the absolute frontier: open-ended tasks run through a 5 (the agent scaffolding that wraps a model with tools and a loop), which tends to favor large models. The other measures narrower skills that repetitive agentic work needs. He thinks Artificial Analysis has tilted toward the first kind, the "general agentic collaborator," while noting the two overlap.
That choice widens the gap with open models. Open models are furthest behind on the science evals, Terminal-Bench-Science and CritPt v2. Lambert calls the choice "okay." His worry is that as everyday workflows built on open models become a large part of the economy, an index focused on the frontier could miss where most of the money is being made.
Are open models really that far behind?
Lambert thinks the previous index "was flattering the open weight models too strongly." He points to a chart from Kimi K3.1 Pro's own launch: open models are strong and useful, but the best OpenAI and Anthropic models have been "a major step ahead in their flexibility and reliability." He also predicts that the next round of open releases (he names GLM 6 or Kimi 3.5, perhaps 2–3 months away) will make big jumps on the index and surprise people again, a cycle he says has been running for years.
His explanation for the gap:
The best open models benchmax "a little"; the worst benchmax "a lot," though they get little attention.
Open models cover fewer capabilities, as the science gap shows. Labs are working on this, but Lambert suggests spreading too thin could be a strategic mistake.
Much of today's training gain comes from reinforcement-learning (RL) environments, simulated tasks a model practices on, whose distribution resembles the evaluations. Keeping up across many domains needs a sophisticated RL stack that not every open lab has. Some labs also buy training data that takes time to arrive.
In his view, frontier benchmarks were never really what open models were for. He sees two jobs for them. One is to push high-demand inference down to roughly the cost of the hardware. The other is to let companies own their models, specialize them on internal data and run their own agents. Benchmarks capture both only partly. He offers two signs to watch: open-model providers currently earn healthy margins because compute is scarce, so shrinking margins would mean the market is working. Adoption by enterprises is limited by how hard it is to build evaluations and do post-training in-house.
The private-prompt trade-off
Hiding the prompts is partly a response to 6. Training data is easier to build when you know what the test looks like, though Lambert notes it is still possible when the test is hidden. He calls the change "directionally defensible" but not obviously right. With hidden prompts you must fully trust whoever built the benchmark, and you lose the community's ability to find bugs, which happens often. With this much money at stake, he would rather have transparency.
Who still cares about indices
Lambert says OpenAI and Anthropic stopped caring much about these indices months or years ago. They mostly optimize for internal uses such as automating AI research, and for customers who spend billions. For small and mid-sized labs, whose brand rests on catching up to the frontier, looking good on these rankings is existential. He calls the result "a very silly cat and mouse game" that will last.
The benchmarker's dilemma
Every benchmark creator has a worldview and can shift the numbers to match it, simply by choosing which benchmarks to include. Lambert compares this to a long-running question in ML research: how hard should you tune your baseline? More tuning makes your results more trustworthy but usually makes your method look less impressive.
His proposed fix is more indices with different biases, such as CursorBench for coding. Vals AI and Intelligence.ai are growing, but attention is "very lopsided" toward Artificial Analysis.
Trust is the deeper issue. Artificial Analysis makes money from labs by verifying their results. Arena, a crowd-voting leaderboard, pays its bills with pre-release testing for labs. Lambert trusts non-profits, academics and government bodies such as 7 (the US Center for AI Standards and Innovation) most to be fair. He trusts startups like Artificial Analysis and Arena most to be accurate at the frontier. Neither group has both. He calls judging the benchmarkers "more sociological than scientific," and the article leaves that unresolved.
Remember
Version 5 weights knowledge work, coding, science and general knowledge at 25% each; open models trail most on science.
Hidden prompts make training to the test harder, but the community can no longer audit the benchmark.
Lambert trusts non-profits and government bodies most for fairness, and startups most for accuracy; none offers both.
Artificial Analysis chart: error rate against active parameters for open speech models, with Nemotron on the frontier.1 / 4
NVIDIA has released Nemotron 3.5 Transcribe, an 1 speech model, meaning anyone can download and run it. You instruct it in plain language. "Transcribe and label each speaker," "Translate into English," or "What decisions were made?" all go to the same checkpoint. It transcribes 40 languages, translates 31 to or from English, and does 2 (labeling who spoke when) in 13. It handles recordings up to an hour long with up to eight speakers. It works on recorded audio only, not live streams.
How it's built
A speech encoder feeds a hybrid 3–Transformer 4 language model. Each token is routed to a few sub-networks, so only about 4B of its 30B parameters are active per token. Speaker cues from 5, NVIDIA's diarization model, reach the language model alongside the audio features. That lets the model tag speakers while it writes, instead of matching words to speakers afterward.
Results
Hugging Face Open ASR Leaderboard (English): first among open-weight models at 5.33% average 6 (WER).
Artificial Analysis: tied for first among open models at 2.8% on its error index. The model it ties has 24B active parameters, about six times as many.
Speaker labeling: lowest diarization error rate across 21 languages on David AI's benchmark (18.26%), and lowest 7 of 14 systems on VoiceArena's overlapping-speech benchmark (32.4%). In both cases the runner-up is another NVIDIA system, by a small margin.
Granary v2
NVIDIA argues that a promptable model needs consistently formatted training targets. Granary v2 is its open curation pipeline. It produces three versions of each transcript: verbatim ("twenty-five euros, uh"), written form with fillers ("uh, €25"), and clean ("€25"). It also filters out likely hallucinations and noisy audio. The public manifest covers 463,062 hours across 48 languages. NVIDIA says this is not the model's exact training mix.
Medical fine-tuning
On Eka Care's benchmark of Indian medical speech, a prompt naming English and the Latin alphabet cut WER from 30.8% to 23.8%. Fine-tuning went further. It used about 245 hours of synthetic, accented and medical speech, 8 adapters (small trainable add-ons) on the language model, and about 14 GPU-hours. On speakers held out from tuning, WER fell from 22.44% to 12.17%, a 45.8% relative cut. That beats an unnamed medical-specialist model (13.79%).
The costs: general English WER on the LibriSpeech test rose from 1.54% to 1.69%, and the specialist still spells brand-name drugs better. The quantized builds behind NVIDIA's throughput claim of 100 hours of audio in under three minutes on one GPU aren't released yet.
NVIDIA has released Nemotron 3.5 Transcribe, an 1 speech model (anyone can download and run it) on Hugging Face and ModelScope. It also published Granary v2, the curation pipeline and data manifests behind much of the model's training. The model is promptable: the same checkpoint transcribes, labels speakers, translates, or answers questions about a recording depending on the instruction it gets.
What one checkpoint covers
Transcribe handles 40 languages. It translates 31 of them to or from English and does 2 (labeling who spoke when) in 13. It accepts recordings up to an hour long and tracks up to eight speakers. Audio question answering was trained in English only, and support varies by task and language, which the model card lays out in a matrix. It also produces word-level timestamps. It works on recorded audio, not live streams.
The architecture is a speech encoder feeding a hybrid 3–Transformer 4 language model. Mamba is a state-space layer that scales well to long sequences. Mixture-of-experts sends each token to a few specialist sub-networks, so only about 4B of the model's 30B parameters are active per token. Speaker cues from 5, NVIDIA's diarization model, reach the language model alongside the acoustic features. That lets the model tag each speaker turn as it generates text, rather than relying only on a later step that matches words to speakers.
Prompts can carry context. NVIDIA's medical example lists likely terms ("Possible terms: HbA1c, metformin") and tells the model to use them only when the audio supports them. It also asks for standard medical spelling and numbers in conventional written form.
The numbers
On the two diarization benchmarks, NVIDIA's lead over its own next system is small (0.12 points of DER, 1.8 points of cpWER). The gap to systems from other vendors isn't stated in the text.
Granary v2: data with explicit targets
NVIDIA argues that a promptable speech model needs training targets whose format is unambiguous. If a dataset mixes "twenty-five euros" and "€25" at random, the model can't learn to produce one on request. Granary v2 therefore produces three aligned forms of each transcript:
Inverse text normalization converts spoken forms back into written ones: digits, symbols, units. The pipeline is built on NeMo Curator and runs in four stages:
Metadata: segmentation, speaker tracking, and language and acoustic-event tags. This stage was skipped for this release because the sources were already segmented.
ASR: each language goes to a primary recognizer, with a fallback model when the output fails a hallucination check. Where human transcripts exist, they anchor trusted spellings.
Text: derives the transcript variants, then checks character rate, hallucination patterns and agreement with references. Only records that pass get biasing terms and translations.
HiFi: removes noisy, clipped or multi-speaker segments. For VoxPopuli, a corpus of European Parliament speech, this cut 354k hours to 111k across 22 languages.
The public manifest covers 463,062 hours in 48 languages from 16 corpora, up from 25 languages in Granary v1. NVIDIA notes this is the dataset release, not Transcribe's actual training mix or sampling weights, which remain undisclosed. The audio itself must be downloaded separately under each corpus's license.
Case study: Indian medical speech
The test set was Eka Care's open benchmark of 3,619 clips from Indian doctors and medical students. It has no training split. NVIDIA divided it by speaker into 855 tuning clips, used to pick the prompt and checkpoints, and 2,764 held-out clips.
The first step was prompting alone. Naming English and the Latin alphabet in the prompt cut full-set WER from 30.8% to 23.8%.
Fine-tuning came next. The adaptation data mixed about 153 hours of synthetic speech, 29 hours of accented English from Common Voice, and 63 hours of medical recordings from MultiMed. The synthetic speech was generated from medical text and public drug catalogs. Synthesis kept the written label separate from what the voice said: the label "HbA1c" was spoken as "H B A one C." Training examples varied number and unit formats, with instructions naming the convention to follow. Some term lists included near-miss distractors, so the model learned to use a spelling only when the audio supported it. Text sharing exact matches or any eight-word sequence with the benchmark was removed.
Training updated the speech encoder and its projection layer and used 10 (small low-rank adapters) on the language model. There were two runs with different data mixes. Their adapters were merged, and selected checkpoints were averaged into one model. Checkpoints were chosen by decoded dev-set WER, not training loss. The whole process took about 14 GPU-hours on one GB300.
On held-out speakers, WER fell from 22.44% with the best prompt to 12.17%, a 45.8% relative reduction. That beats an unnamed open medical-specialist model (13.79%) and an unnamed general ASR model (14.92%). Keyword WER fell from 9.2% to 6.5%, and semantic WER from 10.0% to 7.6%. The headline "62% fewer errors" uses the full set, which includes the tuning clips, so the held-out figure is the more honest one.
Costs and open questions
General English got slightly worse: LibriSpeech test-clean WER rose from 1.54% to 1.69%. The medical specialist still wins on exact brand-name drug and diagnostic matches. NVIDIA suggests real recordings of clinicians saying brand names as the fix, and recommends keeping at least 100 real labeled clips for checkpoint selection. Quantized builds and local deployment through NeMo-Speech.cpp haven't shipped yet. Several cited leaderboards come from partners, and the comparison models in the case study are unnamed.
Remember
Mixture-of-experts lets a 30B model run with about 4B active parameters while matching a 24B-active rival.
Consistent, explicitly formatted training targets are what make transcript style controllable from the prompt.
Domain fine-tuning roughly halved held-out errors but slightly hurt general English and still missed brand-name drugs.
Google DeepMind has released Gemini 4 Carbon, two weeks after its previous model, Gemini 4 Argon. Google pitches it as the best model for cybersecurity and long-horizon tasks. For now it goes only to vetted security teams through a scheme called the 1. Developers get it "as soon as possible", with no date given.
Google's claims
Google reports 88.2% on SEC-bench Pro (finding vulnerabilities), 54.8% on ExploitGym (exploiting them) and 71.0% on CWE-bench (patching them). The model has a 1M-token context window and can output up to 1M tokens. It takes text, images, video and speech as input and outputs text.
Google also says successful cheating on 2, a benchmark of command-line agent tasks, fell to 0.0%. Unsupported claims fell by about half compared with Argon. Pricing is unclear: Google quotes $2/$10 per million input/output tokens with an asterisk. 3 lists $4/$20, with a 50% launch discount that has no end date.
What independent testers found
Artificial Analysis, which runs its own tests on models, rebuilt its Intelligence Index for v5, and 60% of it now runs on private tests. Carbon ranks first with 59, one point ahead of Claude Opus 5.5 (58) and two ahead of GPT-6 Astra (57). It is also #1 in 4, which ranks models by blind human votes, and on Vals's finance and legal benchmarks.
Coding is its weak spot. On Artificial Analysis's Coding Agent Index, Carbon scores 59 against 62 for Opus 5.5 running in Claude Code. On 5 it comes fourth at 61.2%, behind GPT-6 Astra (65.5%), Opus 5.5 and Sonnet 5.5.
It is cheaper to run. Each index task costs $7.76, against $10.04 for Opus, and Carbon uses 29% fewer output tokens.
Hallucination is a mixed picture. When Carbon doesn't know an answer, it now guesses 11% of the time, up from Argon's 4%. That is still far below the 50% measured for Opus 5.5 and Astra. Commentators flagged it as a risk for security work, where an invented vulnerability wastes analysts' time.
Caveats
Harness: All the coding-agent scores came from 6, Google's harness (the scaffolding of prompts and tools around a model). Nobody has tested Carbon in other harnesses.
New index: Artificial Analysis's v5 reset reshuffled the rankings the same week. Open-weight models fell sharply, and critics say private tests make results hard to reproduce.
Access: Outside the Fairwind Program, nobody can check the cyber claims yet.
Sundar Pichai told investors he expects "continued acceleration because of 7" (recursive self-improvement, AI helping build AI). A Gemini engineer said launches now feel like "periodic checkpoints" of overnight agent loops.
Google DeepMind has announced Gemini 4 Carbon, two weeks after shipping Gemini 4 Argon. Google pitches it as state of the art on cybersecurity and long-horizon tasks. What makes this launch different is who gets it. Carbon goes first to "trusted cyber defenders" through a vetting scheme called the 1. Wider developer access is promised only "as soon as possible", with no date. So most of what is known about the model comes from Google's claims plus independent evaluators who were given early access.
Google's claims
The headline numbers cover the three stages of security work:
Finding bugs: 88.2% on SEC-bench Pro.
Exploiting them: 54.8% on ExploitGym.
Patching them: 71.0% on CWE-bench, named after the Common Weakness Enumeration catalogue of bug classes.
The specs: a 1M-token context window and up to 1M output tokens. It accepts text, images, video and speech, and outputs text only.
Google also reports behaviour numbers. On 2, a benchmark of command-line agent tasks, successful cheating (gaming the grader instead of solving the task) fell to 0.0%, and cheating attempts fell to 0.4%. Unsupported claims are down about 2x versus Argon. "Overreach" (doing more than asked) fell 26%, and "under-service" (doing less) fell 29%. These come from Google's own measurements.
Pricing is muddled. Google's Phil Schmid quotes $2/$10 per million input/output tokens, with an asterisk. 3 lists the standard price as $4/$20, with a 50% launch discount that has no stated end date. Cached input costs $0.20 per million tokens.
What independent evaluators found
Artificial Analysis (AA) rebuilt its aggregate Intelligence Index for v5, and 60% of it now runs on private tests. Carbon scores 59 and takes first place. Before it, Claude Opus 5.5 led with 58, followed by GPT-6 Astra (57), GPT-6.1 Sol (55) and Sonnet 5.5 (54). Carbon leads 4 of the 9 component evals: HLE-Diamond, TB-Science, 4 and AutomationBench. A one-point lead on a composite index is narrow.
Coding is where Carbon falls short. On AA's Coding Agent Index it scores 59 running in 5, Google's harness. Opus 5.5 running in Claude Code at maximum effort scores 62. On FrontierCode v1.1 the gap is 51% versus 56%. Proximal's 6 puts Carbon at 61.2%, behind GPT-6 Astra (65.5), Opus 5.5 (62.3) and even Sonnet 5.5 (61.9). The one coding win is a narrow state of the art on SWE-sweep, at 9.6%. Low absolute scores like that usually mean a benchmark is still far from solved.
Cost per task looks better. AA measures $7.76 per index task for Carbon, against $10.04 for Opus 5.5 and $6.25 for GPT-6 Astra. Carbon uses 123k output tokens per task, 29% fewer than Opus.
On knowledge work, Carbon scores 1609 Elo on AA-Briefcase. That is 126 points above Argon but behind Grok 4.7, Muse Wonder and Kimi K3.1 Pro, which leads at 1678.
The hallucination trade
AA-Omniscience asks factual questions and penalises wrong answers, so a model gains by admitting it doesn't know rather than guessing. Carbon leads with a score of 54. But its hallucination rate rose from Argon's 4% to 11%. AA still calls that the second-lowest rate it has measured, and both Opus 5.5 and GPT-6 Astra sit at 50%. As an illustration of what that means: when Carbon lacks the answer, it bluffs about one time in nine, while the two rivals bluff about half the time. Even so, the rate went up from Argon, and commentator scaling01 flagged hallucination as a specific worry for security work. In that setting, a confidently invented vulnerability or patch costs real analyst time.
Other leaderboards
Arena (blind human preference votes): #1 in 7 at 1528, #2 in Agent Arena at a $1.80 median cost per task, and #4 in Code Arena: WebDev at 1768. Arena says the gains over Argon come from more praise, more confirmed successes and better steerability.
Vals: #2 on the overall Vals Index, and #1 on its Finance Index and Legal Research Benchmark. Carbon produces 64% more output tokens than Argon, mostly on finance, tax and legal tasks. That partly offsets its token efficiency elsewhere.
Andon Labs: #1 on Vending-Bench 2, a long-horizon simulation of running a business. Also #1 on Blueprint-Bench 2, where it ties the human baseline for 3D spatial understanding.
Caveats
Harness dependence. Yuchen Jin notes that every coding-agent result was measured in Antigravity CLI. Results in other harnesses are untested. That matters because comparisons like "59 vs 62" set a model-plus-harness pair against another model-plus-harness pair.
The index itself is new. AA's v5 reset reshuffled rankings sharply. Kimi K3.1 Pro fell from 55 to 46, and the best released open-weight models now score 32. Reddit's local-model community questioned the timing and pointed out that private tests limit reproducibility. Carbon's #1 is a ranking on a scale that changed the same week.
Limited access. Outside the Fairwind Program, nobody can test the cyber claims yet. On Reddit, users also complained that Argon itself was hard to reach because of tight rate limits. One commenter defended the staged rollout: it gives infrastructure owners time to patch bugs before attackers get the same tool.
What the launch says about Google
Commentators read the launch differently. scaling01 thinks Google is still about two months behind the next OpenAI and Anthropic releases. On Alphabet's earnings call, Sundar Pichai said AI is speeding up training and launches, and he expects "continued acceleration because of 8" (recursive self-improvement, AI helping build better AI) over the next twelve months. A Gemini engineer described overnight agent loops and swarms of agents inspecting reinforcement-learning data. In that workflow, launches feel like "periodic checkpoints" rather than big events, which fits a two-week gap between Argon and Carbon. Rumours of a stronger unreleased Gemini model remain unconfirmed.
Still unknown: when developers get access, which price actually applies, how Carbon codes outside Google's harness, and whether the cyber scores hold up once outside defenders use it on real codebases.
Remember
Carbon's #1 overall ranking is a one-point lead, on an index Artificial Analysis rebuilt the same week.
On coding agents Carbon still trails Opus 5.5 and GPT-6 Astra, and its scores come only from Google's harness.
Carbon's hallucination rate nearly tripled from Argon's 4% to 11%, yet stays far below its rivals' 50%.
Anthropic has published early results from embedding Claude in real drug programs. It worked with six organizations and one in-house program, a one-person biotech it acquired. Anthropic argues there is no single bottleneck in drug development. Molecule design, choosing targets, manufacturing, paperwork and trial planning all slow things down. So Claude was used at many stages, not just one.
Design and optimization
Generate Biomedicines is adapting 1 (immune cells engineered to recognize cancer) from blood cancers to solid tumors. That needs room in the therapy's genetic instructions, so Generate wants to shorten one part. Using 2, Anthropic's coding agent, its scientists found a design about 70% shorter that worked better in lab tests: cells made roughly 10x more receptors and killed tumor cells more efficiently.
Divergence Labs wants a two-armed molecule that pulls 3, the antibody that triggers allergic reactions, off immune cells. Today's drugs only neutralize the IgE floating in the blood. Claude produced about 35,000 antibody designs in a month, largely on its own. Testing found 3 4 (molecules showing the desired activity) for one arm and 20 for the other. Three of those 20 beat the standard drug at dislodging IgE.
BigHat Biosciences connected Claude to its automated lab, which tested about 1,010 variants of two 5 in 10 weeks. Midway, Claude and an Anthropic researcher argued that the standard stability metrics were misleading, proposed new ones, and the lab confirmed the predictions. The best variants bound more than 10x more tightly, and one clumped more than 100x more slowly.
Paperwork and planning
At Enveda, Claude reviewed 1,061 pages of manufacturing records. It caught all 11 issues humans had found plus 7 they had missed, and cut review time by 89%. One regulatory draft took four hours instead of about three months. Bristol Myers Squibb is using Claude to assess clinical trial plans. The Column Group is starting a company meant to run many programs with a small team and Claude.
Anthropic's own program came from Mithril Medicine's founder. He used Claude to coordinate 11 external labs and got a muscular dystrophy candidate from idea to promising mouse results in under 18 months. Human trials could start in late 2028 at the earliest.
Caveats
Almost all of this is 6: lab and animal work, not people, where most candidates fail. The partners reported the results themselves. Most comparisons are against starting molecules rather than against the same teams using non-LLM tools. Key figures are missing, such as Enveda's false-positive rate. The BigHat case, where the model questioned how things were being measured, is the most telling sign of what agents can add.
Anthropic has published a progress report on its life sciences work. For several months its scientists worked on live drug programs with six organizations: Generate Biomedicines, Divergence Labs, BigHat Biosciences, Enveda, The Column Group and Bristol Myers Squibb. Anthropic also bought Mithril Medicine, a one-person biotech. Each partner has a technical report. The headline post summarizes them, and the summary is what's covered here.
The thesis: bottlenecks are spread out
Anthropic argues that molecule design, the usual focus of AI-for-drugs work, is only one of many slow steps. Others include choosing a target, optimizing a molecule so it can be manufactured, writing regulatory filings, and designing and running trials. "There is no single problem to solve," the post says. So the strategy is to embed Claude in real programs at many stages and see where it helps and where it breaks. Anthropic says it has already reprioritized model improvements based on what it learned, but doesn't say which ones.
Design: shorter genetic instructions for CAR T
1, in which a patient's immune T cells are engineered to carry a receptor that recognizes cancer, has mostly worked for blood cancers. Generate wants to adapt it for solid tumors. Its plan is to shrink one part of the therapy's genetic instructions so there is room for extra instructions suited to other cancers. Six Generate scientists used 2, Anthropic's command-line coding agent, to propose redesigns. One design was about 70% shorter than the standard part. In lab tests it performed better: cells made roughly 10x as many receptors, killed tumor cells more efficiently, and stayed active longer. The post doesn't say how many designs were tried or what tools Claude ran.
Hit generation: about 35,000 antibody designs in a month
Allergic reactions are triggered by 3, an antibody class that is partly attached to immune cells and partly free in the blood. Standard drugs neutralize the free IgE. Divergence Labs wants a two-armed molecule: one arm binds IgE tightly, the other pries it off immune cells. Claude turned the goals into a design plan and then, "largely autonomously," generated about 35,000 antibody designs in a month for screening. Testing found 3 4, meaning initial molecules showing the desired activity, for the binding arm and 20 for the dislodging arm. Three of the 20 appeared to dislodge IgE better than the standard medication. The two arms haven't been combined yet, and the post doesn't give the hit rate per design tested.
Lead optimization: the result where Claude questioned the measurements
BigHat pairs models with an automated lab. Claude was connected to BigHat's lab data system, Reccy, and proposed changes to two 5, partial antibodies used as drug building blocks. One targets cancer cells. The other is meant to keep a drug in the body longer. The goal was to improve three properties together: binding, heat stability and resistance to clumping. The lab made and tested about 1,010 variants in about 10 weeks.
Partway through, the team suspected the standard stability and clumping metrics were misleading. Claude and an Anthropic researcher reanalyzed the raw data and proposed different metrics, plus a protocol change to test them. BigHat scientists refined and ran it, and the main predictions held. Against the originals, the best variants bound more than 10x more tightly. The cancer-targeting lead stayed in solution at temperatures 18°C higher, and the longevity lead clumped more than 100x more slowly. Of all the cases, this one is the most interesting for practitioners. The model didn't just search; it helped question the assay. Humans still checked the work at each step.
Paperwork: manufacturing records and regulatory drafts
Enveda's manufacturing chemists spent up to 75% of their time on documentation. Enveda connected its manufacturing documents to Claude and had it flag possible quality issues. Across 1,061 pages, Claude caught all 11 issues human reviewers had found and flagged 7 more they had missed. Review time fell 89%, from over 25 hours to under 3. The post doesn't report false positives, which decide whether a flagging tool actually saves time. Claude also drafted two regulatory documents. One took four hours versus about three months by hand. Enveda's normal human review found one minor error in the other.
Operations: company creation and trial planning
The Column Group, a biotech investor, is starting Pleiades, a company meant to run more drug programs, and more complex ones, than a small team usually could, with Claude doing much of the work. The thinking is that cheaper discovery would let investors back riskier, more novel ideas. So far the only evidence is its first Claude "6," a packaged set of instructions and tools for judging whether a drug target is worth pursuing. It was evaluated on 45 target-disease pairs, and no results are given. Bristol Myers Squibb is using Claude to review the trial plans of an early-stage drug and estimate its odds of success. An executive says one prompt produced an analysis that would otherwise take days. Its accuracy isn't reported.
The in-house program
Mithril's founder, Cody Tranbarger, used Claude to manage contracts, logistics and experiments across 11 external labs. He took a candidate for one rare childhood muscular dystrophy from idea to promising mouse results in under 18 months. Now at Anthropic, he plans more programs. Anthropic says human trials could start "as soon as late 2028" if the candidates hold up.
How to read the numbers
Some caveats:
Nearly all of it is 7. Lab and mouse results often fail to carry over to people, and the post says plainly the work is early.
The baselines are weak. Most results are compared against the starting molecule or the standard approach, not against the same team using non-LLM computational tools. The post can't tell us how much of the gain came from Claude itself rather than from the partners' existing platforms and labs.
Self-reported by interested parties. Anthropic and its partners wrote the reports. The technical reports may contain more rigor than this summary does.
The post also mentions surrounding programs: Claude Science, a research workbench; credits for life scientists; and the 8, which lets vetted organizations use the strongest models with looser biosecurity safeguards, since those safeguards can block legitimate biology. There is also a call for guest investigators.
Open questions remain. Will any of these candidates survive human trials? And are the gains in speed and throughput Claude-specific, or something any capable agent connected to good lab data would deliver?
Remember
The most useful contributions were broad design search, reanalyzing lab data, and document review, not one breakthrough molecule.
BigHat's case stands out: Claude helped show the standard assay metrics were misleading, and lab tests confirmed it.
No result is benchmarked against non-LLM computational methods, so how much Claude itself contributed remains unclear.
Validation: 6, 54.8%. The model must show a vulnerability is actually exploitable. DeepMind's reason this matters: defenders have too many findings to fix and need to know which are dangerous. A working exploit settles that. This is also the most dual-use of the three skills, since it is useful to attackers too, and probably a big part of why the rollout is gated.
Remediation: Collinear's 7, 71.0%. CWE, the Common Weakness Enumeration, is the standard catalog of vulnerability classes, such as buffer overflows and SQL injection. The model must turn a confirmed finding into a working fix at the root cause.
Hugging Face Open ASR Leaderboard: first among open-weight models in English at 5.33% average WER (6). Within the model card's comparison, it has the lowest multilingual WER (4.62% mean over seven languages) and the lowest long-form WER (10.98%).
Artificial Analysis: 2.8% on AA-WER, the independent ranker's error index. That ties for first among open models with a model that has 24B active parameters, about six times as many. On AA-AgentTalk, a held-out set of voice-agent speech that makes up half the index, it scores 1.8% against 2.3% for the next open model. NVIDIA says it also beats several widely used proprietary APIs on AA's all-model chart.
David AI diarization track, 21 languages: lowest diarization error rate (DER, the share of speech time attributed to the wrong speaker or missed) of 14 systems, at 18.26%. NVIDIA's dedicated Nemotron 3 Diarization model is second at 18.38%. On the combined transcription-plus-speaker track, Transcribe is first among open models and second overall.
VoiceArena Diarization-Bench: English full sessions with overlapping speech and 2–8 speakers. Transcribe has the lowest 7 of 14 systems at 32.4%. cpWER is WER computed after matching each predicted speaker to the best reference speaker. Next is NVIDIA's own Parakeet + Nemotron 3 Diarization at 34.2%. Deleted words were 8.32% of the reference, against a median of about 31%. Dropping speech is a common failure on overlapping talk.
Throughput: 100 hours of audio in under three minutes on one RTX PRO 6000 Blackwell GPU. That figure uses the not-yet-released 8 (4-bit floating-point) quantized version, on 30-second clips at high concurrency, so it is a best case. The full-precision checkpoint runs on vLLM and SGLang-Omni today.
Form
Example
Normalized, verbatim (tn_raw)
I paid, uh, twenty-five euros.
9, with disfluencies (itn_raw)
I paid, uh, €25.
Inverse text normalization, clean (itn_clean)
I paid €25.
Missing denominators. There are no false-positive rates for Enveda's flags, no hit rate per design for Divergence, and no accuracy figure for the BMS forecasts.
