DeepSeek moves off Nvidia and open-sources its serving engine. Elsewhere: a vocabulary rebuilt to cut serving cost, ChatGPT's ad system and parallel research agents.
Picked from 32 new articles across 24 sources · arrow keys for other days
Today in 30 seconds
30s left
DeepSeek ported its kernels to Huawei's Ascend 950PR: matrix kernels reach 99.7% of peak, but absolute speed lands between A100 and MI300X. story 1 ↓
DeepSeek open-sourced DeepInfer, its production inference engine, reporting 155 trillion tokens a day and 96% cache hits. It withheld the cache and routing layers. story 2 ↓
Dharma-AI rebuilt a 4B essay grader's vocabulary around serving cost. Average responses fell from 350 to 101 tokens and inference cost by 77%. story 3 ↓
OpenAI explained ChatGPT's ads: a tool-calling LLM runs a separate ranking pipeline beside the answer. If it's late, no ad appears. story 4 ↓
OpenAI gave each mouse-brain gene knockout its own agent: 700+ papers cited versus 11 for one agent. Blinded experts preferred their reports 75% of the time. story 5 ↓
Microsoft Research reports a proof of the 1995 KLS conjecture, which governs how efficiently randomized algorithms sample and estimate high-dimensional volumes.
1/5SemiAnalysis· Bryan Shan· Oct 21
DeepSeek ports its kernel libraries to Huawei Ascend chips
DeepSeek added Huawei Ascend support to six of its open-source GPU libraries. SemiAnalysis's benchmarks find Huawei's software close to Nvidia's in efficiency, while the hardware still trails Nvidia by years.
Why it matters · A credible non-Nvidia stack for a frontier lab would change where open-weight models get trained and served, and which kernels and tools you end up depending on.
Know this first
Context from the editor, not the article · 23 sec
1 · Why DeepSeek needs Huawei
US export controls stop Chinese labs from buying Nvidia's best accelerators. Huawei's Ascend chips are the main domestic alternative. Their software stack, CANN, plays the role that CUDA plays for Nvidia.
2 · DeepSeek's kernel libraries
In 2025 DeepSeek open-sourced low-level GPU code for Nvidia only. DeepGEMM handles matrix multiplies, FlashMLA handles attention, and DeepEP handles routing tokens between experts in mixture-of-experts models.
3 · Utilization vs raw speed
Benchmarks often report what fraction of a chip's theoretical peak a kernel reaches. A slow chip can score high utilization and still lose on absolute throughput, which is why both numbers matter.
2/5Latent Space· Oct 21
DeepSeek open-sources DeepInfer, the inference engine behind its production API
DeepSeek released the code for DeepInfer, the engine that serves its V4.1 Flash model, along with production figures. Most of the efficiency comes from caching. vLLM says its own engine nearly matches DeepInfer's decode speed.
Why it matters · It is the most detailed public look yet at how a frontier-scale API is served and what it earns. The scheduling and caching lessons carry over to anyone running LLM inference.
Know this first
Context from the editor, not the article · 31 sec
1 · Prefill versus decode
Serving an LLM has two phases. Prefill reads the whole prompt in parallel and is limited by compute. Decode produces one token at a time and is limited by memory bandwidth. Large services often run the two phases on separate machines.
2 · KV cache and prompt caching
While reading a prompt, the model computes intermediate key/value tensors for every token. If the stored KV for a repeated prefix is kept, a later request can skip recomputing it. That is why API providers charge much less for cached input tokens.
3 · Mixture-of-experts serving
DeepSeek's models are mixture-of-experts: each token passes through only a few of many expert sub-networks. Expert parallelism spreads those experts across many GPUs. Uneven routing means some GPUs end up waiting on the busiest experts.
3/5Hugging Face· Oct 21
Rebuilding a small model's vocabulary cut its inference cost by 77%
Dharma-AI rebuilt the tokenizer of a fine-tuned Qwen3-4B essay grader, choosing tokens by serving cost rather than frequency. Measured inference cost fell 77% and grading quality rose, with the base model and training recipe unchanged.
Why it matters · If you fine-tune small open models for one narrow task, the inherited tokenizer may be a bigger cost lever than quantization or batching, and you set it before training starts.
Know this first
Context from the editor, not the article · 28 sec
1 · What a tokenizer does
Models read and write in tokens, chunks of text chosen by a tokenizer. General-purpose tokenizers such as Qwen's are trained on many languages and domains, so text from one narrow domain often splits into many small pieces.
2 · Prefill versus decode
Inference has two phases. Prefill processes the whole prompt in one parallel pass and is limited by compute. Decode generates one token per full pass and is limited by memory bandwidth, so output tokens cost more.
3 · The LM head
A model's final layer, the language-model head, scores every vocabulary entry at every generated token. It holds one row of weights per entry, so a bigger vocabulary means more data to read on every decode step.
4/5OpenAI· Oct 21
How OpenAI built ChatGPT's ad system around an LLM that calls tools
OpenAI describes ChatGPT's ad system: a post-trained model reads the whole conversation, then calls retrieval, filtering, scoring and auction tools in a separate service. OpenAI says it reached $1 billion annualized revenue within 200 days.
Why it matters · It's a production example of an LLM coordinating a classic ranking pipeline under strict time limits, with the model kept separate from the product it serves.
Know this first
Context from the editor, not the article · 27 sec
1 · How search ads usually work
A query or page view triggers one request. The system retrieves candidate ads, ranks them by predicted clicks and relevance, then runs an auction that weighs bids against quality to pick the winner and set its price.
2 · Embeddings and nearest-neighbor search
Text is turned into vectors (embeddings) where similar meanings sit close together. Approximate nearest-neighbor (ANN) search finds nearby vectors quickly across millions of items, giving up a little accuracy for speed.
3 · Tool-calling agents
An LLM can be post-trained (further trained after pretraining) to call external functions in a loop: it chooses a tool, reads the result, and decides what to do next until it reaches a goal.
5/5OpenAI· Oct 21
OpenAI runs thousands of parallel agents over a mouse-brain gene-knockout atlas
OpenAI, PerturbAI and Scripps Research gave one AI agent per gene disruption in a mouse-brain dataset. The agents finished in 32 minutes, matched earlier human findings, and proposed new disease leads that still need lab testing.
Why it matters · It is a controlled test of a design question every agent builder faces: one long-context agent, or many narrow ones? The answer: splitting the work kept each agent's research thorough.
Know this first
Context from the editor, not the article · 29 sec
1 · CRISPR perturbation atlases
Scientists use CRISPR to disable thousands of genes, one per cell, then sequence each cell's RNA to see which other genes change activity. The result is a huge table: disrupted gene × cell type × gene-expression response.
2 · The interpretation bottleneck
Running these screens is now cheap compared with understanding them. Explaining one knockout's effect means reading the literature by hand. With thousands of knockouts, researchers usually chase only the strongest signals and leave the rest unexamined.
3 · Fan-out agent designs
A common way to build agents: split a big job into independent subtasks, run a separate model instance on each, then merge the outputs. This keeps any one context window from overflowing and lets the subtasks run in parallel.
Utilization across 1,055 BF16 GEMM shapes: Ascend 950PR versus H100, against each chip's own attainable ceiling.1 / 4
The news
On September 30, DeepSeek added support for Huawei's Ascend accelerators to six of its open-source repositories:
1, a kernel-writing language.
DeepGEMM, for matrix multiplication.
2, for expert-parallel communication.
FlashMLA (attention), TileKernels and DeepSelect.
SemiAnalysis describes it as a trade. Huawei has chips but no frontier lab to prove them. DeepSeek has frontier models but can't buy Nvidia hardware. SemiAnalysis expects DeepSeek to run mainly on Huawei by 2027 or 2028.
One caveat: at the time of writing, only TileLang worked. The others need 3 9.2.0 (Huawei's counterpart to Nvidia's CUDA), which hadn't shipped yet.
How good is the chip?
SemiAnalysis benchmarked an Ascend 950PR, a variant aimed at prompt processing, with 28 matrix ("Cube") cores and 128 GiB of high-bandwidth memory (4). They ran 1,055 BF16 matrix-multiply shapes through stock PyTorch, which calls Huawei's own kernels.
Compute-bound shapes: the best kernels hit 99.7% of the attainable ceiling.
Memory-bound shapes: the median was 41.5% of attainable, against 46.8% for Nvidia's H100.
Memory bandwidth: beats the H100 on small copies and nearly matches it on large ones.
Absolute throughput: lands between Nvidia's A100 and AMD's MI300X.
SemiAnalysis's verdict is that the hardware may be about five years behind while the software is at the frontier.
Co-design in both directions
Ascend's design is reshaping DeepSeek's code:
FlashMLA cache layout. FlashMLA now stores quantized cache values next to their scaling factors, so one token takes one memory transfer instead of two.
Attention accumulator. Ascend's accumulator buffer can't rescale results in place, so DeepSeek keeps the running attention output in the vector unit instead.
TileLang's Ascend backend is merged into the main repository. Its AutoSchedule pass overlaps data loads with compute and cut single-operation time by up to 40% at small sizes. It also inserts synchronization automatically.
DeepEP-Ascend keeps DeepEP's Python API and swaps in Huawei's interconnect underneath. Its docs admit the combine step and network traffic compete for memory bandwidth, and Huawei plans firmware fixes.
Running the new model
Huawei profiled DeepSeek-V4.1-Flash on 32 Ascend 950DT chips. Each step runs 40 main-model layers, three draft blocks for 5 (cheap draft tokens verified in one pass) and five sampling rounds. In a what-if scenario, doubling matrix throughput speeds up a step by only 1.32×, because so much of each step is other work.
The catch
The setup may work well for DeepSeek and badly for everyone else. Most of the tuning targets Huawei's Arm-hosted SuperPods. On x86 hosts, vLLM-Ascend lacks DeepGEMM support, fails to start out of the box and has no continuous-integration testing.
The release
On September 30, just before China's national holidays, DeepSeek added support for Huawei's Ascend accelerators to six of its open-source repositories:
1, a language for writing GPU kernels.
DeepGEMM, for matrix multiplication.
2, for expert-parallel communication.
TileKernels, FlashMLA (attention) and DeepSelect.
In 2025 several of these shipped for Nvidia only.
SemiAnalysis calls the arrangement a symbiosis. Huawei has hardware and a native software stack, but no frontier lab to prove the chips can train and serve top models. Its own Pangu models don't fill that role. DeepSeek has the models but can't buy the compute it wants. SemiAnalysis predicts DeepSeek will run mainly on Huawei's stack by 2027 or 2028.
One caveat up front: at the time of writing, only TileLang actually worked. The other five need 3 9.2.0 (CANN is Huawei's counterpart to Nvidia's CUDA) and a C++20 compiler, and neither was available yet.
The chip, probed
SemiAnalysis tested an Ascend 950PR_9579 with 28 "Cube" cores and 128 GiB of 4, the high-bandwidth memory stacked beside the chip. Cube cores do matrix math; separate Vector cores do elementwise work. The 950PR is aimed at prefill (processing the prompt) and recommendation workloads. Its sibling, the 950DT, has more cores and more memory bandwidth.
Huawei sells many 950 variants. Too few dies and HBM stacks meet the full spec, so Huawei bins both compute and memory and pairs weaker memory with weaker compute. Nvidia bins only compute.
Microbenchmarks mapped the chip's layout:
Latency steps. Pointer-chase latency stayed near 134 ns up to 18 blocks, rose to 182 ns at 56 and jumped to 391 ns at 64. That points to a second scheduling wave beyond 56 Vector cores.
Two dies. Per-core memory-timing profiles split into two groups, matching the chip's two dies.
Shared resources. Contention tests suggest subsystems of one Cube core plus two Vector cores.
Each chip also carries small Arm CPUs on board, called 5. On this unit, 7 were configured, 6 visible and 5 executing. Their MIDR identifiers suggest they match the middle cores of Huawei's Kirin 9000S/9010 phone chips. They are weak, but they avoid round trips to the host for tiny jobs. In vLLM-Ascend, a version of the vLLM inference server for Ascend, they run the planning, top-2,048 index selection and gather steps of DeepSeek's sparse attention.
Benchmark results
SemiAnalysis tested 1,055 BF16 matrix-multiply (6) shapes taken from open models. They used stock torch.matmul, which calls Huawei's own kernels, not DeepSeek's code.
Compute-bound shapes. Huawei's best kernels reached 99.7% of the attainable ceiling. SemiAnalysis credits part of this to dedicated operand buffers (L0A/L0B) and an enlarged accumulator buffer (L0C), which let data movement and math overlap.
Memory-bound shapes. The median was 41.5% of attainable versus H100's 46.8%.
Power limits. The H100 was power-capped, drawing about 690–698 W of 700 W at 1,335–1,470 MHz against a 1,830 MHz max. The Ascend ran around 600 W at 1,500–1,550 MHz against a 1,650 MHz reference.
Absolute throughput. The 950PR lands between the A100 and the MI300X.
Memory bandwidth. It beats the H100 on small copies and nearly matches its utilization on large ones. A dip in between comes from switching to a copy kernel with higher startup cost.
The authors give two caveats. Kernel-level utilization doesn't guarantee high 7 (model FLOPs utilization, the useful work done in real training), because collectives and checkpointing are hard to overlap. And since China's electricity is cheap, performance per total cost is the fairer comparison; SemiAnalysis defers that to a future article. Their summary is that the hardware may be about five years behind, A100-class, while the software is "right at the frontier."
Engineering runs both ways
The hardware is already changing DeepSeek's software:
FlashMLA cache layout. FlashMLA used to store quantized KV-cache values apart from their scaling factors, so fetching one token on Ascend took two transfers. The new layout stores them together, which cuts requests to the memory transfer engine.
Attention accumulator. The Cube's L0C buffer can't rescale accumulated outputs in place, so DeepSeek keeps the running attention output in the Vector unit's buffer instead.
TileLang's Ascend backend is merged into the main repository rather than living in a Huawei fork. Developers still choose the math, tile sizes and data placement. The compiler handles coordination between Cube cores, Vector cores and data transfers. Its AutoSchedule pass plans overlap and buffering and cut single-operation time by up to 40% at small sizes. It also inserts and prunes synchronization automatically, and checks conditional branches before assuming an operation has finished.
DeepEP across the rack
DeepEP-Ascend ports DeepEP V2.5 behind the same Python API. Underneath, it calls 8 (Huawei's remote memory interconnect) and 9 (Huawei's collective-communication library). Huawei released EP16 dispatch/combine benchmarks on a 950DT SuperPod. The comparison isn't one-to-one: Huawei's dispatch runs as three sequential operators, while GB300's is one fused call.
In July 2025 a DeepEP maintainer declined Huawei's proposal to host the port under the deepseek-ai organization. It now lives under DeepSeek's own organization.
The docs admit that the combine step and URMA traffic compete for HBM bandwidth. Huawei plans firmware fixes. SemiAnalysis calls this co-design in practice: the model exposes a bottleneck, and the platform's priorities respond.
Profiling V4.1-Flash
DeepSeek gives Huawei model weights early so optimized support ships on day one. Huawei published a profile of one rank of DeepSeek-V4.1-Flash on 32 Ascend 950DT chips. Each step runs:
40 layers of the main model and its LM head (the final layer that turns hidden states into token scores).
3 "DSpark" drafter blocks and a draft LM head. These serve 10, where cheap draft tokens are checked in one pass of the full model.
5 sampling rounds.
Kernels that fuse computation with communication leave only 4 CPU-launched HCCL operations per step, against 43 expert dispatches.
V4.1-Flash is DeepSeek's first model to use 11. The article doesn't define it beyond noting it takes token IDs as input. That lets the lookup be fetched asynchronously during layer 0. Offloading it to DRAM costs nothing, but a PCIe 4.0 SSD slows things down.
The main stream is busy 88.8% of the window. SemiAnalysis's what-if: doubling Cube throughput, with everything else held fixed, gives only a 1.32× step speedup. They label this a scenario, not a prediction. Fusing quantization is another open opportunity.
What's left open
Liang Wenfeng, DeepSeek's founder, is quoted from an unverified May investor transcript. He said four Huawei cards match one Nvidia card, that Huawei is about two years behind to market, and that the software gap "will eventually disappear."
The open question is whether this works for anyone besides DeepSeek. Most of the optimization targets Arm-hosted SuperPods. On x86 hosts, vLLM-Ascend 0.26.0rc1 has these gaps:
No DeepGEMM support.
Hardcoded Arm headers block startup.
CPU binding is skipped.
There is no NPU-plus-x86 continuous integration.
Remember
Huawei's own matmul kernels reach 99.7% of the attainable ceiling, but the chip's absolute throughput sits between A100 and MI300X.
Ascend's hardware quirks are already changing DeepSeek's code, from FlashMLA's cache layout to where the attention accumulator lives.
Most of the tuning targets Arm-hosted SuperPods; on x86 hosts, vLLM-Ascend still breaks or lacks features.
Latent Space's per-GPU production math: prefill and decode throughput, cache hit rate, and resulting revenue.1 / 3
DeepSeek has released DeepInfer, the engine that serves its V4.1 Flash model, as Apache-2.0 open source. It started as a customized fork of 1, the popular open-source serving engine, and targets NVIDIA Blackwell GPUs and Huawei Ascend 950 chips. The release came with an engineering report that matters as much as the code.
What DeepSeek claims
One cluster serves 155 trillion tokens a day, with a 96% input-cache hit rate, 400 ms average 2 and about 230 tokens/s per user. DeepSeek says it earns $38 per accelerator-hour at off-peak prices. The token count mostly reflects cached prompt tokens, which are cheap to serve. These figures are self-reported.
How it works
Prefill (reading the prompt) and decode (generating tokens) run on separate machines. Prefill uses eight-accelerator boxes; decode uses 64–128-accelerator supernodes that spread the model's 3 layers widely so batches stay large. Several design choices stand out:
Prefill work is assigned by , not raw token count.
What's missing
The parts that turn the engine into a service are not in the release: KV-cache management, 7 and replica load balancing. DeepSeek says they are still tied to internal infrastructure.
The pushback
The vLLM team reports decode throughput within 6% of DeepInfer on GB200 under a matched configuration, with reproduction recipes. That covers decode only, not DeepSeek's caching or end-to-end service, and it is a competitor's own measurement.
The money
Latent Space works the $38/hour figure into a ">80% inference margin" and an 81-day payback. With accelerator costs of $5–10/hour, the accelerator-only margin is about 74–87%. Those numbers leave out the SSD fleet, networking, idle capacity, staff and training, and they rely on a very cache-heavy workload. DeepSeek never said which hardware produced the $38. Treat the margin as a scenario, not a disclosure.
What was released
DeepSeek has open-sourced DeepInfer under Apache 2.0. It is the inference engine behind DeepSeek V4.1 Flash, released on day one of PyTorch Conf. DeepInfer began as a customized fork of 1, the widely used open-source serving engine, and credits SGLang and NVIDIA's TensorRT-LLM. It supports distributed inference, 2, structured output, expert-parallel load balancing, asynchronous scheduling, batch-invariant inference and deterministic sampling. It targets NVIDIA SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs; you pick the platform at build time.
A second repository, deepseek-recipe, is a Rust library with Python bindings. It translates between the Chat Completions, Responses and Messages API formats, parallelizes tokenization, and tells real control tokens apart from identical-looking strings a user types. That last part stops user text from being parsed as structural delimiters. It is not a general prompt-injection defense.
DeepSeek calls the release "fully open source", but the pieces that make it a production service are missing. Those are KV-cache management, 3 (running the two phases on separate machines) and load balancing across data-parallel replicas. DeepSeek says they are still tied to internal infrastructure. Its storage system, 4, was open-sourced earlier.
The production numbers
The accompanying engineering report matters as much as the code. DeepSeek reports, for a single cluster:
155 trillion tokens served per day
a 96% input-cache hit rate
400 ms average 5
230 tokens/s generation per user, 300 for coding
$38 of revenue per accelerator-hour at off-peak prices ($0.15/$0.60 per million input/output tokens)
Latent Space's translation adds about 1.5 million cache-inclusive input tokens/s per prefill GPU and 14,000 output tokens/s per decode GPU, across roughly 2,400 GPUs. All of these are first-party claims, not audited figures. The 155T figure mostly counts cached tokens, which are cheap to serve. It is not 155T newly computed tokens.
How the scheduling works
DeepInfer runs vision encoding, prefill and decode as separate stages, each scaled against its own bottleneck. The recommended setup puts prefill on ordinary eight-accelerator machines and decode on 64–128-accelerator "supernodes". Decode uses wide expert parallelism, spreading the 6 layers across many devices so batches stay large. The disclosed Blackwell configuration appears to use no tensor parallelism at all, unlike default vLLM and SGLang recipes. One commentator traces this to constraints from DeepSeek V3's H800 era.
Prefill is scheduled by estimated compute, not token count. Attention cost grows faster than prompt length, so DeepInfer estimates each request's cost and varies chunk sizes to fit a time budget.
Prefill ignores cache affinity. All cached prefixes live in shared storage, so any free worker can take a request. It doesn't have to go to the worker that last saw that prefix.
Decode balances two signals: how many requests are running (scheduling pressure) and how much KV each has accumulated (memory pressure).
Popular experts are replicated and redistributed periodically to cut waiting caused by routing skew. This costs GPU memory (HBM), with diminishing returns.
CPU scheduling overlaps GPU execution, removing idle gaps between decode steps.
Inference engineer Su Jin Yan cautions that wide expert parallelism in prefill only pays off at high traffic. He prefers pipeline parallelism for small deployments.
How the cache works
The persistent KV cache sits in a separate 3FS storage cluster with about 100 PiB of SSD. Agent traffic is retained for three days and chatbot traffic for seven. OpenAI, by comparison, just announced optional 12-hour caching on top of its 30-minute default. Long retention suits agents that resend a growing history of instructions, code and tool outputs on every turn. Su Jin Yan estimates the long tail of retention adds only 0.5–1 percentage point of hit rate. Cache blocks load asynchronously, overlapped with layer execution. They use GPUDirect RDMA transfers rather than the CUDA copy engine, which handles many small block copies poorly.
The model also uses 7 (SWA): some layers only attend to recent tokens. That state lives in host memory and expires sooner. When it's missing, 8 rebuilds it from only the latest 128 tokens, while reusing the persistent cache for everything earlier. The result is an approximation rather than an exact rebuild. DeepSeek says this nearly halves total prefill compute. Whether the approximation is safe for other model architectures is an open question.
For speculative decoding (a cheap draft proposes several tokens and the main model verifies them in one pass), DeepSeek's DSpark predicts how likely drafts are to be accepted and adjusts how many to verify. It reports 6% more throughput than always verifying five tokens. That gain is measured against speculative decoding, not against plain decoding.
Did vLLM match it?
The vLLM team says it got within 6% of DeepInfer's published decode throughput on GB200 under a matched configuration, and published reproduction recipes. That undercuts the claim that only DeepSeek can serve this efficiently. But it is decode-stage parity only. It says nothing about the 100-PiB cache, routing or end-to-end latency under real traffic, and the result is self-reported by a competing engine team. Su Jin Yan thinks experienced inference engineers will get more from the engineering report than from the repository.
The margin math
Revenue of $38 per accelerator-hour is not profit. Assuming $5–10 per accelerator-hour in cost, the accelerator-only margin is about 74–87%. Latent Space assumes $5–8 and calls it ">80%".
A worked estimate by @zephyr_z9 assumes the billed tokens are 97% cached input, 2% uncached input and 1% output. One billion tokens then needs roughly 875 GPU-seconds (250 for prefill, 625 for decode). At $3–6 per GPU-hour that costs $0.73–1.46, against $11.91 of off-peak revenue. The author later corrected the estimate: the ideal case implies $49/hour, and DeepSeek reports $38, so production runs at about 80% of benchmark. The revised margin range is 84–96%.
None of these numbers cover the SSD fleet, CPUs, networking, spare capacity, staff or training. All of them depend on an unusually cache-heavy token mix. Latent Space's headline extrapolations ($22M a day, roughly $166B per gigawatt, an 81-day payback) are scenario math, not disclosures. DeepSeek also never said which hardware produced the $38 figure. Blackwell-shaped configurations hint at NVIDIA, but Ascend support is confirmed only as compatibility, not as comparable production efficiency.
Remember
155T tokens/day mostly counts cached tokens. The 96% cache hit rate does most of the economic work.
Shared cache storage lets any worker serve any request. DeepSeek schedules prefill by compute cost, not token count.
vLLM says it nearly matches DeepInfer's decode speed. The unreleased cache and routing layers are the real gap.
A span the grader writes in every response: 12 tokens with Qwen's tokenizer, 1 with VocabX.1 / 4
Dharma-AI rebuilt the tokenizer of a production essay grader built on 1, a 4-billion-parameter open model, and cut measured inference cost by 77%. The base model, data and training recipe stayed the same. The method, called VocabX, comes from a paper that is still under review, so the numbers may change.
The waste in an inherited vocabulary
The grader scores Portuguese essays and returns JSON with a grade and comment per competency. It still used Qwen's general tokenizer, with 151,669 entries covering many languages. That caused two problems:
Fragmentation. The text the grader writes on every request, like the JSON envelope and competency names, was split into many tokens. One span took 12 tokens instead of 1.
Excess. At every generated token, the model's final layer (the 2) scored all 151,669 entries, including Chinese tokens it never uses. That layer holds almost 10% of Qwen3-4B's parameters and is read in full on every step.
Output tokens cost more
Reading the prompt (3) happens in one parallel pass. Generating (4) takes a full pass per token and is limited by memory bandwidth. On the grader, a generated token cost 7.4× an input token. The ratio varies with hardware and batch size.
What VocabX does differently
Earlier methods, including fast vocabulary transfer, multi-word tokenization and Liquid AI's in-place tokenizer expansion, pick new tokens by frequency and work on only one layer. VocabX makes three changes:
It weights by cost. A token saved in the response counts 7.4 times one saved in the prompt.
It works on two layers. It rebuilds how domain words split into sub-word pieces, and it turns repeated spans (phrases, whole lines, JSON envelopes) into single "5". Five search routines, called "miners", find the candidates.
It estimates what sampling misses. Candidates are scored on a 400-document sample, and phrases absent from it usually score zero. VocabX estimates their value instead. On contracts, that alone raised token reduction from 44.7% to 52.8%.
Results
Seven grader versions differing only in tokenizer each ran 500 requests on an RTX 5090. With VocabX:
average response length fell from 350 to 101 tokens
cost fell 77.0%
the quality score rose from 0.703 to 0.810
the vocabulary shrank to 64,767 entries
One early version merged each grade into its JSON macro token, and quality dropped to 0.689. Giving grades their own token fixed that. The authors' point: don't compress the span that carries the model's actual decision.
Across five public domains and three model families (Qwen3, Llama-3.1, Gemma-3), VocabX beat the reference methods on compression in all 15 combinations. Its cost advantages there are estimates, not measurements.
The post doesn't explain how new token embeddings are initialized, or why quality improved rather than just holding steady.
Dharma-AI runs a production grader for Portuguese essays built on 1, a 4-billion-parameter open model. In a Hugging Face post previewing a paper that is still under review, the team describes VocabX, a method for rebuilding a fine-tuned model's vocabulary around what each token costs to serve. The base model, data, training recipe and seed stayed the same. On the grader, measured inference cost fell 77.0% and quality went up. The team warns that the numbers may change before the paper is published.
The problem: a vocabulary built for everyone
The grader takes an essay, the prompt and supporting texts, and returns JSON with a grade and a comment for each competency. It still used Qwen3-4B's original tokenizer, which has 151,669 entries and was built for many languages and kinds of text. The authors identify two kinds of waste.
Fragmentation. The text the domain repeats most gets split into many pieces. Every grader response opens with the same JSON envelope, competency names and evaluation phrases. One such span takes 12 tokens with the base tokenizer and 1 with VocabX.
Excess. The model scores every vocabulary entry at every step, including tens of thousands it never uses. A Portuguese-only grader still computes a probability for every Chinese token.
Why counting tokens isn't enough
The authors argue that the token count is not the bill. In 2, the pass over the input, each weight loaded from GPU memory is reused across all input tokens, so the limit is compute. In 3, each generated token needs its own full pass and also reads the 4, the stored attention state for the context so far. The GPU spends most of that time waiting on memory. Batching spreads the weight reads across requests, but it doesn't remove the gap. That matters most for small models served at moderate batch sizes. On the grader, a generated token cost 7.4 times an input token. The ratio depends on hardware, batch size and context length, but the direction holds.
The authors' analogy: a shipper charges per box, and outgoing boxes cost seven times more than incoming ones. A method that compresses input and output equally misses most of the savings.
Vocabulary size is a third cost. The 5, the final layer that scores every possible token, has one row per vocabulary entry. In Qwen3-4B it holds almost 10% of the parameters, and the whole matrix is read for every generated token. Adding tokens can therefore lower the count and still raise the bill. VocabX doesn't put this term in its selection criterion. It controls size another way: it replaces the base sub-word vocabulary with a smaller domain-specific one and adds multi-word tokens under a fixed budget. The grader's output layer shrank from about 152,000 rows to 64,767.
What earlier methods missed
Three published methods already adapt vocabularies:
fast vocabulary transfer (FVT)
multi-word tokenization (MWT)
Liquid AI's in-place tokenizer expansion, which continues 6 training from the original vocabulary (BPE, byte-pair encoding, builds tokens by repeatedly merging frequent pairs)
The authors say all three pick tokens by frequency, so a phrase common in the prompt beats a rarer but more expensive phrase in the response. Each also works on only one layer: either how words split into sub-word pieces, or merging whole phrases. Long repeated spans like a JSON envelope are hard to reach with sub-word merges alone.
How VocabX works
Cost-weighted selection. Each token a candidate eliminates is weighted by the price of the field it appears in: system prompt, user message or model response. On the grader, a token saved in the response counts 7.4 times one saved in the input.
Two layers, one criterion. At the sub-word layer, VocabX rebuilds how domain words are split. At the multi-word layer, it turns repeated spans, from short phrases up to whole lines and JSON envelopes, into single "7". Both layers use the same cost criterion, each with its own budget. Five "miners", automatic searches for repeated spans, find the candidates. Each covers one pattern: long words, whole lines, expressions, full sentences, and punctuated structures like JSON and tags. The structure miner contributed most on average across eight corpora, and on code it alone accounted for 17.5 points. Without all five miners, token reduction on the grader would drop by 55.6 points.
Estimating what the sample misses. A greedy selection needs each candidate's remaining gain after earlier picks. Recomputing that over the full corpus tens of thousands of times is impractical, so the usual approach measures on a sample, and candidates absent from the sample score zero. Sampling misses a lot. A phrase that appears 37 times in 50,000 documents shows up in a 400-document sample only about 26% of the time. In a contracts corpus, only 16.6% of candidates appeared in the sample. VocabX splits the gain into two parts. Total corpus savings are counted exactly. Overlap with tokens already chosen is estimated, and for unseen candidates the estimate comes from how the sampled ones behave. With the same sample and budget, this raised token reduction on contracts from 44.7% to 52.8%.
Results on the grader
The team trained seven versions of the grader that differed only in tokenizer. Each ran the same 500 requests on one RTX 5090 using 8, a popular open-source inference server, and cost came from measured runtime.
Average response length: 350 → 101 tokens
Inference cost: −77.0%
Bench score, combining grading accuracy and detection of off-topic or rule-breaking essays: 0.703 → 0.810
The first version went too far. It fused each competency's grade with its surrounding JSON into one macro token, which scored 0.689, below production. With the grade baked in, every grade-and-structure combination became a separate token, and the model had to pick both at once. Giving grades their own tokens added two tokens per response but raised the score to 0.810. It was also cheaper in the measurement: 77.0% savings against 73.3% for the fused version. The authors' lesson is not to compress the span that carries the decision the model exists to make.
Beyond the grader
On five public domains (contracts, medical papers, scientific papers, news, Python) and three model families (Qwen3, Llama-3.1, Gemma-3), VocabX compressed more than the strongest reference method in all 15 combinations, by 1.2 to 9.3 points. The pruned VocabX also had the lowest estimated inference cost in all 15, with average leads of 10.2 points on Qwen3, 7.5 on Llama-3.1 and 13.6 on Gemma-3. On contracts it reached 62.6% token reduction with 113,426 entries, against 61.8% for the best in-place BPE setup with 207,803.
What's left open
The paper isn't out. The post doesn't explain how embeddings for new tokens are initialized, how much fine-tuning they need, or why quality rose so much over production rather than just holding steady. Grader costs come from one GPU at one workload, and the public-domain costs are estimates, not measurements. The 7.4× weight is specific to this setup and would need re-measuring elsewhere. The method also suits workloads with highly repetitive, structured output. A free-form chat model would likely gain less.
Remember
For fine-tuned small models, output tokens can cost several times more than input tokens; on this grader, 7.4 times.
Weighting candidate tokens by serving cost beats picking them by frequency, especially for repetitive structured output like JSON.
Don't fold the model's key decision into a macro token: merging grades into JSON lowered the grader's quality score.
OpenAI has explained how ChatGPT chooses ads. A custom, 1 LLM (a model given further training after pretraining), which OpenAI calls the 2, reads the conversation and runs the ad pipeline by calling tools. It runs in a service completely separate from the one that writes ChatGPT's answers. OpenAI says ChatGPT Ads reached a $1 billion annualized revenue run rate within 200 days of launch, with more than 100,000 advertisers. The post gives no relevance, latency or click-through figures.
Why the usual pipeline doesn't fit
Search ads react to one event, such as a query or a page view: fetch candidates, rank them, run an auction, show the winner. Chat has no single moment like that. Intent builds over many turns. A question about training for a race becomes a question about shoes several messages later. If the system acted on the first message, it would target the wrong need.
Three design choices
Stateful sessions. The ad system follows the conversation as it develops and can remember earlier chats. Past conversations are used only if the user opts into personalized ads, and only "nonsensitive" information, which the post doesn't define.
Separation from the answer. Ads are chosen in parallel, using different models and code, so they can't influence what ChatGPT says. If the ad service is slow or down, answers are unaffected. Showing no ad counts as a normal outcome.
An LLM orchestrator. The model decides which tool to call next, repeats steps when results are poor, and starts again when new turns arrive.
How an ad gets chosen
The orchestrator first turns the conversation into one or more summaries of what the user might buy. For example, a Yosemite trip, a weather question and a question about a brand's new products might become two queries: "shoes for a hiking trip" and "new [brand] products good for travel." These are converted into 3 (vectors that capture meaning) and searched with 4 search across several indices.
Candidates are then filtered on budget, 5, targeting and policy, and scored for relevance, predicted engagement and past performance. The orchestrator only nominates ads for the auction. A separate auction tool picks the winner and sets the price.
A winning ad can still be dropped. The orchestrator may judge it unsuitable for this particular conversation, or it may arrive after ChatGPT's answer is ready. ChatGPT never waits for ads.
Feedback
Strong reasoning models grade the system's results and train the cheaper orchestrator. OpenAI says human signals, such as clicks, hides, surveys and advertiser feedback, steer it more. The evidence offered is commercial: OpenAI publishes no evaluations comparing this design with a conventional ranking stack.
OpenAI has published an engineering account of the ad system behind ChatGPT. Its central design choice is a custom, 1 LLM, meaning a model given further training after pretraining. OpenAI calls it the 2. It reads the conversation and runs the whole ad pipeline by calling tools: retrieval, eligibility filtering, scoring and the auction. The ad system runs in its own service, separate from the system that writes ChatGPT's answers. OpenAI says ChatGPT Ads passed a $1 billion annualized revenue run rate less than 200 days after launch, and that more than 100,000 advertisers have used it. The post gives no figures for relevance, latency, click-through rate or how often ads appear.
Why chat breaks the standard ad pipeline
Search and display ads respond to discrete events. A query or page view arrives, the system retrieves candidates, ranks them, runs an auction and shows the winner. ChatGPT has no such moment. People bring goals such as planning a trip, choosing business software or training for a race, and those goals take shape over many turns and sometimes across several conversations. In OpenAI's running example, a user asks generally about training for a race and only later asks about shoes. If the system acted on the first message, it would target the wrong intent.
OpenAI lists four risks of treating each message as a standalone ad request:
oversimplifying what the user wants
starting the ad flow before the need is clear
showing ads that interrupt rather than help
wasting compute by rerunning the whole pipeline every time the context shifts
Three design decisions
Stateful sessions. An ad session lasts for an eligible turn and keeps updating as the exchange develops. It also remembers context across conversations. If a user opts into personalized ads, the orchestrator can use "nonsensitive" information from past chats. The post doesn't define what counts as nonsensitive.
Strict separation from answer generation. Ad selection runs in parallel with ChatGPT's response, using different models and different code. OpenAI gives two reasons. The ad system can't influence what ChatGPT says. And ChatGPT doesn't depend on the ad system, so if ads are slow or down, answers are unaffected. Returning no ad counts as a normal, successful outcome.
An LLM as orchestrator. Instead of a fixed sequence of steps, the model decides which tool to call next and repeats steps until it finds an ad that serves both the user and the advertiser. As new turns arrive, it can start a fresh round of tool calls with the updated context.
The pipeline, step by step
Delivery is not the same as an impression
The ad service returns a bundle with two parts. One is the advertiser's creative: title, description, link and call to action. The other is a short, neutral, model-generated preamble explaining how the ad relates to the conversation. Keeping them apart preserves the advertiser's wording without putting ad copy in ChatGPT's voice. If the bundle arrives before the response finishes, the ad appears below it, labeled as sponsored. It counts as delivered only after the frontend displays it and acknowledges that.
Logging records exactly where each attempt stopped, for example:
the turn was ineligible
a session started but no retrieval was requested
retrieval returned nothing
candidates failed 6 (loading full ad details) or eligibility
an ad was delivered and later clicked
The point is to improve each stage separately. Better opportunity detection needs different evidence from better retrieval or more reliable delivery.
Self-improvement and its limits
OpenAI's strongest reasoning models judge the system's outputs and use those judgments to train the orchestrator. This lets smaller, cheaper models handle orchestration and retrieval. OpenAI says human signals matter even more: user clicks and hides, surveys, and advertiser feedback through Ads Manager. Its line is that "people shape taste." The team also credits 7, OpenAI's coding agent, with letting a small team build the system within a year, using guardrails for shipping code and plugins into the data infrastructure.
What's left open
This is a company describing its own revenue product, and the evidence is commercial: run rate and advertiser count. It gives no evaluations showing that the orchestrator beats a conventional ranking pipeline, and no latency budgets. It doesn't say how often ads time out, or how the "users first" balance is measured. Having a strong model grade and train a smaller one raises an obvious question about whose idea of "relevant" gets optimized, and the post doesn't address it. Next, OpenAI plans more dynamic formats, including letting users talk directly to businesses through ads.
Remember
The LLM only nominates ads; a conventional auction tool picks the winner and sets the price.
Ads are dropped if they arrive after ChatGPT's answer; the chat never waits for them.
Strong reasoning models grade outcomes and train smaller orchestrator models, but human feedback steers direction more.
OpenAI, the startup PerturbAI and Scripps Research ran thousands of AI agents in parallel over a large existing dataset of gene disruptions in mouse brains. The run took 32 minutes. The team estimates that humans reviewing the same data and literature one gene at a time would need months to a year. The agents run on 1, which OpenAI describes as a model for biology; the post doesn't explain how it differs from OpenAI's general models.
The problem
The dataset is a whole-brain 2 atlas. CRISPR, the gene-editing tool, was used to disable many genes across millions of mouse brain cells, and each cell's gene activity was then measured. That produces thousands of 3, meaning individual gene disruptions, each with its own downstream effects. Explaining any one of them means connecting it to papers scattered across many fields, so researchers usually chase only the strongest signals.
The method
Each agent took one perturbation. It analysed the cells' responses, searched the literature, and proposed explanations. Their reports were merged into a public "4" that links genes, effects, papers and candidate mechanisms. OpenAI calls this "5".
The useful result: one agent vs. many
The team held the model and resources fixed and gave the whole job to a single agent. As the workload grew, that agent did less research. At the largest workload it cited 11 references, against more than 700 for the distributed agents. In blinded comparisons, experts preferred the distributed-agent reports 75% of the time. The lesson for agent builders: narrow scoping keeps each agent's depth constant, while one agent with a huge task spreads itself thin.
OpenAI also says the agents reproduced earlier human findings, and that their ratings of novelty and value agreed with ratings from blinded experts. The post gives no figures for either claim.
Two leads
Tsc1 vs Tsc2. These two genes jointly restrain 6, a protein that regulates cell growth. In the cerebellum, a movement-coordination region, disrupting Tsc2 changed dozens of genes involved in communication between neurons, while disrupting Tsc1 barely changed anything. That fits the human disorder 7, where cerebellar problems track TSC2 mutations specifically.
Caveats
None of these leads has been tested in a lab. The post doesn't report whether the citations were accurate, how many experts judged or how they judged, or what the run cost in compute. The "months to a year" figure is the team's own estimate.
OpenAI, working with the startup PerturbAI and the Scripps Research institute, ran thousands of AI agents in parallel over a large existing dataset of gene disruptions in mouse brains. The run took 32 minutes. The team estimates that the same analysis and literature review, done by humans one gene at a time, would take months to a year. The agents run on 1, which OpenAI describes as a model for biology. The post says nothing about how it differs from OpenAI's general models. OpenAI calls the approach "2", and it has published the merged output as a searchable "3".
The problem: more data than anyone can read
The dataset is a whole-brain 4 atlas. CRISPR, the gene-editing tool, was used to disrupt many genes across millions of mouse brain cells. Each cell's gene activity was then measured to see how different brain cell types responded to each disruption. Experiments like this produce thousands of 5, meaning individual gene disruptions, each with its own pattern of downstream effects.
The hard part is interpretation. To say what a response means for human disease, someone has to link it to decades of papers spread across neuroscience, metabolism, genetics and clinical work. Scientists have limited time, so they go after the strongest signals. Subtle or off-topic responses tend to go unexamined, even though, as the post argues, some of an experiment's most interesting clues lie outside the question it was designed to answer.
How it works
Each agent got one perturbation. It analysed the cellular responses to that gene disruption, searched the scientific literature, and proposed explanations for what it saw. The individual reports were then merged into a single reasoning atlas that links each gene to its cellular effects, the relevant publications, and candidate biological mechanisms.
The post does not say exactly how many agents ran ("thousands"), what tools they had, how the literature search worked, or how reports were checked before merging.
The key experiment: one agent versus many
The most useful result for engineers is an ablation. The team asked whether a single agent given the whole dataset could do as well as many parallel agents, and held the model and the resources fixed. As the workload grew, the single agent searched less and less of the literature. At the largest workload it cited 11 references in total. The distributed agents cited more than 700. In blinded comparisons, human experts preferred the distributed-agent reports 75% of the time.
The pattern matches what people building agents tend to see. An agent with a huge task spreads its effort thin. It summarises instead of investigating, and literature lookups are the first thing it drops. Narrow scoping keeps each agent's depth constant however big the dataset gets. The post gives the result but not the mechanism behind it. It doesn't say whether the single agent hit context limits, tool-call budgets or time limits.
Validation claims
OpenAI makes three claims about quality:
The agents reproduced findings from earlier human analyses of the atlas.
They also produced extra leads beyond those analyses.
Their ratings of how novel and how scientifically valuable each finding was agreed with ratings from blinded human experts.
The post gives none of the underlying numbers. It doesn't say what fraction of known findings were recovered, how many experts judged, or how closely their ratings agreed.
Two example leads
Tsc1 vs Tsc2. These two genes work together to restrain 6, a protein that regulates cell growth, metabolism and aging. In most neuron groups, disrupting either gene had similar effects. In the cerebellum, a brain region that helps coordinate movement, they diverged. Disrupting Tsc2 changed the activity of dozens of genes involved in communication between neurons, while disrupting Tsc1 changed little. This matters because in 7, a human genetic disorder, cerebellar abnormalities are strongly linked to mutations in TSC2 specifically. Drugs that dampen mTOR treat some of the disease's tumors and seizures, but not its learning and behavioral problems. The agents' finding points to one neuron population and one set of changes to test: which of them existing drugs might correct, and which might need a new approach.
DCTN1 and cholesterol. DCTN1 helps transport cargo inside neurons. Mutations in it cause 8, a neurodegenerative disease with Parkinson-like symptoms. In the atlas, disrupting Dctn1 raised the activity of genes that make cholesterol. The agents connected this to research suggesting that when cholesterol transport fails, cells respond by making more. That leads to a testable hypothesis: DCTN1 loss may disrupt where cholesterol ends up inside neurons, that may contribute to neurodegeneration, and restoring normal distribution might protect the cells. Human experts rated this hypothesis novel and worth pursuing. UCSF neuroscientist Tomasz Nowakowski describes it as clues "scattered across different fields" that hadn't been connected.
What to make of it
The architecture is simple. The interesting part is the controlled comparison, which gives evidence that fan-out improves the quality of reasoning, not only speed. The size of the citation gap, 700 against 11, suggests that the single-agent setup degrades badly at this scale rather than being a strong alternative.
The limits are significant:
Nothing has been tested in a lab. The leads are hypotheses for prioritising experiments, and OpenAI frames them that way.
More citations doesn't mean better reasoning. The post doesn't report whether the citations were accurate or relevant, which matters given how often language models invent or misattribute references.
The blinded evaluation is lightly described. The 75% preference and the agreement with expert ratings come without sample sizes or protocol.
The cost is unstated. Thirty-two minutes of wall-clock time across thousands of agents could be a lot of compute. The post doesn't compare that cost with researcher time.
The time-saving figure is an estimate. "Months to a year" is the team's own guess, not a measured baseline.
The more lasting contribution may be the public reasoning atlas. Other researchers can query it, check the agents' explanations against their own expertise, and run experiments to test them.
Remember
With the model and resources held fixed, one agent per subtask beat one agent doing everything: 700+ citations versus 11.
Single agents given a growing workload cut back on research first, the familiar pattern of an agent spreading itself thin.
The outputs are untested hypotheses, and the post gives no figures on citation accuracy, evaluation size or compute cost.
Cached prefixes live in a shared 100 PiB SSD store (DeepSeek's 4), so any free worker can pick up any request. Agent traffic stays cached for three days, chat for seven. OpenAI just moved to 12 hours.
Decode balances both the number of running requests and how much cached state each one holds.
5: when short-lived 6 state has expired, the engine approximately rebuilds it from only the last 128 tokens. DeepSeek says this nearly halves prefill compute.
An adaptive speculative decoder adds 6% throughput over a fixed five-token setup.
Query preparation. The orchestrator turns the conversation into one or more short text summaries of the commercial intent. Multiple queries keep separate needs separate; a trip, for example, may involve transport, lodging and activities. OpenAI's example: a user asks what to cover on a two-day Yosemite trip, then whether it will be cold next week, then about a brand's new product lines. The user never links these, but the orchestrator might produce two queries: "shoes appropriate for hiking trip" and "new [brand] product lines good for travel." It is designed to rewrite these queries if they don't retrieve good ads. It also has to tell current intent from stale context, recognize hypotheticals, and avoid treating every product mention as a wish to buy.
Retrieval. The queries become 3 (vectors that capture meaning), which are searched against several approximate-nearest-neighbor (ANN) indices. Each index is tuned for a different kind of ad. For example, the index for product-feed ads takes positive and negative reviews into account. The goal at this stage is a deliberately broad pool of candidates.
Filtering. This step removes ads that fail practical checks:
whether the campaign or creative is active
locale targeting
advertising policy
remaining budget
4 (whether the campaign should spend now or later in the day)
placement compatibility (which kinds of conversations the ad may appear in)
per-advertiser deduplication, so the pool is fair across businesses
Scoring. The orchestrator calls scoring systems for semantic relevance, predicted engagement (views, clicks, conversions) and historical performance, plus relevance and safety filters.
Auction. The orchestrator nominates plausible candidates but does not choose the winner or set the price. A separate auction tool handles admission thresholds, pacing, bid computation, quality-adjusted ranking, pricing and final selection. This split keeps the model's judgment away from the money.
Final checks. An auction winner can still be dropped in two ways. The orchestrator may block it as unsuitable for this particular conversation, even when both the conversation and the ad passed earlier safety checks; 5, which decide when ads are allowed at all, cleared the conversation. Or the ad may simply arrive too late. ChatGPT never waits for an ad, so if the answer is ready first, the ad times out.
DCTN1 and cholesterol. DCTN1 helps move cargo inside neurons, and its mutations cause 8, a Parkinson-like disease. Disrupting it switched on genes that make cholesterol. The agents linked this to research on faulty cholesterol transport, suggesting misplaced cholesterol might drive neurodegeneration. Experts rated the idea novel.
