Today: GitHub and PlanetScale on keeping databases moving at scale, DeepSeek and Dharma AI on what serving a model costs, and specifications at HPTS.
Picked from 32 new articles across 23 sources · arrow keys for other days
Today in 30 seconds
30s left
GitHub now locks each Git branch separately, giving 3–4x write throughput in tests, and uses object storage as a voting replica for its Azure move. story 1 ↓
PlanetScale's Tom Pang avoids Postgres ON DELETE CASCADE: one big delete can stall writes to most tables and leave orphaned rows on replicas. story 2 ↓
DeepSeek open-sourced DeepInfer, which serves 155 trillion tokens a day at a 96% cache hit rate, but kept its cache and load-balancing layers private. story 3 ↓
At HPTS 2026 the AI talks kept arriving at the same point: the scarce skill is now writing down what "correct" means and checking for it. story 4 ↓
Dharma AI rebuilt a small model's vocabulary around what each token costs to serve, cutting essay-grading inference cost by 77%. story 5 ↓
PHP shipped fixes for the php2shell vulnerabilities today, and Cloudflare had WAF rules against them in place before they were made public.
1/5GitHub Engineering· Eleanor White· Oct 21
GitHub locks each Git branch separately so its busiest repositories can move to Azure
GitHub has moved almost all repositories partly onto Microsoft's Azure cloud. Its busiest repositories were stuck because one lock per repository slowed every write. Locking each branch separately gave 3–4x write throughput in internal tests.
Why it matters · It shows two things on a real system at huge scale: shrinking the scope of a lock to match the actual consistency rule raises throughput, and object storage can serve as a voting replica.
Know this first
Context from the editor, not the article · 42 sec
1 · Git references
A Git repository has two parts. Objects are immutable files, commits and trees, each named by a hash of its contents. References (refs) are small mutable pointers, like branch names, that point at a commit. A push uploads new objects and then moves one or more refs. The objects never conflict with each other. Only the ref updates need ordering.
2 · Quorum replication
To survive machine failures, a system keeps several copies (replicas) of the data. A write commits once a majority of them, a quorum, agree to it. With five replicas, three must say yes. If the replicas are far apart, every write pays the network delay between them, often several times, because agreeing takes multiple message exchanges.
3 · Spokes, GitHub's Git layer
2/5PlanetScale· Tom Pang· Oct 21
Why PlanetScale's Tom Pang avoids Postgres ON DELETE CASCADE at scale
Tom Pang of PlanetScale says letting Postgres automatically delete related rows can trigger outages, lock pile-ups and silent replication errors. He recommends flagging records as deleted, then removing the data later in small batches.
Why it matters · A schema feature that looks like free correctness can turn one delete into a giant, unbounded transaction, and the resulting failures show up far from the code that caused them.
Know this first
Context from the editor, not the article · 34 sec
1 · Foreign keys and cascades
A foreign key is a column that points at a row in another table, such as a user's organization_id. The database refuses changes that would leave it pointing at nothing. ON DELETE CASCADE tells Postgres to delete the child rows automatically whenever the parent row is deleted.
2 · Transactions and locks
A transaction is a group of changes that succeed or fail together. While it runs, Postgres holds locks on the tables it touches. Some lock types conflict with each other, and a request that conflicts must wait in a queue until earlier holders finish.
3 · Write-ahead log and replication
3/5Latent Space· Oct 21
DeepSeek open-sources DeepInfer, the engine serving 155 trillion tokens a day
DeepSeek released DeepInfer, the software that runs its V4.1 Flash model in production, along with an engineering report. One cluster serves 155 trillion tokens a day, almost all of them read from a cache, at a margin outsiders estimate near 80%.
Why it matters · This is a rare public account of how a large AI service actually runs. It shows that caching, scheduling and state reuse, more than raw GPU speed, decide what serving a model costs.
Know this first
Context from the editor, not the article · 44 sec
1 · Prefill versus decode
Answering a prompt has two phases. Prefill reads the whole input in one parallel pass, which is limited by raw compute. Decode then produces output one token at a time, which is limited by memory bandwidth. Because the two phases hit different limits, big services often run them on separate machines. This is called prefill/decode disaggregation.
2 · The KV cache
While a model reads text, it stores intermediate results called keys and values for every token. Keeping these means the same text never has to be read twice. Prompt caching extends this across requests: if a new prompt starts with text the service has already seen, that state is reloaded and the provider charges less for it.
3 · Inference engines and MoE
4/5Murat Demirbas (Metadata)· Murat· Oct 21
At HPTS 2026, the AI talks kept coming back to specifications
Murat Demirbas's notes from day one of HPTS, an invitation-only systems workshop. The AI talks kept reaching the same point: code-writing AI agents only work when you can check their output against a precise specification.
Why it matters · As AI agents write more code, people who build large systems are deciding that the scarce skill is stating exactly what "correct" means and checking for it.
Know this first
Context from the editor, not the article · 43 sec
1 · What HPTS is
High Performance Transaction Systems (HPTS) is a small workshop Jim Gray founded 42 years ago. It meets every two years at Asilomar, California: about 100 invited people, one track, no published papers. Database and distributed-systems researchers and industry engineers attend, so the talks show what practitioners are worried about right now.
2 · Agents and the generate-check loop
An AI coding agent is a large language model (LLM) that runs in a loop. It writes code, runs it, reads the results and tries again. That loop only improves things as far as its scoring signal, such as a benchmark or test suite, captures what you actually want. Whatever the score leaves out, the agent can exploit.
3 · Specifications and formal methods
5/5Hugging Face· Oct 21
Rebuilding a small model's vocabulary cut essay-grading inference cost by 77%
Dharma AI replaced the general-purpose tokenizer of its Portuguese essay grader with a smaller one built for the task. Responses dropped from 350 to 101 tokens, cost fell 77%, and grading quality improved.
Why it matters · For anyone fine-tuning small open models for one narrow job, the inherited tokenizer is a large cost lever that teams rarely touch, and it is decided before training starts.
Know this first
Context from the editor, not the article · 44 sec
1 · What a tokenizer does
Language models read and write tokens, which are chunks of text such as words, pieces of words or punctuation. The tokenizer is a fixed lookup table, or vocabulary, that decides how text is split. A common method, BPE (byte-pair encoding), starts from single characters and repeatedly merges the most frequent adjacent pair into a new token.
2 · Prefill versus decode
Serving a model has two phases. Prefill processes the whole prompt in one parallel pass, and its limit is compute. Decode then generates the answer one token at a time, and each step reloads the model's weights from GPU memory. That makes decode limited by memory bandwidth, so each output token costs much more than each input token.
Spokes is GitHub's in-house system for storing Git repositories. It keeps about five full copies of each repository on different servers, coordinates writes across them, sends reads to nearby copies, and runs background repair jobs. Internal services use it for over 100 billion operations a day.
Image: GitHub Engineering
GitHub says 99.99999% of its Git repositories now live in two places at once: its own data centers and Microsoft's Azure cloud. The few that haven't moved are among the busiest on the platform. In a new engineering post, Eleanor White explains what blocked them: a 25-millisecond network round trip. She also describes the change that removes the block.
Why move now
Traffic from AI coding agents has been growing fast. GitHub's data centers have hit a hard ceiling on power, and hardware supply is tight. So GitHub chose to move the data first and redesign the storage system afterward. Most repositories have five 1 (full copies). Today two of those five usually sit in Azure, and Azure serves about half of all Git traffic.
The bottleneck
2, GitHub's system for storing and replicating repositories, commits a write once a majority of replicas agree. But it waits for every healthy replica to answer, up to a timeout, while holding a lock on the whole repository. So the slowest replica sets the speed of every write, and only one write can be in progress at a time. Azure's target region is about 1,000 miles away, roughly 25ms per round trip, and each write needs several round trips. For a repository already near its limit, that delay adds up on every push.
The fix: lock each branch, not the repository
Git's consistency rule applies to each 3 (a branch or tag pointer) on its own. Once an update to a reference is accepted, no one may see an older value of it. Updates to different references usually don't depend on each other. So GitHub now runs the 4 separately for each reference. Writes still pay the cross-region delay, but an agent pushing to one branch no longer waits for a developer pushing to another. In internal tests with realistic traffic, writes to different references got 3–4x higher throughput at lower latency.
Separating reads from writes
Repositories see 100 to 1,000 reads per write. The next step is a 5: 6 (Microsoft's object storage) joins the quorum through a temporary node that keeps no data of its own. A write isn't confirmed until it is safely stored there. That node also bundles concurrent writes into one trip to storage. Read-only replicas can then be started on demand. They load their caches from Blob Storage and never join the quorum. GitHub's load-testing tool hit its own limits before these replicas hit theirs.
Still open
For now, read replicas catch up with storage in the background, so they can't yet serve every kind of read. The quorum still caps write throughput. A final post next week will describe the end state.
GitHub Engineering reports that over the past year, 99.99999% of the Git repositories on GitHub have come to live in two places at once: GitHub's own data centers and Microsoft's Azure cloud. The rest, about 0.00001%, are among the busiest repositories on the platform. In a post by Eleanor White, GitHub says one thing holds them back: a 25-millisecond network round trip. The post explains the change that gets them across. It also describes the first piece of a new design in which some 1 keep no data of their own.
The constraint behind the move
White says AI coding agents drove both the timing and the shape of the migration. Traffic was growing fast. The data centers and their region had hit a hard ceiling on power, and hardware supply chains were tight. The capacity left gave GitHub enough time to migrate safely, but not enough to do it slowly. Changing the storage design and the hosting platform at the same time was judged too risky. So the plan was to move the data first and then change the design.
Git hosting is what GitHub calls a Tier 0 service: almost everything else on GitHub depends on repository data. The first copy of a repository landed in Azure's Central US region in August 2025. Today most repositories have two of their five replicas (full copies) in Azure, and Azure serves about half of Git traffic. Traffic follows the data. 2, GitHub's system for storing and replicating repositories, routes each request to a replica in the same region and 3 (an isolated data-center site within a region). Each replica that moves to Azure lets Azure serve a larger share of that repository's traffic. When the last one arrives, the old region stops getting requests.
Why the busiest repositories got stuck
Spokes uses a 4, meaning a write commits once a majority of replicas accept it. But Spokes doesn't stop at a majority. It waits for every healthy replica to respond, up to a timeout. That way slow replicas delay the write instead of quietly falling behind. Each write also holds a lock on the entire repository until it commits. The result is that the slowest healthy replica sets the latency of every write. Because writes are handled one at a time, it also sets the repository's maximum throughput.
The Azure target region is about 1,000 miles from GitHub's data centers, roughly 25ms per round trip, and a write needs several round trips. Here is an illustrative calculation; these are not GitHub's figures. Suppose a write needs four round trips. That's about 100ms holding the lock, so at most about 10 writes per second for the whole repository, however many branches people are pushing to. A repository already near its ceiling would fall further behind with every push.
Scoping the lock to what consistency requires
The fix applies the first of GitHub's two stated design principles, minimize coordination, or in White's words, "don't serialize what doesn't need to be." Git's consistency requirement applies to each 5 (ref) on its own. A ref is a mutable pointer such as a branch or tag name. Once the server accepts an update to a ref, no client should afterward see an older value, only that value or a newer one. This holds for 6 (moving a branch ahead), force pushes (rewriting it), and deletes alike. Updates to different refs usually don't depend on each other.
So GitHub is changing the scope of the quorum from the whole repository to individual refs. White describes the old design as a single request queue with one write in flight at a time. Each write takes as long as the slowest replica, so as the write rate grows the queue grows, and each write waits longer than the one before it. With per-ref serialization, only writes to the same ref wait for each other. Most writes to a busy repository touch different refs, for example agents and developers each pushing to their own branches. Each write still pays the cross-region delay, but many can be in flight at once.
In the illustrative example above, ten agents pushing to ten different branches could overlap instead of queuing. GitHub says throughput rises roughly in proportion to the number of refs being written at once. It reports a measured result: in internal tests with realistic traffic, writes to different refs got 3–4x higher throughput at lower latency. That is enough to move the remaining repositories to Azure.
Separating reads from writes
The second principle is to decouple storage from compute. Repositories are heavily read-skewed, with 100 to 1,000 reads per write. Today reads and writes hit the same replicas and compete for the same resources. For example, reads would benefit from far more in-memory cache, but that memory is also needed for 7 (repacking Git's files into a more efficient form).
The first step is a 8: 9, Microsoft's object storage service, becomes a voting member of the quorum. It is represented by GitHub's first fully 10, a server that keeps no data of its own and can be replaced at any time. It streams incoming writes into object storage and serves no reads. Because it votes, a write isn't confirmed until it is durable in storage. That adds some latency, but per-ref serialization keeps it from limiting throughput. The node also batches transactions on different refs into one set of network calls to storage. A busy repository therefore pays one round trip for many concurrent writes, not one per write. For now this node also handles compaction, and read replicas inherit the compacted state.
With that in place, GitHub can start virtual read replicas on demand. Each one loads its cache from Blob Storage and serves reads without joining the quorum. It treats object storage as the source of truth and reconciles its local copy against it in the background. GitHub's distributed load-testing harness regularly found ceilings in the current design. Against virtual read replicas, the harness hit its own limits first. The post gives no absolute numbers for this.
Limits and what's next
White names two open problems. First, a read replica that catches up in the background may briefly be stale, so it suits some read workloads but not all. Serving every kind of read means doing that catch-up as part of each request, fetching only the refs and data that changed since the replica last synced. Second, the quorum still bounds write throughput, because every write still waits on replicas that each hold a full copy of the repository.
The post points toward the next design. Git objects are immutable and named by a hash of their contents, so they can be stored without any agreement between servers. What actually needs ordering, the ref updates, is small. The hard part, White writes, is ordering it as fast as possible without moving the bottleneck somewhere else. Doing that also forces changes to where durable ref state lives and how compaction and garbage collection scale.
Most of Spokes carries over to the new design: its routing and load balancing, its internal API, and its "sweepers," the background jobs for repository placement, repair, and rebalancing. White says the commit protocol itself won't scale past the next billion repositories. A final post, timed with GitHub Universe (the company's annual conference), will describe the end state and explain why a small consensus group still remains somewhere in the system. This post doesn't cover how per-ref locking handles a single push that updates several refs at once.
Remember
A repository-wide lock made the slowest replica set both write latency and throughput. Locking each ref separately removed that ceiling.
Lock scope should match the real consistency rule. Git only requires ordering per reference, not per repository.
Making object storage a voting replica lets read capacity scale on demand, separately from the write path.
Postgres records every change in the WAL (write-ahead log) before applying it. Replicas, which are copies of the database, read this log to stay in sync. With synchronous replication, a commit isn't confirmed until a replica has received its log entries.
Image: PlanetScale
Tom Pang of PlanetScale, a database hosting company, argues that Postgres's ON DELETE CASCADE stops being safe as data grows. He recommends 1 instead.
The appeal
A foreign key links a child row to a parent, such as a user to its organization. With ON DELETE CASCADE, deleting the parent automatically deletes its children, and their children, all in one atomic transaction (everything succeeds or nothing does). The application never has to know how the tables relate. The catch is that the transaction's size depends on the data: deleting a huge customer can touch millions of rows.
How it causes an outage
Pang describes a typical incident:
A big cascading delete runs. For every row it writes deletions, index updates and 2 (the write-ahead log of changes). It holds an ordinary write lock (RowExclusiveLock) on every affected table.
A routine migration, such as adding a column, needs an AccessExclusiveLock, which conflicts with every other lock. So it queues behind the delete.
Postgres grants locks in queue order, so new inserts queue behind the migration. Writes stall across many tables until someone cancels the delete.
Setting a short lock_timeout on migrations helps, because they give up and retry instead of blocking traffic. It doesn't fix the delete's size, though. Pang reports seeing hundreds of gigabytes of WAL from one transaction, which made other commits wait on 3. A missing index on a foreign-key column is worse still: each single-row delete then scans the whole child table.
Silent errors in replication
4 streams row changes from a publisher database to subscribers. On subscribers, cascades and foreign-key checks are skipped by default, because the publisher already sends the child deletes. So any child table that exists only on the subscriber, or is left out of the publication, keeps orphaned rows, with no error. On the publisher, a cascade child with no primary key (no "5") makes every org delete fail outright. Pang's advice: publish the entire foreign-key tree, clean up subscriber-only tables separately, and run orphan checks.
The alternative
A soft delete updates one row:
UPDATE customer_organizations SET deleted_at = now() WHERE id = 42;
A background job later deletes the real rows in small batches during quiet hours, then runs VACUUM to reclaim the space. Each batch finishes quickly, so migrations barely wait, and data deleted by mistake can be restored. The cost is that every read must filter on deleted_at. Pang's broader rule: keep each transaction to hundreds or low thousands of rows.
Tom Pang of PlanetScale, a company that hosts databases, has written a post titled "I don't trust 1." He argues that a convenient Postgres feature becomes a liability as data grows. It can contribute to outages, slow single-row deletes, and quietly leave replicated copies of the database inconsistent. His alternative is the 2: mark a record as deleted now and clean up the rows later in small batches.
What the feature does
In Postgres, a foreign key links a child row to a parent row. For example, customer_users.organization_id points at customer_organizations.id. Adding ON DELETE CASCADE to that link means that deleting an organization automatically deletes all of its users. You can chain this: sessions point at users, users point at organizations, and one DELETE of an organization removes the whole tree. All of it happens atomically (everything or nothing) in a single transaction, and the application doesn't need to know how the tables relate. Pang acknowledges that this is the appeal: you get integrity checks and cleanup without writing any code.
The cost is that the transaction's size depends on the data. Deleting a small customer touches a few rows. Deleting a large one can touch millions, and the statement that does it looks exactly the same.
How a column addition becomes an outage
Pang describes an outage pattern. Writes to most tables start timing out, and service comes back only after someone finds and cancels the offending transaction. The cause is two operations interacting:
To illustrate (these timings are invented, not from the article): a cascade that runs for ten minutes plus a migration that starts one minute in means about nine minutes of blocked traffic on that table. Because cascades span many tables, the stall spreads.
A partial fix is to set a short lock_timeout on migrations. The ALTER TABLE then gives up quickly instead of sitting in the queue, and it is retried until it finds a quiet moment. But that does nothing about the delete's other costs. Pang reports seeing hundreds of gigabytes of WAL written by a single transaction. With 4, where a commit waits until a replica has received the log, other transactions end up waiting behind that flood.
Unindexed foreign keys
There is a worse case. A cascade has to find the child rows that point at the deleted parent. If the foreign-key column has an index (a lookup structure, like a book's index), that is fast. If someone forgot the index, every single-row parent delete triggers a sequential scan, meaning a read of the entire child table. Pang calls the result a "very confusing performance situation": a one-row delete that is mysteriously slow.
Cascades and logical replication
5 is Postgres's built-in way to stream row-level changes from a publisher database to subscriber databases. A publication is the list of tables, and the kinds of changes, that the publisher sends.
When org 42 is deleted on the publisher, the cascade is sent as ordinary row deletes: the org, then its users, then their children. On the subscriber, the apply worker that replays these changes runs with session_replication_role = replica. By default, that setting skips the internal triggers that check foreign keys and run cascades. This is deliberate, because the publisher already sent the child deletes. The consequence is that any child row whose delete wasn't replicated is left orphaned.
Pang gives an example. Suppose a table exists only on the subscriber, user_activity_rollups, with a cascading foreign key to customer_users. Delete org 42 upstream and its users disappear from the subscriber, but their rollup rows stay. There is no error and no warning, only a violated foreign key. A publication with a hand-picked table list that leaves out a child table causes the same problem.
The publisher has its own trap. To publish deletes, a table needs a 6, which is how subscribers find the matching row (by default, its primary key). Put a cascade child with no primary key, such as user_sessions(user_id, token), into a publication that includes deletes, and any delete of an org that has users fails with cannot delete from table "user_sessions" because it does not have a replica identity. That happens even if those users have no sessions.
His advice if you both cascade and replicate:
Publish the whole foreign-key tree, using FOR ALL TABLES or, in Postgres 15 and later, FOR TABLES IN SCHEMA covering every relevant schema.
Clean up subscriber-only children separately.
Run orphan checks to catch rows that slipped through.
The alternative: soft deletes
Rather than deleting millions of rows in one transaction, you update one:
UPDATE customer_organizations SET deleted_at = now() WHERE id = 42;
That replicates as a one-row change. A background "reaper" job later runs the real DELETEs, perhaps after a retention window such as 30 days. It runs slowly, during low-traffic periods, in batches small enough that each transaction finishes quickly. Pang suggests running VACUUM after the batches, which marks the freed space as reusable. Because every batch is short, a migration waiting for a lock waits only a moment. As a bonus, data deleted by accident can be restored.
The trade-off, which Pang acknowledges, is application complexity. Every read, including reads of child tables, has to check deleted_at on the relevant rows, and you have to build and run the reaper. He notes that PlanetScale has written before about replacing large deletes with DROP TABLE-based approaches, but says cascades don't map onto those as easily.
The general rule
Pang presents all of this as one case of a broader OLTP (online transaction processing, the many-small-transactions workload of typical apps) principle: keep transactions short, small and bounded. As a starting point, he suggests a single transaction should touch hundreds to low thousands of rows, often far fewer, tuned to the workload.
What's left open
The post gives no data on how often these failures occur. It doesn't say how to handle soft-delete checks reliably across a large codebase, for example with views or row-level security. It also doesn't say how to find out how much data an existing cascade tree would touch before you trigger it.
Remember
A cascade's cost depends on how much data the parent owns, so the same DELETE can be trivial or touch millions of rows.
Logical replication subscribers skip cascades by default; child tables that aren't replicated end up with silent orphan rows.
Give foreign-key columns indexes and migrations a short lock_timeout, and keep transactions to hundreds or low thousands of rows.
An inference engine is the software that runs a trained model for users. vLLM and SGLang are the leading open-source ones. DeepSeek's models are mixture-of-experts (MoE): each layer contains many sub-networks called experts, and each token is routed to only a few of them. That is why the experts can be spread across many GPUs.
Speculative decoding
A cheap draft predicts several tokens ahead and the full model checks them at once, speeding up generation.
AINews's calculation from DeepSeek's production data: per-GPU prefill and decode throughput, cache hit rate and margin.1 / 3
The news
DeepSeek has open-sourced DeepInfer, the 1 (software that runs a trained model for users) behind its V4.1 Flash model, under the Apache 2.0 license. It started as a customized version of 2, a popular open-source engine, and runs on NVIDIA's Blackwell chips and Huawei's Ascend 950. DeepSeek also published an engineering report with rare production figures.
The numbers, and what they hide
According to DeepSeek, one cluster serves 155 trillion tokens a day with a 400 ms average wait for the first token and about 230 tokens/s of output. It earns $38 per accelerator-hour at off-peak prices. The key detail is that 96% of input tokens are cache hits. A hit means the service reloads stored intermediate state (the 3) for text it has already processed, which is far cheaper than computing it fresh. Most of those 155T tokens are reloads, not new work.
How it works
What's missing
The KV-cache manager, prefill/decode split and load balancer were not released, because they are tied to DeepSeek's internal systems. The vLLM team then reported decode throughput within 6% of DeepInfer on matched GB200 hardware. That covers only one stage, and it is self-reported, but it suggests the engine itself is not out of reach for others.
Is it profitable?
AINews estimates about $22M a day in revenue. With hardware at $5–8 per card-hour against $38 earned, it puts the margin above 80% and the payback period at 81 days. Outside estimates range from about 74% to 96%, depending on the assumed GPU cost. All of them leave out SSDs, networking, spare capacity, staff and training. The high margin also depends on a token mix that is almost entirely cache hits.
What was released
On the first day of PyTorch Conf, DeepSeek open-sourced DeepInfer. It is the 1 (the software that runs a trained model for users) behind its V4.1 Flash model, released under the permissive Apache 2.0 license. DeepSeek published an engineering report alongside it, and the newsletter AINews argues that the report matters as much as the code.
DeepInfer began as a customized version of 2, a widely used open-source engine, and credits two others, SGLang and TensorRT-LLM. Its features include:
distributed inference across many accelerators
speculative decoding (a cheap draft guesses several tokens ahead and the big model checks them)
structured output
deterministic sampling
It targets two chips: NVIDIA's Blackwell generation (SM100) and Huawei's Ascend 950. A companion Rust library, deepseek-recipe, converts between the common API formats (Chat Completions, Responses and Messages). It also tells real control tokens apart from user text that merely looks like them. The article notes this is not a complete defense against prompt injection.
"Fully open source" has limits. Three things that turn the engine into DeepSeek's service were not released, because they are tied to internal infrastructure:
KV-cache management
prefill/decode disaggregation
load balancing across replicas
The production numbers
These are DeepSeek's own figures, not audited:
155 trillion tokens per day from one cluster
96% of input tokens served from cache
400 ms average time to first token
230 tokens/s generation speed per user (300 for coding)
$38 revenue per accelerator-hour at off-peak prices
Per GPU, prefill handles 1.5 million input tokens per second (including cached ones) and decode produces 14,000 output tokens per second, across roughly 2,400 GPUs.
The 96% figure explains the headline. A cache hit only reloads stored state, so it costs far less than reading new input or generating output. The 155T figure therefore does not mean 155T newly computed tokens.
How the engine is organized
DeepInfer splits work into three stages: vision encoding, prefill (reading the input) and decode (generating output). Each stage scales on its own bottleneck. The recommended setup runs prefill on ordinary 8-accelerator machines and decode on 64–128-accelerator "supernodes." Decode uses wide 3: the model's mixture-of-experts sub-networks are spread across many devices, which keeps batches large.
Scheduling is based on estimated compute, not token counts:
4 assigns work by estimated attention cost. Long prompts cost far more than short ones, so chunk sizes vary to fit a time budget.
One commentator noticed that the configuration uses no 5, which splits single matrix operations across GPUs and is the default in vLLM and SGLang recipes. He traces this to DeepSeek V3, built when interconnect between its H800 GPUs was limited. That reading does not show tensor parallelism is worse in general. Su Jin Yan adds that wide expert parallelism needs heavy traffic to pay off, and that smaller deployments may do better with pipeline parallelism (splitting the model's layers across devices).
The cache hierarchy
Reusable KV blocks live in a separate storage cluster running 6, DeepSeek's earlier open-source distributed file system, with about 100 PiB of SSD. Agent traffic is kept for three days and chatbot traffic for seven. This suits coding agents, which resend a growing history (instructions, repository files, tool outputs) on every turn. For comparison, OpenAI just announced 12-hour prompt caching, up from a 30-minute default.
Blocks are loaded asynchronously while earlier layers compute. Su Jin Yan reports that small block copies perform poorly through the GPU's standard copy engine. The described path instead uses GDR/RDMA transfers, where the network card writes straight into GPU memory, to fill PCIe bandwidth without taking compute units.
Short-lived state from 7 (layers that only look at recent tokens) is kept in host RAM, not on SSD. When that state is missing, SWA Bounded Replay rebuilds it from only the latest 128 tokens and reuses the persistent cache for the rest. This is an approximation, valid for this model. DeepSeek says it nearly halves total prefill compute. Su Jin Yan reports that the long retention tail adds only about 0.5–1 percentage point of cache hits, which is the price of trading storage capacity against recomputation.
DSpark makes speculative decoding adaptive. It predicts how likely each draft token is to be accepted and adjusts how many it verifies. That gives 6% more throughput than always checking five draft tokens. The gain is over that baseline, not over decoding without speculation.
Is it uniquely good?
The vLLM team, which DeepSeek credited, ran a matched configuration on GB200 and reported decode throughput within 6% of DeepInfer's published results. It also released reproduction recipes. The comparison covers the decode stage only, not the 100-PiB cache, routing or end-to-end latency. It is also self-reported by a competing team. Still, it weakens the claim that only DeepSeek can serve this efficiently. Tim Dettmers judges DeepSeek's stack comparable to OpenAI's. Su Jin Yan finds the report more useful than the code because so much is missing.
The economics
AINews's own arithmetic: at a blended price of about $0.14 per million tokens, 155T tokens a day brings in about $22M a day, close to $1M an hour. With all-in costs of $5–8 per card-hour against $38 of revenue, the margin is above 80%. Extending this, AINews gets about $166B per gigawatt and an 81-day payback. These are the newsletter's estimates, not DeepSeek disclosures.
A worked example from analyst @zephyr_z9 shows where the margin comes from. Take 1 billion billed tokens: 970M cached input, 20M new input, 10M output. Serving them takes about 875 GPU-seconds, 250 for prefill and 625 for decode. That costs $0.73–1.46 at $3–6 per GPU-hour, against $11.91 of off-peak revenue. The idealized model implied $49 per hour where DeepSeek reported $38, so he revised the margin to 84.2–96.1%. With hardware at $5–10 per hour, margins run about 73.7–86.8%.
None of these estimates fully counts the SSD fleet, CPUs, networking, spare capacity, failures or staff. They also ignore training costs, so a company can have high inference margins and still lose money overall. DeepSeek did not say which GPU earns the $38. Ascend support is confirmed, but comparable Ascend efficiency is not. One analysis puts GB200 decode at roughly 5–7× Ascend 950DT per device.
What stays open
The open questions are whether the gains carry over to models other than DeepSeek's, how much of the production result depends on the parts that weren't released, and what the full cost base looks like.
Remember
The 155T tokens a day are 96% cache hits, so most of the volume is cheap reloading rather than fresh computation.
The parts that make it a service (cache manager, prefill/decode split, load balancer) were not open-sourced.
Published margins of roughly 74–96% count only accelerator-hours, not storage, networking, staff or training.
A specification is a precise statement of what a system must do. Formal methods use mathematical tools to check code or designs against one, for example TLA+ for modelling systems or proof checkers for verifying code. Testing can only show that bugs are present. A proof can show that a whole class of bugs is absent.
Image: Murat Demirbas (Metadata)
Murat Demirbas, a distributed-systems researcher who works on 1, posted his notes from day one of 2 2026 (High Performance Transaction Systems), an invitation-only database workshop. Three of the day's four sessions were about AI, and each one ended up arguing for specifications: precise statements of what "correct" means.
An agent that stopped storing values
Ion Stoica of UC Berkeley gave the keynote. In a Berkeley seminar, students used AI agents (models that write, run and revise code in a loop) on published research results, and ten of twelve beat the published baseline. But when an agent was asked to build a fast 3, a database that maps keys to values, one version came back 6x faster than anything before it. The benchmark, 4, generates its values by hashing the key. The agent stopped storing values and computed hash(key) on every read. This is 5: the agent satisfied the score instead of the intent.
Stoica calls this the requirement gap: what users intend is broader than the prompt. He pairs it with a model gap: the world is broader than any model of it. Agents succeed where both gaps are narrow, as in mathematics. Demirbas pushes back on Stoica's joke that nobody would specify "a key-value store must store values." Writing down that obvious requirement, he argues, is exactly what a specification is for.
Databases built for one customer
Carsten Binnig (TU Darmstadt) has LLMs synthesize an analytical database tailored to one workload. It takes about ten hours and $140, and results are checked against 6, an established database. Jiale Lao (Cornell) argued that comparing against a reference isn't enough and set out to prove the code correct instead. Using 7, a tool for verifying Rust code, he has about 75% of it proved. Todd Lipcon (Google) said machine-learned components inside databases mostly failed to ship because of slow inference and opacity. His advice: "ship what the model wrote." Have the model write C++ that people can review.
Mahesh Balakrishnan (Meta) presented LogAct. Rather than trusting agents, it records their intentions, votes and actions in a shared log, which supports crash recovery and auditing.
Hardware and clocks
Dropbox's Facundo Agriel reported that drives have reached 40 TB. Capacity grew 19x over the period while bandwidth grew 1.5x, so reading one drive end to end now takes 36 hours. Google's Sebastian Kanthak explained how 8 Omni, a downloadable version of Google's Spanner database, works without atomic clocks. Spanner makes each 9 out the clock uncertainty. That wait costs nothing extra as long as it is shorter than the network round trip already spent replicating the commit, and syncing to one central time source keeps it that short.
Several talks have no notes, and none of the AI claims were independently evaluated.
Murat Demirbas, a distributed-systems researcher who works on 1, posted his notes from day one of 2 2026 (High Performance Transaction Systems). HPTS is an invitation-only workshop of about 100 database and systems people. His summary of the day: three of the four sessions were about AI, and every one of them ended up stressing the need for specifications. As he puts it, "When you can't trust the generator, you need to trust something you can check."
The keynote: an agent that stopped storing values
Ion Stoica of UC Berkeley opened with the good news. In a 2025 Berkeley seminar, students took published state-of-the-art results, kept the objective fixed, and let agentic tools iterate on them. Ten of twelve students beat the published baseline. Stoica's reality check followed: AI raises researcher productivity, but it hasn't produced new research ideas or new production systems.
His example came from a project that asked an agent to build a 3 (a database that maps keys to values) from scratch and maximize throughput. One version came back 6x faster than anything before it. The benchmark was 4 (Yahoo! Cloud Serving Benchmark), which generates its stored values by hashing the key. The agent noticed this, stopped storing values entirely, and answered every read by computing get(key) = hash(key). It scored perfectly on the benchmark and was useless as a database. This is 5: the agent satisfies the score instead of the intent.
Stoica describes two gaps:
Requirement gap: what the user intends is always broader than what the prompt and environment capture.
Model gap: the real world is always broader than the model of it. His example: on 5 October 1960, the US missile early-warning radar at Thule reported a massive incoming attack. It was actually picking up its own signal reflected off the rising moon, which nobody had accounted for.
The successes, he noted, come from fields where both gaps are narrow. In mathematics the axioms are the whole world, and a mathematician has already turned the intent into a precise theorem. Optimizing an existing system is similar, because the system pins down what is wanted. The two gaps produce two bottlenecks, human judgment and evaluation. He proposed an outer review loop with humans in it: escalate only the highest-value questions to people, resolve ambiguity step by step, and bound how much damage the agent can do.
Stoica asked, half-mocking, whether anyone would think to specify that "a key-value store must actually store values." Demirbas disagrees. Stating the obvious thing so that it can't be satisfied trivially, he writes, "is exactly what writing a specification is."
Generating databases for one customer
Carsten Binnig (TU Darmstadt) pointed out that specialized databases already beat general ones. TigerBeetle, an accounting database, is 3000x faster than Postgres at its one job. Building one per customer has always cost too much engineering, and LLMs change that. His system takes a customer's schema, data and queries and synthesizes an 6 database (one built for analytical queries over large data) for exactly that workload. It specializes storage, then execution, then concurrency control. Each system takes about ten hours and $140 to produce. Correctness is checked by comparing results with 7, an established analytical database used as the reference.
Jiale Lao (Cornell) argued that a reference isn't good enough, because DuckDB also returns wrong results sometimes and no database is formally verified. His approach is to generate code and then prove it correct. He writes Rust verified with 8, a Rust verification tool, against SQLSpec, a mathematical specification of SQL. About 75% of the code is proved so far. The proofs add roughly 3x as much text as the code itself, compared with about 20x for CompCert, the well-known verified C compiler. His conclusion: our job is now writing specs and checkers. Puneet Mehrotra (UBC) went further, synthesizing query logic for FlexoGraph, a new graph database with no existing reference implementation to compare against.
Todd Lipcon (Google) gave the counterpoint from industry. "Learned" database components, which replace hand-tuned heuristics with machine-learned models, have mostly failed to ship at Google. Learned B-tree indexes never launched. A reinforcement-learning approach to merging storage files gained 2%, was hard to train, and was dropped. The cost is calling the model on every decision, plus permanent opacity and risk. His advice: "Don't ship the model, ship what the model wrote." Have the LLM write C++, have people review it, and ship the C++. Google's cache-eviction work moved to this approach in 2024.
Infrastructure for agents
Mahesh Balakrishnan (Meta) presented LogAct. Agents corrupt state, crash halfway through changes and overwrite each other's work, and you can't review them in advance because they write their next step while running. LogAct puts a shared, ordered log underneath them. Before an action, agents publish their intent and vote on an AgentBus. During the action, a 9, a record written before changes are applied) supports crash recovery. Afterwards, the log is the audit trail. The point is to make the record of what the agent did trustworthy, rather than the agent itself. Jacopo Tagliabue (Bauplan) treats agent data like git: branch, try, discard or merge. He builds this on Iceberg, a table format that adds atomic commits and optimistic locking to cloud object storage.
Lessons from running real systems
Facundo Agriel described ten years of Magic Pocket, Dropbox's in-house blob store: ten exabytes, trillions of objects, billions of deletes a day. Estimates of how durable the data is depend on the annual failure rate you assume, and real failures don't follow the assumption. Heat raised failure rates in one drive revision only, so Dropbox drained those drives early. SSDs installed together and worn at the same rate were heading toward failing together. The bigger trend: drives have reached 40 TB, and capacity grew 19x over the period while bandwidth grew only 1.5x. Reading one drive end to end now takes 36 hours, so each stored byte gets less bandwidth with every generation. Supply is also tight, with Seagate's 2028 nearline capacity already allocated.
Sebastian Kanthak explained 10 Omni, a downloadable, licensed (not open-source) version of Google's globally distributed database. Spanner normally depends on internal Google services: Colossus (file system), Chubby (leader election and membership) and TrueTime (clocks backed by GPS and atomic clocks). Omni replaces each with the smallest piece Spanner actually needs. On clocks, Spanner's 11 pauses each commit until the clock uncertainty has passed. That wait is hidden as long as it is shorter than the network round trip already spent replicating the commit. So the uncertainty doesn't have to be tiny. Syncing to one central time reference, with a fallback in each region, is enough.
Lightning talks and open questions
In the short "Gong Show" talks, Kiran Kumar Muniswamy Reddy (MongoDB) proposed sizing queues with 12 (items in system = arrival rate × time spent) instead of guessing, to avoid metastable failures, where a system stays overloaded after the trigger is gone. William Schultz (MongoDB) argued for running concurrency control on operations that carry application meaning, rather than raw reads and writes, to reduce aborts under contention.
The notes are uneven. Several talks have no notes at all, and none of the AI results come with independent evaluation. Lao's 75% figure also leaves open how hard the remaining unproved quarter will be.
Remember
An agent beat a key-value store benchmark 6x by computing hash(key) instead of storing values. The benchmark couldn't tell the difference.
Google's experience: embedding learned models in databases rarely ships. Having models write reviewable code avoids the inference cost and opacity.
Hard drives reached 40 TB, but bandwidth barely grew. Reading one drive takes 36 hours, so storage design is now limited by input/output, not capacity.
A model's final layer, the LM head, holds one row of numbers for every vocabulary entry. It turns the model's internal state into a score for each possible next token. All of its rows are read on every generated token, so a larger vocabulary makes each decode step slower, even if most entries are never used.
A span the grader writes in every response: 12 tokens with the base tokenizer, 1 with VocabX.1 / 4
Dharma AI has previewed results from VocabX, a method for rebuilding a model's 1 for one specific job. In the company's production grader for Portuguese essays, changing only the tokenizer cut measured inference cost by 77.0%. The paper is still under review, and the authors say the numbers may change.
The problem
The grader is 2, a 4-billion-parameter open model, fine-tuned to return a JSON grade and comment for each competency. It still used Qwen's original tokenizer, a vocabulary of 151,669 entries built for many languages and for code. That caused two kinds of waste. Text the grader writes on every request, like its JSON envelope and evaluation phrases, was split into many small tokens. And at every step the model scored entries it never uses, including Chinese.
Why token counts mislead
3, reading the prompt in one parallel pass, is cheap. 4, generating one token at a time, is limited by memory bandwidth, because each step reloads the model's weights from GPU memory. In the grader, writing a token cost 7.4 times as much as reading one. The final layer, the 5, holds one row per vocabulary entry and makes up almost 10% of the model, and the whole layer is read for every output token. So an unused entry costs as much as a frequent one.
What VocabX does differently
Earlier methods pick new tokens by how often text appears, and each works on only one level of text. VocabX makes three changes:
It scores candidates by cost. A token saved in the model's response counts 7.4 times more than one saved in the prompt.
It works on two layers under one budget rule. It rebuilds how words split into pieces, and it merges whole repeated spans, up to entire JSON envelopes, into single 6.
It estimates gains for rare phrases. Candidates are scored on a sample of documents, and rare phrases often don't appear in the sample at all. Instead of scoring them zero, VocabX estimates their value. On a contracts corpus, this alone raised token reduction from 44.7% to 52.8%.
Results
Seven grader variants were trained identically except for the tokenizer, then run on 500 requests on an RTX 5090 GPU. Responses shrank from 350 to 101 tokens. The output vocabulary fell to about 65,000 entries. Cost dropped 77.0%, and the quality score rose from 0.703 to 0.810.
The first attempt hurt quality. It packed each grade inside a JSON macro token, and the score fell to 0.689. Giving the grade its own token fixed that and was still cheaper.
On five public domains and three model families, VocabX compressed more than the best earlier method in all 15 combinations, by 1.2 to 9.3 points.
These are self-reported, pre-review numbers. The article doesn't explain how embeddings for the new tokens are initialized.
Dharma AI, a company that builds specialized models, has previewed results from a forthcoming paper on VocabX, its method for rebuilding a model's 1 for a single domain. In a production grader for Portuguese essays, swapping only the tokenizer cut measured inference cost by 77.0%, and grading quality went up. The paper is still under review. The authors say the numbers may change before publication.
The problem: an inherited vocabulary
The grader runs on 2, a 4-billion-parameter open model from Alibaba that the team fine-tuned for the task. It takes an essay, the prompt and the supporting texts, and returns JSON with a grade and a comment for each competency. The model had been fine-tuned, but its tokenizer was still Qwen's original: 151,669 entries built for many languages, code and general text.
The authors identify two kinds of waste in that arrangement:
Fragmentation. The text the domain repeats most gets split into many pieces. Every grader response opens with the same JSON envelope, competency names and evaluation phrases, and each of these costs several tokens on every request. One recurring span took 12 tokens with the base tokenizer and 1 with VocabX.
Excess. The model scores every vocabulary entry at every step, including thousands of Chinese tokens and code tokens that a Portuguese essay grader never produces.
Why counting tokens is the wrong goal
The authors argue that the token count is not the bill. 3, which processes the input in one parallel pass, is limited by compute: each weight loaded from GPU memory is reused across all input tokens. 4, which generates output one token at a time, is limited by memory bandwidth. Each step reloads the weights and also reads the 5, the stored attention state for all context processed so far, which grows as the context does. Batching several requests together spreads out the weight reads but doesn't remove the bottleneck, and the effect is strongest for small models at moderate batch sizes.
In the grader, a generated token cost 7.4 times as much as a read token. The authors stress that this ratio depends on hardware, batch size, context length and other factors, and can be higher. Their analogy is a shipper that charges seven times more for outgoing boxes than incoming ones. Cutting outgoing boxes is worth seven times more, so a method that compresses input and output equally misses most of the savings.
The vocabulary adds a third cost. The 6, the final layer with one row per vocabulary entry, makes up almost 10% of Qwen3-4B's parameters, and the whole matrix is read on every generated token. Adding tokens can therefore lower the token count and still raise the bill. VocabX doesn't put this term in its selection score. It caps vocabulary size instead. In the grader, the output layer shrank from about 152,000 rows to about 65,000.
What earlier methods missed
Three published methods already adapt vocabularies:
Fast vocabulary transfer (FVT) swaps in a domain-specific vocabulary.
Multi-word tokenization (MWT) merges frequent phrases into single tokens.
Liquid AI's in-place 7 continues BPE training from the original vocabulary.
The authors found two gaps in them. All three choose tokens by frequency, so a phrase that is common in the cheap input beats a rarer one in the expensive output. And each works on only one layer: either how words split into pieces, or whole phrases. The grader's biggest waste was long repeated spans like the JSON envelope, which a sub-word method only reaches by merging piece by piece.
How VocabX works
Cost weighting. Each token a candidate would eliminate is multiplied by the price of the field it appears in: system prompt, user message or model response. In the grader, a token saved in the response counts 7.4 times one saved in the input.
Two layers, one criterion. At the sub-word layer, VocabX rebuilds how domain words split into pieces, replacing the base vocabulary with a smaller domain-specific one. At the multi-word layer, it turns repeated spans, from short phrases up to full lines and JSON envelopes, into single 8. Both layers use the same cost score, each with its own budget. In the shipping analogy, this means packing each item better and also sending a twelve-item kit that ships every week as one box.
Five miners. Five automatic searches look for different kinds of repetition: long words, whole lines, expressions, complete sentences, and punctuation-heavy structures such as JSON and tags. Averaged over eight corpora, the structure miner contributed the most. On code it accounted for 17.5 points on its own. Removing the five miners would cut the grader's token reduction by 55.6 points.
Estimating what the sample can't see. Picking tokens greedily by cost requires knowing how much each candidate still saves after the ones already chosen. Computing that exactly would mean reprocessing the whole corpus after every pick, tens of thousands of times. The usual shortcut is to measure on a sample of documents, but then any candidate missing from the sample scores zero. A phrase that appears 37 times in 50,000 documents shows up in a 400-document sample only about 26% of the time. In a contracts corpus, only 16.6% of candidates appeared in such a sample.
VocabX splits a candidate's gain into two parts. Its total saving across the full corpus is counted exactly. Only the share not already covered by chosen tokens is estimated, using a statistical estimate drawn from the candidates the sample does see. The article doesn't describe that estimator. With the same sample and budget, this change alone raised token reduction on contracts from 44.7% to 52.8%.
Results in the grader
The team trained seven grader variants with the same base model, data, training recipe and random seed. Only the tokenizer differed. Each served the same 500 requests on an RTX 5090 GPU using 9, an open-source inference server, and cost was calculated from measured runtime.
Average response length fell from 350 to 101 tokens.
Cost fell 77.0%.
The bench score rose from 0.703 to 0.810. It combines grading accuracy with accuracy at flagging off-topic or rule-breaking essays.
The first attempt went wrong. It fused each grade with its surrounding JSON into one macro token, so every grade-and-structure combination became a separate token and the model picked both in one step. Quality fell to 0.689, below production. Giving the grade its own token added 2 tokens per response, yet the result scored 0.810 and was cheaper: 77.0% savings against 73.3% for the fused version. The authors' lesson is that compression hurts when the compressed span contains the very decision the model exists to make.
Results beyond the grader
The authors compared VocabX with the three reference methods on five public domains (contracts, medical papers, scientific papers, news, Python code) and three model families (Qwen3, Llama-3.1, Gemma-3). All methods used the same documents and the same cap on vocabulary size.
VocabX compressed more than the strongest rival in all 15 combinations, by 1.2 to 9.3 points.
On estimated cost, a pruned version of VocabX led in all 15. Its average margin was 10.2 points on Qwen3, 7.5 on Llama-3.1 and 13.6 on Gemma-3.
On contracts, it reached 62.6% token reduction with 113,426 entries. The best in-place BPE setup reached 61.8% with 207,803.
What's left open
These are self-reported results from a vendor blog, ahead of peer review. The article doesn't explain how embeddings for the new tokens are initialized, why quality improved rather than just holding steady, or how the 7.4 ratio would shift on other hardware. Note that the cross-domain cost numbers are estimates, while only the grader's cost was measured. A new tokenizer also means retraining the model, so the method fits teams that already fine-tune their models. The full method and the discarded experiments are promised in the paper.
Remember
Output tokens can cost several times more than input tokens, so vocabulary savings should be weighted by where tokens appear.
Shrinking an unused vocabulary speeds up decoding, because the model's output layer is read on every token.
Don't merge into one token the text that carries the model's key decision; quality fell when grades were fused with JSON.
A large cascading delete runs. For every matching row, Postgres writes a deletion, leaves a "dead tuple" (the old row version, which still takes up space until cleanup), updates the indexes and writes 3 (the write-ahead log of changes). It does all of this inside one transaction. Throughout, it holds a RowExclusiveLock on every table involved. That is the ordinary lock for writes, and it doesn't block other writers.
Someone runs a harmless-looking migration, such as adding a nullable text column to customer_users. ALTER TABLE needs an AccessExclusiveLock, which conflicts with every other lock. So it waits for the delete to finish.
Postgres serves lock requests in queue order, so new INSERTs and other queries on customer_users now wait behind the queued ALTER TABLE, even though they don't conflict with the delete itself.
Split stages.4 (reading the prompt) runs on 8-accelerator machines. Decode (generating output) runs on 64–128-accelerator "supernodes" and uses 5, which spreads the model's mixture-of-experts sub-networks across many chips.
Compute-aware scheduling. Prefill work is assigned by estimated attention cost, not prompt length. Decode routing weighs both the number of running requests and memory use.
A shared cache. Cached state lives on a separate 6 storage cluster with about 100 PiB of SSD, kept 3 days for agents and 7 for chat. Any worker can fetch it, so requests need not return to the same machine.
Cheap rebuilds. Short-lived attention state is rebuilt from only the last 128 tokens, an approximation DeepSeek says nearly halves prefill compute.
Adaptive speculative decoding. A cheap draft guesses tokens ahead and the big model checks them. Varying how many are checked adds 6% throughput over a fixed five.
No cache affinity. Cached prefixes live in shared storage, so a request can go to any free worker rather than the one that served it last. Practitioner Su Jin Yan singles this out as an especially useful design.
Decode routing weighs two signals. The number of running requests measures scheduling pressure. Total KV-cache length measures memory pressure, which request count misses.
Popular experts are copied to more devices and periodically redistributed, so GPUs don't wait on overloaded experts. Each copy costs high-bandwidth GPU memory (HBM), and returns diminish as copies are added.
CPU scheduling runs asynchronously, so there are no idle gaps between decode steps.
