Today's stories are about storage layers under strain: where data sits, how it moves, and what breaks when you change it.
Picked from 49 new articles across 23 sources · arrow keys for other days
Today in 30 seconds
30s left
Cloudflare removed the range-size setting and now fetches large files in its cache's fixed disk chunks. Slowest-1% first-byte latency on R2 roughly halved. story 1 ↓
OpenAI cut median global read latency on Cosmos DB from 120 ms to 10 ms with caching, change-feed invalidation and a Rust rewrite. story 2 ↓
PlanetScale shows how Neki's shard groups keep related Postgres rows on one server. Reference tables and global indexes cover the queries that cross shards. story 3 ↓
Netflix's Quasar traces removed or changed library methods to the exact call sites in other repos, and says how much to trust each match. story 4 ↓
At HPTS 2026, researchers argued data movement is now the main cost. One talk cut Cassandra's storage overhead from 6.2x to 1.6x. story 5 ↓
1/5Cloudflare· Matthew Gumport· Oct 23
Cloudflare drops the range-size setting for caching large-file partial requests
Cloudflare rebuilt how its cache handles requests for part of a file. It now fetches whole storage-sized chunks instead of a range size each customer had to set. For R2, slow-request latency roughly halved.
Why it matters · It shows how a cache's internal limits ended up as a customer setting, and how putting together correct partial responses from several sources gets complicated fast.
Know this first
Context from the editor, not the article · 42 sec
1 · CDNs and origins
A content delivery network (CDN) such as Cloudflare keeps copies of files in data centers near users. The customer's own server, the origin, holds the real copy. When a file isn't cached (a miss), the CDN fetches it from the origin. Often it goes through several cache layers (tiers) first, and each hop costs bandwidth.
2 · HTTP range requests
A client can ask for only part of a file by sending a Range: bytes=start-end header. The server replies 206 Partial Content, with a Content-Range header saying which bytes came back and how big the whole file is. Video seeking, resumable downloads and reading a file's index all rely on this.
3 · Why ranges are hard to cache
2/5OpenAI· Oct 23
How OpenAI scaled its Cosmos DB storage layer and cut read latency 12x
OpenAI's storage platform, Habitat, holds over 500 petabytes on Microsoft's Azure Cosmos DB. It went back to shared database containers after Azure added new throttling and autoscaling, and caching cut median read latency from 120 ms to 10 ms.
Why it matters · It is a rare, detailed account of what goes wrong when a managed database is pushed past its tested limits: noisy neighbors, partition splits that quietly shrink throughput, and failover stampedes.
Know this first
Context from the editor, not the article · 35 sec
1 · Multi-tenant vs single-tenant
A single-tenant system gives each team its own database, which keeps failures contained but wastes capacity. A multi-tenant system puts many teams on one shared database. That pools capacity, but one heavy user (a "noisy neighbor") can slow everyone else down.
2 · Partitioning a database
Large databases split data into partitions, or shards. A partition key, such as user_id, decides which partition a record lives on. Many logical keys share one physical machine-backed partition. When one key gets very busy (a "hot key"), it can overload its physical partition.
3 · Caches and staleness
3/5PlanetScale· Rohit Nayak· Oct 23
PlanetScale explains how to group tables so sharded Postgres queries stay local
PlanetScale's Rohit Nayak shows how Neki, the company's sharding layer for Postgres, uses "shard groups" to keep related rows on one server. Joins and transactions then stay on a single shard and keep full Postgres guarantees.
Why it matters · On a sharded database, choosing the shard key decides which transactions stay correct, not only which queries run fast. This post shows that trade-off in concrete config.
Know this first
Context from the editor, not the article · 41 sec
1 · What sharding is
Sharding splits one logical database across several independent servers, called shards, so each holds part of the rows. A router sits in front and sends each query to the right shard. A query that touches one shard is cheap. A query that has to go to every shard and merge the results, called a fan-out, is slow.
2 · Shard keys and hashing
Each row is placed by its shard key, a column such as customer_id. The key is hashed into a big number space, and each shard owns one contiguous slice of that space. Hashing spreads keys evenly. Equal keys always produce the same hash, so they always land on the same shard.
3 · Why single-shard transactions matter
4/5Netflix Tech Blog· Netflix Technology Blog· Oct 23
Netflix's Quasar finds which repositories a library change will break
Netflix built Quasar, an internal index of code symbols across its repositories. When a library pull request removes or changes a public method, Quasar lists which repositories call it, which versions they use, and where the calls are.
Why it matters · Coding agents can now edit a library quickly, but nothing tells them who depends on it. Quasar shows one concrete way to give them that cross-repository context.
Know this first
Context from the editor, not the article · 41 sec
1 · Producers, consumers, breaking changes
A shared library is written by one team (the producer) and used by many applications (consumers). If the producer removes or renames a public method, any consumer still calling it fails to build or crashes after upgrading. That is a breaking change. Its "blast radius" is the set of consumers it actually hits.
2 · Why finding callers is hard
Tools that check compatibility can tell that a removal breaks the contract, but not who is affected. Dependency lists show which projects declare the library, not which ones call a given method. Plain text search mixes up methods that share a name, misses inherited calls, and can't tell which library version a caller is built against.
3 · Semantic code indexes
5/5Murat Demirbas (Metadata)· Murat· Oct 23
At HPTS 2026, systems researchers argue that moving data is now the main cost
Murat Demirbas summarized two days of talks at HPTS, an invitation-only database workshop. Speakers kept returning to one point: moving data, not computing on it, now limits speed, power use, cloud storage design and cost.
Why it matters · Several talks overturn defaults engineers rarely question, such as caching in front of S3, triple replication, and naive retries. They also come with numbers you can check against your own systems.
Know this first
Context from the editor, not the article · 41 sec
1 · What HPTS is
High Performance Transaction Systems (HPTS) is a small, invitation-only workshop held every two years at Asilomar, California. It mixes database researchers with engineers from companies like Microsoft, Databricks, Apple and AWS. Talks are short and often opinionated, so they show where practitioners think the field is heading rather than presenting finished results.
2 · Object storage and lakehouses
Amazon S3 stores files ('objects') cheaply and durably, but each request is slow. Analytics systems now keep data on S3 in open file formats like Parquet. A table format such as Iceberg tracks which files make up a table. Together these form a 'lakehouse': warehouse-style queries over shared files that many different engines can read.
A cache stores whole files under a key. Partial requests arrive at arbitrary offsets, so the cache has to choose a unit to fetch and store. The choice is either many small pieces or wasteful big ones. It also has to make sure pieces fetched at different times come from the same version of the file.
Diagram of how the edge assembles one range response from cached, in-flight and newly fetched bytes.
Cloudflare has released Origin Range Requests on all plans. It is a rebuilt way for its cache to handle HTTP 1, which ask for part of a file instead of the whole thing. Customers turn it on with a 2, a per-site traffic-matching setting, and there is no longer a range size to configure.
Why the old way hurt
A range request carries a header like Range: bytes=1048576-2097151. The server answers 206 Partial Content, with a Content-Range header naming the bytes returned. Video seeking, resumable downloads and package managers all depend on this.
Cloudflare's old cache was built on 3, a common web server. It split each file into fixed-size slices, and each customer had to choose the slice size. Configured sizes ranged from tens of kilobytes to tens of gigabytes. Small slices meant floods of tiny requests to the 4 (the customer's own server), each holding memory. Large slices meant downloading far more than the client wanted and moving the surplus across several cache tiers between data centers.
The slice size was part of the 5 (the identifier for stored entries), so changing it threw away cached data. Range caching also couldn't run alongside streaming a file to readers while it was still being cached. Both problems caused real incidents at Cloudflare.
The fix
Cloudflare's newer caching server is built on 6, its in-house proxy framework, and Pingora already stores cached files on disk in fixed-size chunks. So the fetch unit became the chunk. Cloudflare serves the requested bytes it already has, fetches missing chunks from the origin, stores them normally, and returns only what the client asked for. A request smaller than a chunk still costs a full chunk, but that waste is now bounded and predictable. The post doesn't state the chunk size.
The harder part is assembly. One response may combine bytes from disk, from a download already in progress for another client (which a new request can join), and from new origin fetches. Before stitching, Cloudflare checks that the total file size and the 7 (a version fingerprint from the origin) stay the same across every piece. If they don't, or if the file is going to be decompressed, it serves the whole file rather than risk mixing versions.
Results
8, Cloudflare's object storage, used to run with 64 MiB slices, so one small read crossing a slice boundary could move 128 MiB between data centers. After switching, R2's 9 (the wait for the slowest 1% of requests) fell by about half on both cache hits and misses. A later change let a single response mix cached and freshly fetched bytes. That cut tiered-cache traffic for R2 and similar workloads by about 6% week over week, which Cloudflare says is petabytes a day.
The feature doesn't make uncacheable content cacheable, and the origin must support byte ranges for it to help.
Cloudflare has made Origin Range Requests available on all plans. It is a rebuilt way for Cloudflare's cache to handle HTTP 1, which ask for a slice of a file rather than the whole thing. Customers turn it on with a 2, a per-site rule that matches traffic and sets caching behaviour. There is no longer a range size to tune. Cloudflare already runs 3, its object storage service, on the new design. There, 4 (the delay before the first byte arrives, for the slowest 1% of requests) dropped by roughly half. A later change cut traffic between Cloudflare's cache tiers by about 6% for R2 and similar workloads.
The problem: one setting, no good value
When a client sends Range: bytes=1048576-2097151, the server returns 206 Partial Content with a Content-Range header giving the bytes returned and the total file size. Video players use this to jump to a seek point. Download managers use it to resume, and package managers and other tools use it to read only part of a large file.
Cloudflare's old implementation was built on 5, a widely used web server and proxy. It worked like the "cache slicing" approach nginx has described. Each request was mapped onto fixed-size slices of the file, and each customer zone (a domain on Cloudflare) configured the slice size and which file extensions it applied to. Configured sizes ranged from tens of kilobytes to tens of gigabytes.
Both ends of that range caused trouble:
Small slices turned one request into a long line of tiny 6 requests. Each one held memory while Cloudflare merged ("coalesced") the pieces in memory.
Large slices made Cloudflare download far more than the client asked for. The surplus then crossed several hops: origin to a data center, between cache tiers, and on toward the client. Every hop multiplied the waste. Gumport describes a request too small to fill a floppy disk triggering "a few DVDs' worth" of origin transfer, sometimes after the client had already left.
The slice size was part of the 7, the identifier a cache stores entries under, so changing the setting invalidated existing cached data. Concurrent requests that landed in the same slice could block each other even when they wanted bytes far apart. Range caching also couldn't be combined with concurrent streaming, a feature that lets several clients read a file while it is still being written to cache. Turning both on cut responses short. So each slice had to finish downloading before anyone could read it, and popular files needed a separate origin fetch for every slice. Both the small-slice and large-slice patterns caused real operational incidents. The post's point is that customers were being made to understand the cache's internals just to get files served well.
The fix: the chunk is the range
The opening came from Cloudflare's move to 8, its in-house proxy framework written in Rust, as the base for a new caching server. Pingora already stores every cached file on disk in fixed-size chunks so interrupted downloads can resume. The team's insight was that the fetch unit for range requests could simply be that chunk, so it no longer needed to be a setting.
When a range request arrives, Cloudflare serves whatever requested bytes are already cached. For the missing bytes, it fetches origin ranges aligned to chunk boundaries, stores them as ordinary cache chunks, and returns only the bytes the client asked for. In the post's example, a request for bytes 150–250 may fetch the whole chunk containing them, but the client still receives only 150–250.
This doesn't remove over-fetching. A request smaller than a chunk still costs a full chunk, and ranges that don't line up with chunk boundaries take extra work. The post doesn't give the chunk size. Cloudflare's argument is that the waste is now bounded and predictable, rather than set by a customer value with unpredictable effects on its network and on origins.
The hard part: putting one response together
Parsing the header is easy. The difficulty is producing one correct response when the bytes are spread across disk, an origin response, and downloads already in progress. The edge server:
Looks up the file's headers and metadata recording which byte ranges are cached.
Reads what it can from disk or from an in-memory stream.
Fetches missing coverage from the origin in aligned ranges.
Writes the new bytes and records the completed range in metadata.
Stitches the pieces together and sets Content-Range and Content-Length for the client.
As a worked illustration (the numbers are made up), a client asks for bytes 0–30 MB of a video. The first 10 MB are on disk, bytes 10–20 MB are being downloaded for another viewer right now, and the rest is missing. The coordinator streams the cached prefix, attaches to the in-flight download rather than starting a duplicate, and fetches the remaining chunks from the origin. It can fetch more from the origin than it sends on, but the client's status and headers describe only 0–30 MB. Attaching to existing downloads shares the work when traffic for a popular file spikes.
Correctness checks
Stitching is only valid if every byte comes from the same version of the file. The coordinator checks three things on each origin response: it matches the requested range, the total file size hasn't changed, and the 9 (ETag, a version fingerprint the origin attaches) is the same across all subrequests. If a client sends If-Range, meaning "give me this range only if the file still matches this version", and the version has changed, Cloudflare returns the full file rather than mixing bytes from two versions.
Cloudflare also won't stitch ranges for a response it is going to decompress. Byte offsets in compressed data don't match offsets in the decompressed output, so in that case it serves the full transformed response. The result follows standard HTTP: 206 with an accurate Content-Range, 416 Range Not Satisfiable for impossible ranges, and the full file whenever a range can't be served safely.
Results
Cloudflare moved existing range-caching users first, starting with the most painful ones: lots of concurrent range requests against huge files, with spiky traffic. R2 was the largest internal user. Its 64 MiB slice size meant a small read crossing a slice boundary could cost 128 MiB of transfer between data centers. After the move, p99 time to first byte fell by about half across R2's storage regions, for both cache hits and misses.
The latest change lets one response combine bytes from disk with bytes fetched upstream. Before, a request was served entirely from cache or entirely from upstream. For R2 and similar workloads, that cut tiered-cache traffic about 6% week over week, which the post describes as petabytes a day.
Caveats and open questions
The feature doesn't make uncacheable responses cacheable, and it only helps if the origin supports byte ranges. Target workloads include seekable MP4 video, resumable downloads, package repositories, disk images, and AI model files whose selected regions get read. It is enabled in the dashboard or via the API with origin_range_requests: { "mode": "on" }. Cloudflare plans to make it the default for certain file types. The post leaves several things unstated: the chunk size, how the two results break down between the rebuild itself and the later stitching change, and any results for customers other than R2.
Remember
Reusing the cache's existing on-disk chunk size as the fetch unit removed a setting customers could easily get wrong.
Stitching partial responses is only safe when size and entity tag match across every piece; otherwise Cloudflare serves the whole file.
Mixing cached and freshly fetched bytes in one response cut R2's traffic between cache tiers by about 6%.
A cache keeps copies of frequently read data close to the code that needs it. The hard part is knowing when a copy is out of date. You can let copies expire after a fixed time (a TTL, time-to-live) or delete them explicitly when the data changes (invalidation).
noisy neighbor
A tenant whose heavy load uses up shared resources and degrades service for other tenants on the same hardware.
Image: OpenAI
OpenAI has described how it scaled Habitat, its internal storage platform. Habitat serves over 1 billion users across 40 regions and holds more than 500 petabytes on 1, Microsoft's managed, globally distributed database.
How the database divides capacity
Cosmos DB groups data into containers. Each container gets a throughput budget in request units per second (2), an abstract measure of database work. Records are sharded by a partition key such as user_id. Many keys share one 3, and that partition is where limits are actually enforced.
Shared, then split, then shared again
OpenAI started with one shared container and hit three problems. First, different products used the same keys and shared one per-key limit. Second, a busy niche feature could slow login and signup if it landed on the same physical partition. Third, as data grew, partitions split while total throughput stayed fixed, so each partition got less capacity even though traffic hadn't changed.
Giving each workload its own container contained the damage, but every team then had to plan capacity. In a separate container, a launch could bring 1,000x the load seen in testing, for example 1K RU/s in load tests and 1M RU/s at launch. Teams forgot to pre-scale and spent launch days in incident rooms.
So OpenAI went back to one shared container, with features built jointly with Azure:
Hot-key throttling, which throttles the busiest keys first.
Priority tiers, so offline bulk jobs are throttled before requests users are waiting on.
A new serverless mode, where each partition can reach 10K RU/s (up from 5K) and splits happen based on load, not just storage.
OpenAI reports no provisioning or noisy-neighbor incidents in the months since.
Outages at a scale nobody had tested
OpenAI's largest account serves over 20M queries per second. In late 2024 a regional power outage showed that failing over to another region took hours. When the account moved back a month later, a client bug sent every client to the original region at once and overwhelmed its gateway. The two companies now share dashboards, a production-sized test account and direct paging.
Faster reads
Reads make up over 90% of traffic. OpenAI copies small, critical datasets such as authentication metadata to many regions. For everything else it caches data near compute in 4, an in-memory key-value store. Stale copies are cleared within seconds using Cosmos DB's 5, a stream of every committed change. The service was also rewritten from Python to Rust. Median read latency fell from about 120 ms to about 10 ms. The slowest reads, which miss the cache, are still slow.
Next, OpenAI is building its own exabyte-scale store on RocksDB Cloud, a storage engine, and 6, a protocol for replicating data across servers.
OpenAI has published the second part of its account of Habitat, the internal storage platform behind its products. Habitat answers requests in milliseconds for over 1 billion users across 40 geographic regions and holds more than 500 petabytes. Part I covered turning Habitat from a Python client library into a service. Part II covers the database underneath it, 1 (Microsoft's managed, globally distributed document database), and how OpenAI made it reliable, then fast.
How Cosmos DB divides capacity
In Cosmos DB, an account holds a set of containers (roughly, tables) plus a policy for replicating them across regions. Each container is given a throughput budget measured in request units per second (2), an abstract cost unit covering CPU, memory and I/O. Data is sharded by a client-chosen logical partition key. Cosmos packs many logical keys onto fewer 3 and splits a physical partition when it grows too large in storage or needs more throughput. The documented limit is 10,000 RU/s per logical key.
Phase I: one shared container, three failure modes
At first OpenAI used a single default container per account and let any load through, relying on the per-key limit to push back. Three problems followed.
Key collisions. Clients chose their own partition keys, so unrelated products all used keys like user_id or workspace_id and shared one limit per user.
Noisy neighbors. Limits are actually enforced per physical partition, not per logical key, so no single key ever really gets 10K RU/s. A niche feature landing on the same physical partition as login and signup could disrupt those core flows.
Weak partitions. A container's throughput is divided roughly evenly across its physical partitions. Here is an illustrative example (the numbers are not from the article). A container with 100K RU/s on 10 partitions gives each partition 10K. If data growth splits those into 20 partitions, each gets 5K, even though traffic hasn't changed. For storage-heavy, blob-like data, growth alone throttled active workloads.
Phase II: a container per workload
Separate containers fixed most of this, because each gets its own physical partitions. A script kept each container's throughput in step with its partition count, which handled the weak-partition problem. OpenAI started giving every new service or big feature its own container, partly to shield older critical data still stuck in the shared default container. That turned site-wide outages into outages of single features.
The cost was organizational. In a shared pool, one feature jumping from 0 to 1 million queries per second barely moves the total. In its own container, the same launch might bring 1,000x the load seen during internal testing, for example 1K RU/s in load tests and 1M RU/s at launch. Teams forgot to pre-scale, and engineers spent launch days in SEV rooms (incident war rooms) scaling containers by hand. Capacity planning had spread to essentially every product team, and those teams did it with uneven care.
Phase III: back to sharing, with new controls
OpenAI went back to multi-tenancy by default, working with the Azure team to add:
Hot-key throttling, which throttles the busiest keys first so one key can't use up a physical partition.
Quality of Service (QoS) tiers, which give offline work such as bulk inserts and data extracts lower priority, so that work is throttled before the interactive requests users are waiting on.
A new serverless mode. The old serverless mode scaled only on storage and capped each physical partition at 5K RU/s. The new one lets each partition reach 10K RU/s and splits partitions based on RU usage as well as storage.
OpenAI now runs one shared container and reports no provisioning or noisy-neighbor incidents in the months since.
When "managed" stops being managed
OpenAI's largest Cosmos DB account stores tens of petabytes and serves more than 20M queries per second. OpenAI believes no other company runs Cosmos DB at that scale. In late 2024 a power outage hit the primary region for one large account, and failing over to a secondary region took several hours. Neither team had tested that at this size. About a month later, moving back to the original region caused a second outage. A quirk in the gateway-mode client sent every client to the original region at once, even though only some partitions had moved back, and the resulting connection stampede overwhelmed the gateway.
OpenAI argues that at this scale a managed service needs co-ownership. The two teams now share dashboards from Habitat's service layer down to Cosmos DB's physical storage, test changes on a production-sized non-production account, develop SDKs together, and can page each other directly without going through support tiers. Joint fixes added load protection and metadata caching in the gateway, more connection pooling and reuse from Habitat's Python processes, and separate virtual networks (VPCs) for critical and offline workloads.
Making reads fast
Reads are over 90% of requests, and compute often sits in a different region from storage. OpenAI layered three approaches:
The Habitat service was also rewritten from Python to Rust. Together these changes cut median (7) global read latency from about 120 ms to about 10 ms, a 12x improvement this year. Tail latency is still a problem, because uncached reads still go to distant regions.
CDC as a platform
The change feed powers more than the cache. Millions of changes per second, from thousands of containers and millions of partitions, flow through over a hundred Kafka topics (Kafka is a distributed log used for event streaming). They feed secondary indexes, data-movement pipelines and analytics. A control plane assigns work to thousands of workers. Capture reads from replica regions at low priority to protect user-facing traffic. Dead-letter queues hold permanently failing messages so they don't block the ones behind them. OpenAI also modified Azure's SDK to batch checkpoints in memory and write them out periodically.
What's next
OpenAI is building an in-house, exabyte-scale store for bulky Habitat data. It combines RocksDB Cloud (Rockset's cloud version of the RocksDB storage engine) with 8, a consensus protocol for replicating a log across servers. It keeps Cosmos DB's document model but separates storage from compute and replicates at the physical level, trading lower peak QPS for less flash storage. The article gives no timeline or results for it.
Remember
When storage growth splits partitions, each partition gets a smaller share of a fixed throughput budget, even with steady traffic.
Separate containers trade shared capacity headroom for isolation, and push capacity planning onto every product team.
Cache invalidation driven by the database's change stream gave both high hit rates and low staleness, which TTLs alone can't.
A Postgres transaction on one server is atomic: all of its changes commit or none do, and readers see one consistent snapshot. Getting the same guarantee across servers needs a coordination protocol such as two-phase commit. Many sharding systems skip that protocol, so a cross-shard transaction can end up half-applied.
reference tables
Small shared lookup tables that are copied in full to every shard so joins against them stay local.
Image: PlanetScale
PlanetScale's Rohit Nayak has written a guide to designing 1 in 2, the company's system for sharding Postgres. Sharding splits a database's rows across several independent Postgres clusters, called shards. His argument is that you design shards from your queries, not your tables.
The problem
In an earlier storefront example, customers and orders were each sharded by their own ID. A customer and that customer's orders could land on different shards. The router, which forwards SQL to shards, then had to join them itself.
The cost goes beyond speed. A transaction on one shard keeps full Postgres guarantees. One that touches two shards does not: Neki commits each shard separately, "with no 3 and no shared 4." One shard can commit while the other fails, and readers can see a mix of old and new state.
How a shard group works
A shard group is declared in a JSON config and has two parts:
A shard index: which column to hash and how. For example, xxhash, a fast hash function, applied to customer_id, producing a 64-bit number.
Key ranges: which slice of that 64-bit space each shard owns. In the example, hashes below hex 80 (the halfway point) go to store-1 and the rest to store-2.
Every table in the group hashes its customer_id the same way, so all of a customer's orders, items and addresses land on one shard. customers uses the same index but on its id column. The everyday queries and transactions never leave that shard.
Covering the gaps
Caveats
Placement can change later. The 7 workflow changes a table's shard key or shard count. The post leaves some things out. It doesn't say what happens if the index write and the order write, which often sit on different shards, don't both succeed. It also doesn't say how quickly reference-table copies agree with each other.
PlanetScale has published a design guide for 1, its system for sharding Postgres across many clusters. In the post, Rohit Nayak covers one decision that shapes everything else: which tables share shard placement. Neki's name for that unit is the 2, and the post's main argument is in its title. You design shard groups from your queries, not from your tables.
The problem
The post continues an earlier storefront example. There, customers was sharded by its own id and orders by its own ID. A join between a customer and that customer's orders could therefore span two shards. The Neki router, the component that receives SQL and forwards it to shards, had to fetch rows from both and do the join itself.
Speed is only part of the cost. Nayak is explicit that a transaction confined to one shard "keeps full Postgres semantics," and one that reaches two shards does not: "Neki commits each shard separately, with no 3 and no shared 4." No atomic commit means one shard can commit while the other fails, leaving a half-applied change. No shared snapshot means a reader can see one shard's new state next to the other's old state. Where you place data therefore decides which of your invariants Postgres still enforces for you.
How a shard group works
All of this lives in a JSON config Neki calls the data topology. A shard group has two parts:
A default shard index. This names the column to hash and the hash function, for example xxhash on customer_id. 5 is a fast non-cryptographic hash, and here it maps each key to a point in a 64-bit space.
Here is an illustrative case. Say customer 42 hashes to a value beginning 0x3f…, so the customer lives on store-1. Every table in the group (orders, items, addresses) hashes its own customer_id column with the same function. Every row with customer_id = 42 therefore also hashes to 0x3f… and lands on store-1. The one exception is customers, whose key column is id. Neki lets a table inherit the group's index but point it at a different column. The outcome is that all of one customer's data sits on one shard, and the common joins and transactions never leave it.
The rule that makes this work: tables in the same group that hash their keys the same way always place equal keys together.
The cases one key can't cover
Reference tables. Lookup tables such as countries, currencies and tax_rates join with almost everything but belong to no customer. Neki copies them in full to every shard in the listed groups. 6, Neki's workflow for importing data, loads the first copies. After that the router sends every insert, update and delete to all copies. A join against a reference table then stays local. The post doesn't say what consistency the copies have during a write, for instance whether a reader can briefly see different versions on different shards.
Several shard groups. Catalog tables (products, inventory, prices) belong together by 7, the stock-keeping unit code, not by customer. A second group, catalog, hashes lower(sku) and can reuse the same two clusters. Groups don't have to share shards, though. Nayak suggests the catalog might later run on two small shards while customer data grows to eight large ones. There is also a single-shard main group that holds database-wide metadata such as sequences and schema information. Every topology names one group as its authoritative group.
SQL expressions as keys. Shard indexes can hash an expression instead of a raw column. With lower(sku), the vendor-supplied variants AB-1042 and ab-1042 land on the same shard. With left(order_id, 4), an order ID like 4711-00093821 is placed by its store prefix 4711, so each store's orders sit together.
Other placement schemes
xxhash is the recommended type because it spreads rows evenly. Neki also supports two others, mainly so that systems already sharded this way can migrate:
range sends integer key ranges directly to shards. In the example, merchant IDs 0–10,000 (hex 0000–2710) go to eu-1 and 10,000–20,000 go to us-1, which makes placement by geography possible.
modulo computes value % modulus into fixed buckets. Placement is easy to predict, but changing the modulus "moves nearly every row." As an illustration: going from 4 buckets to 5 leaves a key in place only when v % 4 == v % 5, which is true for 4 of every 20 integers. About 80% of rows would move. PlanetScale recommends against modulo at scale.
Queries that lack the shard key
A support agent who runs SELECT * FROM orders WHERE id = 84213 has no customer_id to give, so the router would have to ask every shard. The fix is a global secondary index (8): an ordinary Postgres table that maps order id to customer_id, sharded by order id. With the GSI, the query becomes two single-shard reads. The first gets the customer ID from the lookup table, and the second gets the row from . The cost grows with two lookups, not with the number of shards.
orders "owns" the index, so only writes to orders change it. Neki writes the lookup row "in the same request" as the order row. If the table already has rows, a workflow backfills the lookup table before the GSI is switched on.
One open question: the lookup row is sharded by order id and the order row by customer_id, so they will often sit on different shards. Given the post's own statement that cross-shard commits aren't atomic, it doesn't say what happens if one of those two writes fails, or whether readers can see a stale index.
Changing your mind
Nayak says shard groups are not permanent. The 9 workflow changes a table's shard key or shard count, and MoveTables imports external Postgres databases and can shard them during the import. You will probably reshard as query patterns change. Moving the data is still a significant operation, though, and the post doesn't describe what Reshard costs while it runs.
The takeaway
The design rule fits in one sentence: find the key your most important queries and transactions share, and put every table that carries it in one group. Reference tables, extra groups and GSIs handle what that key misses. The post closes by promoting PlanetScale's tools that examine schemas and query logs and suggest topologies.
Remember
Choosing the shard key decides which transactions keep Postgres's atomicity; cross-shard commits in Neki are not atomic.
Hashing a SQL expression, such as lower(sku) or left(order_id, 4), lets messy real-world keys place rows together.
Modulo sharding is easy to predict but fragile: going from 4 to 5 buckets moves about 80% of rows.
A semantic indexer runs the compiler's own analysis and records every definition (where a symbol is declared) and every reference (where it is used), each under a unique, stable symbol name. The same symbol gets the same name in every repository, so indexes built separately can be joined later.
Overview: Quasar links a proposed API change to observed uses in consumer repositories.1 / 4
Netflix's Java Platform team has built Quasar, an internal tool that tells a library's author which other repositories will be hit by a change to its public API, and how far to trust that answer.
The problem
At Netflix, shared Java libraries are used across many repositories that each change on their own schedule. When a library's author (the producer) removes a deprecated method, a compatibility checker can flag the removal as breaking. It can't say who still calls the method, or on which library version. Without that, producers either ship the break and learn from consumer failures, or keep old APIs forever because the risk is unknown.
Other approaches fall short. Dependency graphs show who declares the library, not who calls a given method. Text search confuses same-named methods and misses inherited calls. Building every consumer against the change works, but it is expensive across the whole fleet.
How it works
Quasar uses 1, an open format in which per-language indexers record every symbol definition and reference under a stable identifier. On each 2 (continuous integration) build, indexes are sent to Quasar and stored as 3 files, a columnar format, in Amazon S3. Each record carries the symbol, dependency version and source location.
On a pull request, Quasar compares API snapshots for the base and head commits and labels symbols as added, removed or signature-changed. For removals and changes, it runs SQL through 4 that joins the changed symbol against stored references to find callers. For Java, it also follows interface implementations so each affected repository is counted once.
In the post's illustrative example, a removed getLegacyValue method is still called by 137 repositories: 86 on version 4.2.1, 38 on 4.1.7 and 13 on 3.9.4. The report includes file paths and line numbers. Rather than merging, the producer would deprecate the method, publish a migration path and chase the stragglers first.
Being honest about evidence
"No references found" doesn't mean safe. Reflection, configuration files and stale or missing indexes can hide callers. So reports state whether the exact commits were indexed, how fresh consumer indexes are, and whether results were truncated. Quasar reports impact. It doesn't enforce release policy.
Limits and next steps
Usage search covers 5, Go, Python, JavaScript/TypeScript and Rust. Judging whether a change is actually breaking works only for JVM and 6, Google's schema language, whose changes Quasar traces through generated Java. The team wants coding agents, which already query Quasar via 7 (Model Context Protocol, a standard for agents calling tools), to check impact while editing and later draft consumer migrations. The post gives no adoption or accuracy numbers.
Netflix's Java Platform team has described Quasar, an internal system that indexes code definitions and references across the company's repositories. It answers a question library authors usually can't answer: if I remove or change this public method, which other repositories still call it, and on which versions?
The problem
Netflix's shared Java libraries are used across a large set of repositories that each change on their own schedule. The producer (the library's author) can see the API diff in their own repository. What they can't easily see is who still depends on the symbols that changed.
The post's running example is removing a deprecated method. A replacement has been available for several releases, the old method can't express options the new behaviour needs, the library's tests pass, and a compatibility checker flags the removal as breaking. The producer still doesn't know whether anyone calls the method, whether callers sit on one library version or several, whether another library wraps the API and so multiplies the impact, or whether a quick message to a few teams will do instead of a full migration campaign.
Without that evidence there are two bad defaults. One is to ship the break and find out from consumer failures. The other is to keep the old API forever because the risk is unknown.
Existing tools each cover only part of this. Binary and source compatibility checkers spot contract changes but not users. Dependency graphs show declared reach but can't prove a particular method is called. Text search misses overloads (same-named methods with different parameters), inheritance, and which artifact version a call resolves to. Building every consumer against the new library gives a strong answer, but doing that across the whole fleet is expensive.
How Quasar works
Quasar is built on 1, an open format for semantic code intelligence. Per-language indexers emit definitions, references and relationships (for example, "this class implements that interface"), each keyed by a stable symbol identifier. Because a symbol has the same identity everywhere, Quasar can link a definition in client-lib to a reference in checkout-service even when the two were indexed separately.
The pipeline:
At its core the match is a join. A producer-side tool supplies the old method's descriptor (its JVM signature). Quasar turns it into a canonical symbol, then joins the requested changes against quasar.symbol_refs on symbol_canonical, returning repository, dependency version and file path.
Worked example
Take an interface change that replaces getLegacyValue(Request) with getValue(Request, Options). The post shows an illustrative report, stitched together from several Quasar queries rather than taken from one real response. It lists 137 indexed repositories that reference the removed method: 86 on library version 4.2.1, 38 on 4.1.7, and a long tail of 13 on 3.9.4. It includes sample call sites, such as CheckoutClient.java lines 84 and 129 in checkout-service and a wrapper class in client-wrapper. It shows a criticality tier of 1, which the library inherits from its most critical downstream application. That tier does not mean all 137 callers are Tier 1. The report also includes flags confirming that both the base and head commits were found in the index.
With that report, the producer's plan changes. Instead of merging the removal, they can keep the method and mark it deprecated, publish a migration path, coordinate the 137 repositories, and track the 3.9.4 stragglers before finally deleting it. The file paths become a work list for engineers or coding agents. If the same diff came with stale or weak evidence, the right move would be to refresh the indexes first.
Being honest about the evidence
The post spends a lot of time on what Quasar's results don't prove. "No references observed" means none were found in the indexed population, nothing more. Reflection (calling methods by name at runtime), configuration files, dynamic loading, stale branches and incomplete indexing can all hide consumers. Reports therefore state whether the requested commits were indexed, how fresh the consumer indexes are, and whether results were truncated. If the exact base or head commit wasn't indexed, Quasar says it can't compare them. It does not fall back to a default-branch snapshot, which could make a missing comparison look like a clean diff.
Quasar also enforces no policy. It reports impact, and the producer decides.
Languages and limits
SCIP lets Quasar index JVM, Go, Python, JavaScript/TypeScript and Rust code for usage queries. Finding usages is a different job from deciding whether a change breaks compatibility, and SCIP doesn't do the second. Today Quasar compares JVM and 8 API definitions. Protobuf is Google's schema language for data exchange, which generates code from .proto files. Using SCIP's relationship data, Quasar can trace a Protobuf change through the generated Java code to repositories that never mention the .proto element. For the other languages, it doesn't yet judge whether a change is breaking.
Toward agents
Today the findings appear on pull requests, and agents can query Quasar over MCP when prompted. The team wants a coding agent to recognize on its own when it is editing a public symbol and check impact before a pull request exists. The pull-request check would still verify the final diff. The authors note that access isn't awareness: the workflow has to detect public-API edits for the agent to know when to ask. They also insist that coverage, freshness and truncation flags travel to the agent in machine-readable form, so it never reads "no observed usages" as "safe to remove." The longer-term plan is for agents to use the call sites to prepare migration changes, while release approval stays with the producer.
What's left open
The post gives no adoption figures, no false-negative rates, no index freshness numbers and no cost data. The 137-repository report is illustrative. The authors' main lessons are to keep "is this breaking?" separate from "who uses it?", and to make evidence inspectable. They argue the pattern carries over to other companies: index code semantically, keep API definitions, diff at the point of change, query consumers, and report how good the evidence is.
Remember
Whether a change breaks the API and who it affects are separate questions. Producers need both answers to choose a release path.
Impact reports must show index coverage, freshness and truncation, so "no usage found" isn't mistaken for "safe".
Semantic symbol IDs let separately indexed repositories be joined with an ordinary SQL query.
Databases keep several copies (replicas) of the data so they survive machine failures. A quorum system like Apache Cassandra confirms a write once a majority of replicas acknowledge it. Consensus protocols such as Paxos and Raft let replicas agree on an order of operations. Every extra copy multiplies storage cost.
witness replicas
Replicas that store only the write journal, not the full data. They vote in quorums to preserve availability without full storage cost.
Murat Demirbas, a distributed-systems researcher, has published notes from days two and three of 1 2026 (High Performance Transaction Systems), an invitation-only database workshop. These are conference notes, not papers, and he recorded nothing for several talks. One theme recurred: moving data now costs more than computing on it.
The tax on every cycle
Devashish Purandare (Microsoft) estimated that 60–80% of datacenter execution is "tax": compression, encryption, copying data, network protocol handling, checksums. No single category dominates. His answer is many small dedicated accelerators. That means 2 (data processing units), network cards with their own cores and fixed-function chips, such as Microsoft's Azure Boost DPU. Natassa Ailamaki (EPFL) found that data movement dominates both time and energy. She argues databases should keep the query plan fixed but pick how to move data at runtime, once they see what downstream steps actually consume.
S3 is built on disks
Viktor Leis (TU Munich) called Amazon S3 "a million magnetic disks": about 10 ms and 80 MB/s per request. In his view, the standard cloud design habits, such as caching, large objects and log-structured writes, all exist to hide disk latency. 3 (a table format for files on S3) manages about 1 transaction per second. He wants SSD-backed object storage, though SSDs now cost 5x as much as disks per byte.
Colin Breck (Tesla) showed the practical side. His team rebuilt a time-series database on S3. Data is stored as 4 files (a columnar format with built-in min/max statistics), which 5, an embedded query engine, reads in place, fetching only the bytes it needs. There are no separate indexes. Six operational headaches disappeared, including weeks-long repartitioning and fragile replication.
Cheaper replication
Scott Andreas (Apple) cut Apache Cassandra's ratio of physical storage to logical data from 6.2x to 1.6x. The steps:
Witness replicas, which vote in quorums but keep only the write log: 4.2x.
Cheaper compaction, so less headroom is reserved: 3.6x.
Better compression: 2.4x.
Two full replicas plus a lightweight "satellite" datacenter: 1.6x.
Failures and coordination
Peter Alvaro (UC Santa Cruz and AWS) called retries an unsolved problem. Retry storms cause 6, where overload persists after its trigger is gone, and postmortems keep blaming individual knobs. Alex Miller (Oracle) tested about 20 consensus protocols on whether each could run a real database. Joe Hellerstein (UC Berkeley and AWS) argued that specifications should mark when inputs become final, because that tells you when coordination is avoidable. Demirbas is unsure that this needs new vocabulary.
Lakehouses
Goldman Sachs said no current system offers fresh data, joins and speed together. Databricks claimed sub-100 ms queries on live tables. SAP's talk title countered that this problem is not solved. An evening panel on AI and code review got heated. Demirbas's verdict: the field "has not processed the AI trauma yet."
Murat Demirbas, a distributed-systems researcher who blogs as Metadata, has posted his notes from the second and third days of 1 2026. HPTS (High Performance Transaction Systems) is a small, invitation-only database workshop. Day one was mostly about AI and specifications. These two days covered hardware, object storage, failures, consensus and lakehouses. They are conference notes, not papers. Demirbas recorded nothing for several talks, and most of the claims below are what the speakers argued on stage. Still, one theme ran through the sessions: moving bytes now costs more than computing on them.
Where the cycles and watts go
Devashish Purandare (Microsoft) said power is the bottleneck in building datacenters today. He estimated that 60–80% of execution goes to "tax." He listed datacenter tax (compression, encryption, data movement, memory allocation, remote-procedure-call handling) and system tax (error handling, checksums, filesystems, the kernel, networking, threading). No single category dominates, so there is nothing obvious to fix first. His answer is many small accelerators rather than one big one. That points to the 2 (data processing unit), a network card with its own processors and fixed-function chips. Microsoft's came from its acquisition of Fungible and now ships as the Azure Boost DPU. It has 6 cores × 4 threads, and each task gets a dedicated accelerator. If the tax is spread over a dozen categories, you need about a dozen accelerators.
Phil Bernstein (Microsoft Research) described the two halves of a DPU. One is a general multicore CPU: low latency, but limited throughput. The other is an ASIC, a fixed-function chip, for repetitive data-path work such as DMA (copying memory without the CPU), compression, encryption and regex matching. The ASIC has high throughput but high latency. DPUs ship from Fungible, AMD Pensando, NVIDIA BlueField and AWS Nitro, possibly millions of units across large cloud datacenters. For a database, the natural job is the storage-to-compute pipeline: decrypt, decompress, filter. His group's storage interface asks for pages by 3 (log sequence number, a position in the database's write log).
Natassa Ailamaki (EPFL) reached the same diagnosis from the database side. Processors compute faster than memory can feed them, and data movement dominates both the time budget and the energy budget. The best data-movement strategy depends on the query and the hardware, with too many interacting factors to model in advance. Her proposal: keep the query plan fixed, but choose the data-movement strategy at runtime, based on how much downstream operators actually consume. In her words, "a system is only as efficient as its least adaptive component."
Bobbi Yogatama (NVIDIA) applied this to GPUs. On a GPU, compute is plentiful while memory capacity and bandwidth are scarce. So the usual trade-off flips: spend compute to save memory, compress data everywhere, and operate on it compressed. His engine, Sirius, plugs into 4 (an embedded analytics database) without modifying it.
S3 is a million hard disks
Viktor Leis (TU Munich) argued that S3 is essentially a million magnetic disks. Files are hash-partitioned across them and protected with 5 (splitting data into pieces plus parity so lost pieces can be rebuilt). Each request takes about 10 ms and moves about 80 MB/s. High throughput, 100 Gbps in his experiments, comes only from many large parallel requests. Leis says the familiar cloud design rules are disk facts in disguise. Seek times force caching. Wide erasure coding rules out in-place writes, which forces log-structured storage. Both push toward large objects. The cost for transactions is stark: 6 on S3 manages about 1 transaction per second (Ginter and Leis, VLDB'26). So every system that wants transactions on object storage builds its own flash cache. Leis says that wears out local SSDs. He wants an SSD-based object store instead. The catch is that SSDs cost 5x as much as disks per byte, up from 3x last year, although flash capacity is improving faster. He dismissed S3 Express, whose 3 ms latency is far from what an SSD can do.
Colin Breck (Tesla) showed the payoff of accepting S3's limits. His team replaced a clustered time-series database with a log-structured one on S3. Data sits in 7 files, and DuckDB queries them in place. It reads each file's footer for the schema and min/max statistics, then fetches only the byte ranges it needs. There are no B-trees or secondary indexes to maintain. Six operational problems went away: weeks-long repartitioning, catastrophic replication failures, risky in-place upgrades, workload isolation, expensive deletes, and custom migration tools. In his account, S3 now handles durability, and those problems came from owning a cluster.
Two proposals addressed sharing. Sergey Melnik (Databricks) wants transactional databases to offer a "diff API." Today's tool is 8 (change data capture), which streams every change event. If a row was updated forty times, CDC hands you forty events to replay. His Differential Change Discovery would return only the net final state since a given LSN, and it can be split up and processed in parallel. Chris Douglas (UC Berkeley) argued that open table formats struggle with concurrent writers and that vendors are responding with centralized commit catalogs that quietly re-create proprietary warehouses. He proposes an append-only shared log of compact "intention records," with governance rules stored in the same log. Each engine checks those rules itself and rejects invalid transactions locally, with no central coordinator.
Replication from 6.2x to 1.6x
Scott Andreas (Apple) measured cost as physical storage provisioned divided by logical bytes stored, for Cassandra fleets of dozens of petabytes. Starting at 6.2x, he cut it in steps:
Witness replicas, which keep only the write journal yet still vote in quorums (an idea from a 1986 paper by Pâris): 6.2x → 4.2x. Cassandra's repair process compares full data copies, which witnesses don't have, so it was replaced with mutation tracking and reconciliation.
Splittable "tablet" ranges, which make moves and compactions cheaper so less headroom needs reserving: 4.2x → 3.6x.
Better compression: 3.6x → 2.4x.
Two durable replicas in two regions plus a "satellite" datacenter: 1.6x.
Failures, consensus and commitment
Peter Alvaro (UC Santa Cruz and AWS) called retries an open problem. Retry storms cause 9, where a system stays overloaded even after the original trigger is gone. Postmortems blame timeouts, queue bounds or token buckets, and none of those explanations settles it. Simulations need calibration, which is hard, so he wants to reason about actual code. Aleksey Charapko (UNH) built MESSI, a simulator that captures dynamics closed-form queueing models miss. He used it on MongoDB Atlas workload management, which uses two queues to keep long tasks from blocking short ones.
Alex Miller (Oracle) ran through about 20 leaderless consensus protocols, asking whether each "can CashDB," meaning run a real database workload. EPaxos: no. Accord and escrow transactions: yes. Joe Hellerstein (UC Berkeley and AWS) argued that specifications need a way to "seal" inputs as final. Sealing turns a non-monotone problem into a monotone one, and by the 10 monotone problems need no coordination. Demirbas isn't convinced that a more detailed ordinary spec couldn't express this.
Lakehouses versus 11
Stefano Stefani (Goldman Sachs) needs fresh data, joins and speed all at once, and no current option delivers all three. Ippokratis Pandis (Databricks) noted that intra-datacenter bandwidth rose from 10 Gbps in 2015 to 6.4 Tbps now. He pitched Lakehouse/RT, which he says serves tens of thousands of concurrent sub-100 ms queries directly on live tables. Daniel Ritter (SAP) countered with his title alone: cloud HTAP (hybrid transactional/analytical processing) is not solved.
Demirbas's main takeaway about people, not systems: an evening panel on AI and code review turned heated. He concluded the field "has not processed the AI trauma yet."
Remember
Leis argues that S3's design rules come from hard-disk physics; Iceberg on S3 manages about 1 transaction per second.
Apple cut Cassandra's physical-to-logical storage ratio from 6.2x to 1.6x, starting with witness replicas.
Microsoft estimates 60–80% of datacenter cycles are overhead spread across many categories, so the fix is many small accelerators.
Store data near users where physical capacity allows. In practice that mostly means data-residency cases, because many regions lack the space, power or cooling.
Replicate small, latency-critical datasets widely. Authentication metadata, needed on every request, is copied to all 40 regions in some cases. This required a routing layer that can move datasets between accounts.
Cache everything else. A lookaside cache built on 4 (an open-source in-memory key-value store) sits near compute. In a lookaside design the application checks the cache first and falls back to the database on a miss. TTLs force a choice between hit rate and staleness, and invalidating on each write breaks under partial failures and cross-region races. Habitat instead follows Meta's TAO: a service reads Cosmos DB's 5 (its stream of committed changes, a form of change data capture or 6) and invalidates cached copies everywhere within seconds. The next read misses and fetches fresh data.
Reference tables such as countries or currencies are copied in full to every shard. The router applies every write to all copies, so joins against them stay local.
More groups: catalog tables can share a separate group keyed on lower(sku), where 5 is the product code. Calling lower() means AB-1042 and ab-1042 land together. Groups can use different shards and shard counts.
Other schemes: range maps key ranges straight to shards, which allows placement by geography. modulo (value % n) is supported but discouraged, because changing n moves nearly every row.
Global secondary indexes: looking up an order by its own id would normally query every shard. A 6 is a lookup table, maintained by Neki, that maps order id to customer_id. The query becomes two single-shard reads.
Key ranges. These split that 64-bit space across physical Postgres clusters. In the example, store-1 owns everything below hex 80 and store-2 owns everything from 80 up. Read as the leading byte of the hash, that is a split at the halfway point.
orders
Ingestion. On each enrolled 2 (continuous integration) build, indexers send SCIP indexes to Quasar. It stores API definitions and consumer references as 3, a columnar file format, in Amazon S3. Each record keeps the symbol identity, the dependency version and the source location.
Diffing. Quasar keeps snapshots of API definitions keyed by commit. Given a pull request's base and head commits, it labels each symbol added, removed or signature-modified. Removals and modifications become candidates for impact analysis.
Querying. Quasar counts direct consumer references with 4, a service that runs SQL over files in S3. A deeper 5 assessment also follows implementations of an interface and the calls made through them, so each affected repository is counted once.
Access. Results are available through REST and 6 APIs and an 7 server. MCP, the Model Context Protocol, is a standard way for AI agents to call tools.
