Chunked caches, shard groups and noisy neighbours

What changed

The range-size trade-off

A range request asks for part of a resource, for example Range: bytes=1048576-2097151 on a 500 MB MP4 file. The server answers 206 Partial Content and gives the exact span in Content-Range: bytes 1048576-2097151/524288000. Clients use this to seek within video, resume downloads, fetch parts of package artifacts, or read the end of a file without downloading what comes before it. Cloudflare never alters what the client asks for. The work is in how the cache satisfies the request.

The previous implementation, built on nginx, worked like the cache slicing nginx describes. Each request was mapped onto fixed-size ranges of the file, and the range size and eligible file extensions were configured per zone. Every possible size had a cost:

• Small ranges turned one request into many small origin requests. All of them were coalesced in memory, which put pressure on Cloudflare's memory.
• Large ranges made Cloudflare download far more than the client wanted. That waste repeated on each hop: origin to the first data center, between cache tiers, then toward the client.
• Changing the size invalidated cached entries, because the size was part of the cache key.

Configured sizes ranged from tens of kilobytes to tens of gigabytes. The article puts it this way: a request smaller than a floppy disk could pull "a few DVDs' worth" from the origin, sometimes after the viewer had already left. R2 used 64 MiB ranges, so a small read that crossed a boundary cost 128 MiB of transfer between data centers. Cloudflare says both failure modes caused real operational incidents.

The fix: use the chunk the cache already has

For a request, the cache checks which of the requested bytes it already holds and serves those. For the missing bytes, it fetches origin ranges aligned to chunk boundaries, stores them in the same chunks used for ordinary caching, and returns only the bytes the client asked for. A request for bytes 150–250 may trigger a fetch of the whole chunk that contains them, but the client receives exactly 150–250.

This does not remove over-fetching. A request smaller than a chunk still costs a full chunk, and ranges that don't line up with chunk boundaries add work. The difference, according to Cloudflare, is that the extra work is now bounded and predictable, not set by a customer-chosen number.

A coordinator, not a lookup

Parsing the header is the easy part. The hard part is building one correct response when the bytes are spread across disk, origin responses, and downloads already in flight. For each request the edge:

1. Looks up the asset's headers and range-coverage metadata.
2. Reads what it can from disk, or from a stream already arriving in memory.
3. Fetches the missing coverage from the origin using aligned ranges.
4. Writes the new bytes and records the completed range in metadata.
5. Joins the pieces and sets Content-Range and Content-Length for the client.

One response might combine a cached prefix, an in-flight download and a newly fetched suffix. If a download already under way covers another request's bytes, the second request attaches to it instead of starting a new origin fetch. That shares work during traffic bursts on popular files. The data model stays the same as before: headers identify the asset version, the body sits in bounded chunks, and metadata records which ranges were written successfully.

Correctness checks

Results and rollout

Cloudflare moved existing range-caching users first, starting with the most painful workloads: many concurrent range requests on very large files with spiky traffic. R2's p99 time to first byte roughly halved across its storage regions. The 6% traffic drop came from the most recent change. Before it, a request was served either entirely from cache or entirely from upstream. Now the coordinator can combine bytes on disk with bytes fetched from upstream in a single response.

Caveats

Enabling the feature does not make uncacheable responses cacheable, and partial origin fetches only help if the origin supports byte ranges. The article gives no figures for small-object or misaligned-range overhead, and no chunk size. Cloudflare plans to make this the default for certain file types. Users can opt in now with origin_range_requests: { "mode": "on" } in the API, or under Caching > Cache Rules in the dashboard.

Why co-location matters beyond speed

How a shard group is defined

Placement lives in a JSON document called the data topology. A shard group has two parts:

Tables in the group (customers, orders, items, addresses) inherit the index. customers gets an override: it uses the same index but with its own id column, because that column holds the customer ID there. As a result, a customer's row, orders, items and addresses all hash to the same point. Per-customer and per-order queries and transactions never leave that shard.

Reference tables

Several groups, one fleet

A product catalog (products, inventory, prices) is naturally keyed by SKU, not customer. Nayak adds a catalog group whose index by_sku hashes lower(sku). It uses the same two shards as customer_data, but only to save money; groups don't have to share shards. If customer data grows, Nayak suggests the catalog could stay on two small shards while customer data moves to eight large ones. A third group, main, is a single-shard home for database-wide metadata such as sequences and schema information. Every topology names one group as its authoritative shard group.

SQL expressions as shard keys

Shard indexes can hash an expression instead of a raw column. lower(sku) sends AB-1042 and ab-1042, which vendors often send in mixed case, to the same shard. If an application embeds a store ID in its order IDs, left(order_id, 4) routes 4711-00093821 by store 4711, so all of one store's orders sit together.

Other placement algorithms

xxhash is the recommended choice because it spreads keys evenly. Neki also supports two other types, mainly so applications that already shard this way can migrate without changes:

• range maps integer key ranges directly to shards. In the example, merchant IDs 0–10,000 (hex 0000–2710) go to eu-1 and 10,000–20,000 (2710–4e20) go to us-1. This is useful when IDs were assigned by geography.
• modulo computes value % modulus, for example 4 buckets with buckets 0–1 on store-1 and 2–3 on store-2. Placement is easy to predict, but changing the modulus moves nearly every row. Going from 4 to 5 buckets, for instance, changes the bucket of about 80% of keys. Nayak advises against it for most systems at scale.

Queries that lack the shard key

Changing your mind later

The design rule

Nayak's conclusion: shard group design starts from your queries, not your tables. Find the key that your most important queries and transactions share, and put every table that carries that key in one group. Then use reference tables, expression keys and GSIs for the rest. The post ends with a pitch for PlanetScale tools that analyse live schemas and query logs and suggest a topology.

Where shared-kernel isolation runs out

The workloads hit hardest were stateful streaming jobs such as Apache Flink stream processors. When one of these falls behind, it may have to restore from a checkpoint and spend minutes catching up. The authors describe this as a stop-the-world event that pages someone, not a slowdown that averages out. The same teams said they never saw it on plain EC2. Netflix has worked on this before: predictive CPU pinning to reduce cache interference, eBPF probes that surface per-container run-queue latency, and an investigation of a global kernel lock. Each helped, but none removed the shared hardware.

Four options, and why a VM won

The authors also argue that the threat model was wrong for the sandboxes. microVMs and gVisor are built to contain hostile tenants. Netflix's internal teams need resource and performance isolation, not protection from each other. A dedicated EC2 VM provides that with no extra virtualization layer.

How it works

When a task exits, its node is drained in two stages, so a pending deploy can reuse it before termination. New host images (AMIs) roll out either in place without touching the running container, or only to newly created nodes.

The costs

Launch latency. On a shared host, a task usually lands on a machine that is already booted and has its image layers cached, so it starts in seconds. A single-tenant task waits for a fresh instance, tens of seconds before the image pull even starts, plus the image pull and application startup. Warm pools of pre-booted instances work less well here. Right-sizing spreads demand across many instance shapes, and a warm r7i.2xlarge does nothing for a task that needs an m7a.8xlarge. Netflix keeps only a small warm pool and narrows the menu to a curated set of sizes, giving up some fit for a better hit rate. It is putting most of its effort into making launches cheaper:

• lazy-pulling image snapshotters (stargz, SOCI) that fetch only the bytes a container actually reads at startup
• CRIU checkpoint/restore of warmed-up processes
• non-metal instances, which boot faster
• disabling Kubernetes preemption logic, which stress tests showed was leaving fresh nodes idle while pods waited to be scheduled

Fixed overhead. Every host runs sidecars, the kubelet and system services, which take a roughly fixed slice of CPU and a few GB of memory. On a small dedicated node that can be tens of percent of the machine. On the largest shapes it is about a couple of percent. Add the gap between what the task requested and the next instance size up, and the guidance follows: single tenancy suits large, long-running, disruption-sensitive workloads sized close to an instance shape. It is a poor fit for small, bursty or elastic ones. This is also why the mode is allowlisted and not self-serve.

Control-plane load. Many small nodes mean more for Titus to track and more EC2 launch calls, which matters most during a regional failover. Netflix migrates in stages and runs scheduled stress tests of the launch path.

Results, and their limits

The qualitative claims are these. Flink, connector and telemetry teams report stable operation and are no longer moved around by repacking. Some GraphQL and edge services have followed. On efficiency, Netflix argues that moving the few perpetrators and the sensitive victims off shared hosts removes the anti-affinity spreading and the pods that block defragmentation, both of which stranded capacity, so the remaining hosts pack more densely. It also says right-sizing captured more efficiency in practice than multi-tenant bin-packing, whose higher theoretical ceiling is lost to holes left by discrete instance sizes. Patching is simpler too: draining a dedicated host means moving one task.

Next steps on the shared fleet: replacing HTB with eBPF-based Earliest Departure Time pacing, which rate-limits by timestamping packets, and evaluating Intel RDT and AMD PQoS cache and memory-bandwidth partitioning plus NVIDIA MIG for GPUs. Longer term, Netflix wants a single Compute API where teams state intent, such as how much isolation they need and how fast a task must launch, and the platform chooses between VMs and containers.

Netflix's Java Platform team has described Quasar, an internal tool that answers a question library owners usually can't: if I change this public method, who breaks? Quasar indexes symbol definitions and references across Netflix's repositories. When a pull request removes or changes an API, it reports how many repositories use that exact symbol, which library versions they're on, and where the calls are.

The problem: compatibility isn't the same as reach

Take a common case: removing a deprecated method. Deprecated entry points enlarge the surface a library team has to test, document and support, and they can block new designs. Suppose a replacement has existed for several releases and the old signature can't express options the new behaviour needs. The library's tests pass, and a compatibility checker flags the removal as breaking. The producer (Netflix's term for the team that owns the library) still doesn't know:

• whether any consumer still calls the method;
• whether consumers sit on one library version or are spread across several generations;
• whether another library implements or wraps the API, which multiplies its impact;
• whether a direct conversation with a few teams is enough, or a full deprecation and migration campaign is needed.

Each existing technique sees only part of the answer, the article notes. Compatibility tools find contract changes but not users. Dependency graphs show who declares the library, not who calls a given method. Text search misses overloads, inheritance and the version behind a call. Building every consumer against the new version is strong evidence but costs too much across the whole fleet. Without usage data, producers either ship and learn from consumer failures, or keep the old API forever because the risk is unknown.

How Quasar works

The pipeline:

SELECT c.change_id, 'DIRECT_REFERENCE' AS evidence_kind,
  r.scm_type, r.org, r.repo, r.dep_version, r.file_path
FROM requested_changes c
JOIN quasar.symbol_refs r
  ON r.symbol_canonical = c.target_canonical

A worked example

The article's simplified change replaces a method on an interface:

public interface Client {
-   Result getLegacyValue(Request request);
+   Result getValue(Request request, Options options);
}

An illustrative report, which the article says combines several Quasar queries rather than one literal response, shows 137 repositories referencing Client#getLegacyValue(Request). By library version: 86 on 4.2.1, 38 on 4.1.7, 13 on 3.9.4. It includes sample file paths and line numbers, such as CheckoutClient.java lines 84 and 129 in checkout-service, and flags that one consumer is client-wrapper, a library that wraps the API. The library carries criticality Tier 1, inherited from its most critical downstream application. That doesn't mean all 137 callers are Tier 1.

That changes the plan. Rather than merging and waiting for failures, the producer can keep and deprecate the method, publish a migration path, and track the 13-repository long tail on 3.9.4 before removing it. The file locations become a concrete work list.

Treating "no usages found" with caution

The article stresses evidence quality. "No references observed" means only that none were found in the indexed population. Reflection, configuration-driven or dynamic loading, stale branches and incomplete indexing can all hide consumers. So reports state whether the base and head commits were found, how fresh the consumer indexes are, and whether results were truncated. If either commit's snapshot is missing, Quasar refuses to compare rather than falling back to the default branch, so a missing comparison never looks like a clean diff. The same removal with stale evidence should lead to a different next step: refresh the indexes first. Quasar reports and doesn't enforce. Release decisions stay with the producer.

Polyglot, with limits

Lessons and open questions

The team lists four lessons. Treat compatibility and reach as separate signals. Make evidence inspectable with call sites, versions and freshness. Keep usage analysis apart from per-ecosystem compatibility rules. Deliver evidence while choices are still open. Next, they want coding assistants to query Quasar while code is being written, before a pull request exists, and to use found call sites to draft consumer migrations.

The article gives no fleet-scale figures: not how many repositories are enrolled, typical index freshness, query latency, or how often findings changed a release decision. It also doesn't quantify how many consumers the blind spots (reflection, unindexed repositories) miss. The authors argue the pattern carries over to other organisations: index semantically, keep API definitions, compare at the point of change, query consumers, and report evidence quality.