Same Cache Key, Different Bytes

Why Hyperdrive exists

Opening a SQL connection takes at least five network round trips. Each open connection also costs the database server resources. MySQL spends less per connection than Postgres because it doesn't use a whole OS process for each one, but large connection counts still cause trouble. Opening a connection per request on a serverless platform would add latency and pile up connections. Hyperdrive instead keeps warm connections open on Cloudflare's network and shares them across many Worker requests. It can also cache common query results in Cloudflare's content delivery network (CDN) cache.

Two smaller bugs first

The post describes two encoding bugs fixed during the beta.

• Missing emojis. Hyperdrive asked for the utf8 character set during the connection handshake. In MySQL, utf8 is a deprecated alias for utf8mb3, which allows at most 3 bytes per character. Real UTF-8 allows up to 4, and emojis need all 4. Switching the handshake to utf8mb4 fixed it.
• Encoding lost after a reset. Between checkouts, Hyperdrive sends COM_RESET_CONNECTION, a lightweight MySQL command that clears session state so nothing leaks from one borrower to the next. The command resets the character set to the server's default, not to the one negotiated at startup. On a server that defaults to latin1, a connection that started as utf8mb4 became latin1 after the first reset. Hyperdrive now sets utf8mb4 again after every reset.

The rare one

The beta customer with the most MySQL traffic saw a small, steady rate of empty responses to queries that should always return rows. At about 0.001% of requests, even a higher trace sampling rate rarely captured an end-to-end example. The team found no network explanation. The only solid clue was that the bug appeared only on Hyperdrive instances with caching turned on.

Topology: two nodes, sometimes one cache

Each Hyperdrive instance runs as two nodes:

• the Client, on the same machine as the user's Worker. It handles caching and forwards cache misses onward.
• the Endpoint, in the data center closest to the origin database. It holds the connection pool.

Each node uses the CDN cache of the data center it runs in. When both are in the same data center, they share a cache. That normally helps: the cache key is a normalized hash of the query's wire messages, and the value is the result bytes. A Client can then serve a result that its Endpoint cached a moment earlier. This only works if both nodes store the same bytes for the same response.

Where the bytes diverge

Hyperdrive is a server to the Worker and a client to the origin database. Toward the origin it always uses a fixed set of capabilities so it can parse the replies. The Worker's driver may support a different set, so the Hyperdrive Client translates each response into the format that driver expects.

The capability that mattered was CLIENT_DEPRECATE_EOF. Without it, a result set has an EOF packet after the column definitions and another EOF at the end of the rows. With it, the first EOF is dropped and the last one becomes an OK packet. MySQL 5.7.5 and later supports this, but only uses it if the client asks. Hyperdrive asks for it. The popular Node.js driver mysql2 doesn't. So for mysql2 users, the Client rewrites every result: it adds the metadata EOF and turns the final OK into an EOF.

The flaw came from the Postgres implementation. The Client cached exactly the bytes it wrote back to the Worker. Postgres doesn't need this kind of translation, so that was safe there. In MySQL, the Client's bytes are in the Worker's format and the Endpoint's bytes are in Hyperdrive's format, yet both sit under the same cache key.

Two ways to fail

Endpoint writes, Client reads. The Endpoint gets response R for query Q from the origin and caches it as R_hyperdrive, in Hyperdrive's format. A Client in the same data center gets Q, finds R_hyperdrive in the cache, and sends it to the Worker without translating it.

Client writes, Endpoint reads. A Client caches its translated R_worker. The Endpoint, handling Q for a different Client, finds R_worker and forwards it. That Client translates the already-translated bytes, then either sends an EOF too early, which the driver reads as an empty result, or hangs.

To make this concrete (an illustrative reading, not a trace from the article): suppose a mysql2 Worker receives R_hyperdrive. Its driver expects an EOF after the column definitions and doesn't see one. That throws off where the driver thinks the rows start and end. Both failures need the two nodes to be in the same data center, caching on, and one node to read the other's entry. That explains why the rate was so low.

The fix and the result

Both nodes now write one canonical form to the cache: result bytes in Hyperdrive's own capability encoding, no matter which node stores them. Translation to the Worker's format happens only on the way out. After the release, the empty results stopped and connection success from Workers reached the team's 99.99% target.

What's left open

The post doesn't cover how the team checked the fix beyond the success-rate number. It doesn't say whether other capability flags, such as compression, could cause similar mismatches. It also doesn't list the other ~17 fixes. The general lesson is clear, though: a cache key has to identify both the logical result and the exact encoding of the stored bytes. If any writer can produce a different encoding, store one canonical form or include the format in the key.

Netflix has replaced its first cold-storage design for time-series data, a Kafka and Flink pipeline that copied events into S3. In the new design, Cassandra reads old data directly from the disaster-recovery backups it already keeps in S3. The new tier holds more than 15 PB of compressed data. Disk usage on the affected clusters fell from 80% to 7%. p90 cold-read latency is about 30% better than the old pipeline, and p99 stays under roughly 500 ms.

The problem

The TimeSeries Abstraction is Netflix's service for storing and querying event data ordered by time. It runs on Cassandra and backs datasets of several petabytes, on storage tuned for millisecond reads. Users keep asking for longer retention, and most of that history is rarely read. It still has to be available for audits and investigations, sometimes with sub-second latency. Keeping all of it on dense hot nodes is expensive, and the density makes scaling, deployments and cluster stability harder. The goal was to move old data to object storage without changing the API.

Attempt one: stream a second copy to S3

A daily compaction job fixed that. It merged the small files into one daily Parquet file per namespace and applied three read optimisations:

On the largest datasets the job takes about 12 hours, a choice that favours low compute cost over speed. When it finishes, the files are registered with the control plane and the hot slice is truncated.

On the read side, the data plane sends each query to both tiers and merges the results. Because metadata prunes so much, reading a full day for one ID takes at most three sequential S3 reads. Latency per read was about 150 ms at p50 and about 500 ms at p99, at up to 1,250 cold reads per second globally.

It worked, and it shrank both the local disks (EBS volumes) and the backup footprint. But Kafka, Flink and the compaction job cost money, add moving parts to on-call, and pay to transfer data into S3 that the backups were already putting there. So the team asked whether Cassandra could read its backups directly.

Attempt two: serve reads from the backups

Using the backup layout. Backups are stored per node, keyed by the node's ring token. Each snapshot has a JSON manifest listing every SSTable component: the large Data.db file, plus Index, Filter (Bloom filter), Summary, Statistics and CompressionInfo. Every component except Data.db is small. So Cassandra downloads those to local disk and does query planning locally, then fetches only the needed data chunks from S3.

When the memtable opens, a state machine pulls the metadata files from S3 once. Several safeguards protect this step:

• S3 calls retry with exponential backoff.
• The download runs on its own thread pool, away from normal Cassandra work.
• A file whose size doesn't match the manifest is deleted and downloaded again.
• A .timestamp file ties the download to one backup. If the table's backup timestamp changes, the download restarts.
• Reads that arrive during setup wait at most half the read RPC timeout.

Reads run S3 fetches in parallel across requests, SSTables and chunks. A shared cache removes duplicate fetches and prefetches chunks. Three settings cut down S3 calls:

Isolating load. Reading from S3 costs much more CPU than reading local disk, and a burst of cold reads could hurt hot traffic. So cold reads can optionally run on a separate Cassandra cluster whose memtables point at the hot cluster's backups. That cluster must match the original's topology but can run on smaller, cheaper, storage-light instances.

Results

The rollout covered the most storage-heavy clusters:

• More than 15 PB of compressed data now sits in the cold tier.
• Disk usage fell from 80% to 7%.
• The tier serves thousands of cold reads per second in each region.
• p90 latency is about 30% better than the Flink design, which Netflix credits to more efficient S3 access and smaller reads.
• p99 stays under roughly 500 ms.
• Cassandra's chunk cache absorbs bursts.

What's left open

The post gives no dollar figures for either design. Backups now carry live reads, so backup retention, deletion and corruption become availability concerns, and the post doesn't say how those policies changed. A slice can only go cold after its backup is verified, which ties tiering to the backup schedule. The design relies on Cassandra 4.1 internals and a custom memtable and compaction strategy, and the post doesn't say whether either will be released.

The planner's default: push work down

In Neki every shard is a complete Postgres with its own planner. Rydberg says the ideal case is relaying the whole query to one shard, but that is usually impossible because each shard holds only its slice of the data. In the most general case a shard only has to return raw rows, and the router, the component that receives client queries, must be able to do everything else.

Take SELECT age + 1, lower(name) FROM animals. The expressions could be evaluated on the shards or on the router, and both give correct results. Choosing between them is an optimisation problem with several inputs: which side has spare resources, which evaluates faster, whether the computed value or the raw column is cheaper to send over the network, and whether a nested expression should be split between the two. The post assumes a simple heuristic for clarity: push evaluation to the shard whenever that is allowed.

Router-only expressions

That gives Neki's general planning model: find the parts of the query that can't run on shards, plan them to run in the router, and push everything else down.

Stable isn't the same as safe to push down

Sequences: one authority, every call central

Cardinality: how many times to call it

A worked example from the post: animals has rows (0, BISMUTH) and (1, DOGE). feedings has (0, morning), (1, midday), (1, evening) and (2, morning). The inner join on id gives three rows, because DOGE appears twice. Adding WHERE feedings.time = 'morning' leaves one row, so nextval('tag_seq') runs once and uses one value. Without the WHERE clause it runs three times. An optimiser might notice that age + 1 and lower(name) are the same on both DOGE rows and compute them once. It must not do that with nextval().

Join order adds a further subtlety. Filtering before or after a join is legal and is a standard optimisation, but Neki and Postgres may choose different orders. If a volatile expression sits inside a WHERE or ON predicate, Rydberg notes that results are "not always deterministic, but they are correct".

Distributed joins

Network round trips between router and shard dominate Neki's costs. A general cross-shard join pulls both tables to the router and joins them there. When the join predicate uses each table's shard key (for example, both tables sharded on id and joined on id), matching rows always live on the same shard. Each shard can then do the join locally and return its share of the table expression.

Take SELECT age + 1, lower(name), nextval('tag_seq'), now() FROM animals JOIN feedings ON … with both tables sharded on id. The join and the two immutable columns run on the shards. The router evaluates now() once and copies it to every row. The router can't know how many rows each shard will return, so it can't fetch a block of sequence values ahead of time. It has to call nextval() on each row as the merged result streams through it.

UPDATE with nextval()

UPDATE animals SET tag_id = nextval('tag_seq') is harder. The row count is unknown, the rows are spread across shards, and the shards can't evaluate nextval() themselves. Rydberg says there is more than one possible solution. Neki currently does this:

1. Send each shard a SELECT for only the columns it needs, here the primary key, which is also the shard key (id), with FOR UPDATE so the rows stay locked.
2. As each row arrives at the router, evaluate nextval().
3. Send a parameterised update for that row, routed to its shard by id and filtered to the single row by primary key.

This makes one update statement per row, plus an initial locking read. The post gives no performance numbers and doesn't say how this costs out on large tables. That trade-off is the obvious open question. The post also leaves the stable-expression version (SET col = now()) as an exercise, calling it simpler because it doesn't depend on row count.

GitHub has shipped an API endpoint that returns up to 30 weeks of daily star counts for a repository in one request, with no user identities attached. It replaces a workaround that third-party tools had used for years. Those tools paged through the full list of people who starred a repo, kept the timestamps and threw away the names. Taylor Cuilty's write-up for GitHub Engineering explains why the obvious redesign, returning counts instead of people, would not have made the queries cheaper, and what did.

Two problems with one endpoint

The old endpoint, GET /repos/{owner}/{repo}/stargazers, answered "who starred this repository?" It returned up to 100 users per page. If the caller asked for a custom media type, each entry also carried the time of the star. Tools built growth curves from those timestamps. That design caused trouble in two places.

Privacy. Because the usernames were public and could be pulled in bulk, bad actors scraped stargazer lists to target spam and phishing campaigns.

Database load. The cost of building a curve grew with the repo's popularity. react/react has about 251,000 stars, so its full curve took about 2,511 requests, and tools often hit rate limits or only sampled some pages. A pagination bug made each request worse. To work out how many pages existed, every request counted all of the repo's stars, even though it returned only 100. In the week before GitHub restricted access, nearly 37,000 of these count queries ran slowly enough to be logged. They averaged almost 3 seconds each, and database safeguards killed more than 1,000 of them, which contributed to request timeouts.

The data model added a quirk. The star table stores the current set of stargazers. It is not an event log: when someone un-stars a repo, the record is deleted. Any curve rebuilt from it shows when today's stargazers starred the repo, not the true historical count.

Step one: restrict, then ask

Approaches that didn't fix the cost

Timestamps without identities. The first prototype kept one record per star and dropped the username for callers without access. This was a small change and kept existing tools working. But clients would still page through hundreds of thousands of records to build a curve. It also left trend data inside a pagination model designed for listing people.

Aggregate, then paginate. The next idea was to group stars into daily or weekly buckets in SQL and page through the buckets. The trouble is the order in which a simple query does its work:

all matching stars -> bucket every row -> sort buckets -> OFFSET/LIMIT -> response

LIMIT and OFFSET run last. Each page request still reads and groups every star in the repo, then throws most of the buckets away. The JSON response gets smaller, but the database does the same amount of work. For react/react, reading the whole history in 24 pages would mean examining about 6 million rows: 24 requests × ~251,000 stars.

What shipped: filter by time first

The new endpoint paginates by calendar week. It turns page and per_page into a range on the indexed created_at column, fetches only the stars in that range, and then groups them into daily counts:

indexed date range -> group rows in that range -> daily buckets -> response

Each response covers up to 30 weeks, and higher page numbers go further back in time. A worked example using the article's numbers for react/react:

• Old path: ~2,511 requests of 100 users each, each one also counting all ~251,000 stars.
• Aggregate-then-paginate: 24 requests, ~6 million rows examined in total.
• Shipped design: ~24 requests, ~251,000 rows examined in total, because each star falls into exactly one week range.

The number of requests now depends on how old the repo is, not on how many stars it has. The response has the same shape as the existing weekly commit-activity endpoint (/stats/commit_activity). GitHub picked weeks because maintainers usually look at growth over weeks or around a known event.

Side effects

Trade-offs that remain

Filtering by date limits each request to a fixed span of time, but not to a fixed number of rows. A week with an unusually large number of stars, such as a launch that goes viral, can still produce a slow, expensive query. The article calls these dense weeks a remaining scaling limit and doesn't say whether there is a cap or fallback.

Because the star table drops un-starred records, the curve still reflects only current stargazers. The article lists removed stars and calendar boundaries as parts of the API contract users should know about. It doesn't explain how partial weeks at either end are handled, or the exact name of the new endpoint and its parameters beyond page and per_page.

The argument

Cuilty draws four lessons. A smaller response doesn't mean a cheaper query, so look at the order the work happens in. Start from what users are trying to do; here that was a trend, not a list of people. Return only what that use needs, because anything extra can be misused. And state the remaining limits openly, as part of the contract. The engineering result comes from the first lesson: moving the filter ahead of the grouping step so the index does the narrowing.

Netflix engineer Prashanth Gedde Narayanaswamy expected a two-week tuning job on member-sessionizer. It took two months. The pipeline is one of Netflix's largest: every hour it groups tens of billions of client events (taps, playbacks, engagement signals) into viewing sessions per profile. His write-up covers two things. One is how he changed his workflow so he could run many full-scale experiments at once. The other is what those experiments found, which was several Spark memory traps, the cause of each, and the fix.

Why small samples didn't work

The cause was skew. A Netflix tab left idle and streaming overnight produces tens of thousands of telemetry events, while a normal session has a handful. A 1% sample almost never contains those extreme sessions, and they are what kill executors. So every test had to run on a full, unsampled hour of production data. Each run took hours, which allowed only three or four attempts a day.

Running experiments in parallel

That makes each branch one experiment. He made the scheduler's settings into parameters (session cap size, number of shuffle partitions, a threshold for flagging heavy devices, feature flags) and gave each variant its own branch and output table. Five variants then ran at the same time over the same hour of data. Wall-clock time dropped from the sum of five runs (about 10 hours) to the length of the slowest one.

Per-branch tables cost nothing extra to manage. Dataflow creates and updates tables from declared schemas, so changing the event-bag structure between experiments needed no hand-written CREATE TABLE or ALTER TABLE statements. The real limit was the shared cluster: at peak hours, competition with company-wide jobs sometimes cut him to two concurrent runs.

The experiment log

Every run added a row to a Markdown table in the repository with these columns: run number, JAR version, the change from the previous run, the config, and the key metrics (spill, failed tasks, peak memory, duration). He says the "change vs. previous run" column enforced discipline. Whenever he changed several things at once, the log made it obvious that the result couldn't be attributed to any one of them, and he had to rerun. A separate document turned the findings into a lookup table from symptom to the setting that fixes it.

Root cause: rows got heavier

The OOMs traced back to collect_list. Spark holds each session's entire event bag in memory, so what matters is the size of each bag, not the total volume of data. The legacy pipeline had processed the same number of rows without crashing, which meant each event had become heavier. Two changes did that:

Removing Option from fields that are never null cut peak executor memory from about 112% of the slot to 58%, with zero spill. Dropping unused fields and moving per-session values (app version, country, connection type) into the session header so they are stored once also helped. The author's conclusion is that reducing each event's size did more for stability than adding cluster memory would have.

Two results he didn't expect

A fix that only worked off-peak

To cap session size, he used row_number().over(Window...) and kept the first N events. The change passed off-peak tests and then crashed at peak. Spark's Window operator buffers an entire partition (here, a whole session) before it assigns row numbers, so the filter limited the output but not the memory used during buffering. The fix was to truncate before the Window stage. His lesson is to test against worst-case skew.

What shipped, and what's still open

Four changes shipped, each tested alone first: unboxing never-null fields, slimming the event bag, capping session size before the Window stage, and the 10 MB runtime broadcast threshold.

The author is clear that the underlying problem remains. Putting a whole session into one row has a hard ceiling. When the upstream telemetry adds attributes, each event gets heavier again, and memory pressure will come back with no code change on his team's side. In his words, tuning "buys time, not infinite scale." The data model has to change eventually. The post doesn't say what the new design would be.