Same Answer, Different Bytes

What Hyperdrive does

Two smaller bugs first

Both were about character encoding. Hyperdrive asked MySQL for utf8 encoding when it connected. In MySQL, utf8 is a deprecated alias for utf8mb3, which allows at most 3 bytes per character. Real UTF-8 can use 4 bytes, and emoji need all 4, so emoji came back corrupted. Switching to utf8mb4 fixed that.

Then Hyperdrive started cleaning each connection between uses with COM_RESET_CONNECTION, a MySQL command that clears session state so one user's settings don't leak into the next user's requests. That command doesn't restore the encoding agreed at connection time. It restores the server's default encoding, which might be latin1. The fix was to set utf8mb4 again after every reset.

The hard bug

The highest-volume MySQL beta customer saw a small, steady rate of empty results from queries that should always return rows. Telemetry, user logic, network checks and a raised trace rate all turned up nothing. At a 0.001% rate, a full end-to-end trace of a failing request was hard to capture. The one clue was that the bug only appeared when caching was turned on.

How Hyperdrive is laid out

Each Hyperdrive instance has two nodes:

• Client: runs on the same machine as the user's Worker, handles local caching, and forwards queries when needed.
• Endpoint: holds the connection pool and runs in the data center nearest the origin database (the customer's actual MySQL server).

Each node uses the CDN cache of the data center it runs in. When both nodes sit in the same data center, they share one cache. That helps performance, because the Client can serve a result the Endpoint just cached. The cache key is a normalized hash of the query's wire messages, and the value is the raw result bytes. The scheme only works if a given response is stored as the same bytes no matter which node stored it.

The protocol detail that broke it

The flaw came from the Postgres code. When Client-side caching was built for Postgres, it assumed the Client could cache exactly the bytes it sent back to the Worker. That holds for Postgres. For MySQL it fails, because the Client's output has already been translated into the Worker's format. The Endpoint always works in Hyperdrive's own format.

Two ways it failed

Both cases need the Client and Endpoint to be in the same data center, sharing a cache.

The Endpoint caches and the Client reads. The Endpoint gets response R for query Q from the origin and caches it in Hyperdrive's format (call it R_hyperdrive). A Client then gets Q, finds R_hyperdrive in the cache and sends it to the Worker untranslated. A mysql2 driver expecting EOF packets gets OK-style packets instead.

The Client caches and the Endpoint reads. A Client finishes serving Q and caches its translated bytes, R_worker. The Endpoint, serving Q for some other Client, finds R_worker and forwards it. That other Client then translates bytes that were already translated. It either ends the result early with a premature EOF, which shows up as an empty result, or it hangs.

An illustrative analogy: two clerks file the same document in one cabinet under the same label. One files it in English and the other in French. Each clerk assumes the cabinet holds their own language, and the label doesn't say which one it is.

The fix and results

The cache now always holds one canonical version of the result, encoded with Hyperdrive's capabilities, whichever node writes it. Translation into the Worker's format happens only on the way out. After the change, the empty results stopped. Client connection success rate reached 99.99%, the availability target Cloudflare set for the beta.

What's left open

The post doesn't say how the team confirmed the diagnosis. It doesn't mention a reproduction, a test, or how many hours of traffic were checked after the fix. It also doesn't explain why the rate was as low as 0.001%. Presumably it depended on how often Client and Endpoint ended up in the same data center and on cache timing, but the article doesn't measure this. The other ~18 fixes aren't described. The wider lesson, which the post implies more than states, is that any cache shared by components speaking different dialects needs one agreed encoding at rest.

The team's central claim is that a judge is "not a static artifact." It has a lifecycle with four phases, and humans own each one.

Why a judge is hard to trust

The judge solves the volume problem and raises a new one: who checks the checker? If it slowly starts approving bad text, nothing downstream notices. There is also a quieter failure. A judge can agree with humans on every verdict and still be getting there for the wrong reasons. That looks fine on a dashboard. But the judge's explanations steer the generator's rewrites, so wrong reasons mean misleading feedback. And reasoning that happens to produce the right labels is unlikely to keep doing so on new data.

The cast

Four automated parts:

Around them are human writing experts, who define the criteria, and trained raters, who supply labels and rationales.

Phase I: a benchmark with reasons attached

Public judge benchmarks test general text quality. That doesn't fit short, title-specific blurbs held to Netflix's own editorial bar, so the team built its own benchmark from three sources:

• adversarial cases written by experts, each with a known failure;
• LLM-made examples deliberately pushed close to the pass/fail line, then rated by humans;
• real samples from the generator.

Every FAIL label comes with a free-text rationale. The labeling guidelines double as the judge's starting rubric. The benchmark is never frozen: monitoring adds new human-rated examples every week.

Phase II: Reasoning-Aligned Rubric Tuning

1. Scores the training data with the current rubric.
2. Measures alignment on a held-out validation set and stops once every metric clears its threshold.
3. Otherwise, collects a "focus set" of mistakes and hands it to the Reflector, which proposes a new rubric.

What's new is how a mistake is defined. The focus set includes ordinary label disagreements, plus a second kind: cases where judge and human both said FAIL but the Meta-Judge finds their reasons don't match.

The worked example: a member watched a musical, a family fantasy with a young heroine and a well-known ensemble cast. The system recommends an adventure film that is also a family fantasy with a young heroine, but has no songs and no ensemble cast. Judge and human both fail the explanation. Side by side, though, the judge's reason turns out to be incomplete. A label-only check counts this as agreement. RART counts it as an error and makes the next rubric revision account for it. The Meta-Judge was checked first: on a sample of agreed-fail pairs, it matched trained human raters on "nearly all." The article gives no exact figure.

Each criterion is scored on three metrics:

The ablation. "Vanilla" tuning is the same loop with no Meta-Judge and no human rationales. On two of the three must-have criteria, both methods ended in the same place, because the starting rubric was already near its ceiling. On the third, vanilla tuning rewrote the rubric into "degenerate forms." Specificity and reasoning agreement fell every iteration and ended below the starting point. RART stayed stable on that criterion, improved both metrics, and gained a little pass-recall. The post shows these results as a chart only, with no exact figures in the text.

Phase III: gate and critic

The aligned judge does two jobs. As a gate, it rejects any explanation that fails any must-have criterion. As a critic, its rejection reason is added to the generator's prompt for a retry. That makes a generate → judge → revise loop.

Netflix ran 1,000 explanations through up to 12 revisions with four different generator models. The share that passed rose with every round. Most of the gain came early, and the gap between generators was large. The authors' summary: revision "amplifies a capable generator rather than rescuing a weak one." In production they allowed 3 retries with the strongest generator. Anything still failing was dropped, not shown. A bad explanation damages trust, while a missing one only loses an opportunity.

Each week, about 300 explanations were sampled across served, revised and dropped outcomes, weighted toward newer titles. A panel of at least three raters reviewed each one, and the majority label counted as ground truth. The bar for the judge is set by how much the humans disagree with each other. For each metric, the judge must score at least the raters' average minus two standard deviations. On a hard week, raters disagree more and the band widens. If the judge falls below the band on any metric for any criterion, that is a drift event and triggers re-tuning on the enlarged benchmark. No new rubric goes live without a person signing off, and the previous version is kept for instant rollback.

Reviewers also caught problems the metrics couldn't:

• accurate attributes that overstate how similar two titles are;
• vague adjectives like "fun" that technically pass;
• reference titles a member wouldn't see as comparable;
• stand-up specials with matching tags but different tone;
• generic "word soup."

Some fixes went into the guidelines, which update the raters' instructions and the judges' rubric together. Others became product rules outside the judge, such as blocking certain title types from being used as references.

Caveats

Most results appear only as charts. The Meta-Judge's own accuracy is described as "nearly all," with no number. The reasoning-alignment gain shows up on only one of three criteria. The full numbers are in the accompanying research paper, which the post summarizes.

Each Neki shard is a complete Postgres database with its own planner, but it holds only a slice of the data. A router in front receives each query. Rydberg says the ideal is to pass the whole query through to one shard. Usually that isn't possible, so in the general case shards supply raw rows and the router does whatever they can't.

Take SELECT age + 1, lower(name) FROM animals. Either side could compute age + 1, and choosing well is a real optimization problem: which side has spare capacity, which is faster, and whether shipping raw columns costs more than shipping computed values. For the post, Rydberg assumes a simple rule: push work down to the shards whenever that's allowed.

Router-only expressions

Some expressions must never be pushed down. now() returns the wall-clock time at the start of the transaction. If each shard computed it, clock drift between machines and different request arrival times would give different rows different timestamps. That breaks what now() promises. So Neki computes it on the router and attaches the same value to every row.

The router doesn't know in advance how many rows will arrive, so repeated evaluations must return the same answer. Rydberg describes two ways to do that: compute the value once into a one-row, one-column table and join it to the incoming rows, or cache it for the length of the statement. Neki's general model follows from this. Work out which parts of a query can't run on shards, plan for those parts to run on the router, and push everything else down.

Immutable, stable, volatile

During semantic analysis, the planning phase that works out what each part of a query means, Postgres labels every expression with one of three classes:

Why sequences are router-only

Sequences are created with CREATE SEQUENCE, with identity columns, or with DEFAULT nextval('seq'). In Neki the sequence objects exist on every shard but stay empty. Any query can reference any sequence, so letting shards count independently would produce duplicate values. Instead, all sequence operations run on the router. The router gets fresh values from a single authoritative shard, which physically hosts every sequence, and hands them out.

Table expressions decide "once per row"

• The inner join on id gives three rows. DOGE appears twice, once for each of its feedings.
• Adding WHERE feedings.time = 'morning' leaves one row, so nextval('tag_seq') runs once and uses up one value.
• Without the WHERE, there are three rows. A smart optimizer might notice that DOGE's age + 1 and lower(name) repeat and compute them only once. It must not do that for nextval(), which has to produce a fresh value on each of the three rows.

Planners may also reorder joins and filters, for example filtering feedings before joining. Neki and the shard's own Postgres planner may not pick the same order. Rydberg notes that if a volatile expression appears inside a WHERE or JOIN … ON condition, the results can vary from run to run, but they are still correct.

Distributed joins

Network round trips between router and shards are Neki's biggest cost. A cross-shard join normally means pulling both tables to the router and joining there. If both tables are sharded on the column being joined (here id), Neki can join locally on each shard and simply concatenate the results. That is correct because matching rows always live on the same shard.

With that in place, Neki handles SELECT age + 1, lower(name), nextval('tag_seq'), now() FROM animals JOIN feedings ON … like this: the join and the two immutable columns run on the shards. now() is computed once on the router. nextval() is computed on the router for each row as that row streams in. Values can't be fetched ahead of time, because the router doesn't know how many rows each shard will return.

The hard case: UPDATE

UPDATE animals SET tag_id = nextval('tag_seq') modifies every row, but the router doesn't know how many rows exist or how they're spread across shards. Neki handles it in two phases:

1. It first sends each shard a query that selects only the columns it needs, here id, which is both the primary key and the shard key, using FOR UPDATE to lock those rows.
2. As each id reaches the router, Neki calls nextval() and sends a parameterized single-row update to the right shard. The id picks the shard and the WHERE clause picks the row.

The post shows the exact queries as images. Above they're described in words rather than quoted.

Open questions

Rydberg says other designs are possible but doesn't compare them. The two-phase update costs one network round trip per row, and the post gives no performance numbers for it. Fetching sequence values in batches, a common workaround, isn't discussed. Sequence resets and cycling are explicitly out of scope. How a stable expression like now() would work in that UPDATE is left as an exercise. It's simpler, because it doesn't depend on the row count.

Netflix's compute platform team has described how it keeps a growing GPU fleet healthy for model training and live inference. These GPUs train and serve the recommendation models that build each member's home screen, along with newer generative-AI work. Netflix reports two results since the system landed. GPU nodes now stay above 99.95% available. The share of jobs that succeed rose from about 98% to above 99.5%.

Why ordinary health checks miss the costly failures

Netflix starts with the cloud provider's checks. These confirm that the virtual machine's host has power and network, that the operating system is reachable, and that attached storage works. They also announce planned maintenance. Most teams then add a GPU check that confirms the driver responds and the GPUs are visible. That catches a dead GPU, but Netflix says it misses the failures that eat the most time:

• Faults that appear under load. A GPU can answer a status query and still have a fault in its on-chip memory or firmware. The fault only shows up once a job allocates memory or runs sustained compute.
• Degraded networking. A high-speed network adapter can be present but show fewer links than expected. Collective operations still work, at a fraction of the bandwidth. The job doesn't crash. It crawls.
• Broken checks. The diagnostic tool can hang when the driver hangs, and a check that waits on it hangs with it. A driver upgrade can delete a binary a check relies on, so the check stops running and reports nothing.

The team's rule is that "the hardware can lie and the tools that inspect it can break."

Four layers

Netflix arranges reliability in four layers, each catching what the one below misses:

1. Node and hardware health: detect bad GPUs, adapters, interconnects, memory errors and overheating, then taint or recycle the machine.
2. Scheduler and platform: retry tasks and jobs, validate machines before they take work, and drain and recycle them safely.
3. Training loop: checkpoints and elastic workers, so a job survives losing a machine.
4. Observability: metrics, service-level objectives and attribution of each failure to a cause.

How detection works

Four design rules address the "broken checks" problem:

Graded, automatic repair

The repair matches the fault. Killing the job clears a software wedge. Resetting one GPU clears many transient faults. A reboot clears firmware state that has drifted over a long uptime. Replacement is kept for hardware that fails a trusted test. In that case Netflix records the GPU's permanent hardware ID, reports it to the cloud provider and keeps its own blocklist, so the same device can't come back under a new instance.

A new machine has to prove itself before it gets work. It passes system checks, confirms its GPUs and their layout, and runs GPU diagnostics. Then it waits for two health-check cycles, because slower checks such as the adapter count need more than one cycle to report. Only then is the startup taint removed. Netflix also found a race: machines were handed to the scheduler while their checks were still settling. It now holds capacity until the checks settle.

Surviving loss, and retrying only the right failures

Measuring it

What's missing and what's next

The results come down to two numbers. The post gives no availability figure from before the change, no repair times, and no failure rates per fault class.

Planned work includes:

GitHub has shipped an API endpoint that returns a repository's star history as daily star counts, up to 30 weeks per request, with no user identities attached. For react/react, a repository with about 251,000 stars, rebuilding the full history now takes about 24 requests instead of about 2,511. The more interesting part of the write-up by Taylor Cuilty is why the obvious fix, returning counts instead of people, would not have made the database's job any lighter.

The old way and its two problems

Maintainers look at star history to answer one practical question: did a launch, release or event change the project's trajectory? GitHub had no endpoint for this. Tools rebuilt the curve from GET /repos/{owner}/{repo}/stargazers, which lists the people who currently star a repository. With a custom media type, a request header that asks for an extended response format, each entry also included the time of the star. Tools fetched every page, kept the timestamps and threw away the usernames.

The data model added a quieter limit. Stars are stored as the current set of people starring a repository. They are not an event log, so an un-star deletes the record. Any history rebuilt from them only reflects people who still star the project.

Stop the bleeding, then ask what people need

This also broke tools such as Star History. GitHub talked to maintainers and tool builders and found they wanted the curve, not the names. Identity had looked like part of the contract only because every timestamp used to arrive attached to a user.

Two designs that didn't go far enough

Timestamps without identities. The first prototype kept the one-record-per-star endpoint but removed the user for callers without collaborator access. That closed the scraping path with a small change, and existing tools would keep working. But clients would still page through thousands of records, and a trends use case would stay tied to an endpoint built for listing people.

Aggregate, then paginate. The next idea was to group stars into daily or weekly buckets in SQL and page through the much smaller set of buckets. The problem is the order in which a database runs that query:

all matching stars → bucket every row → sort every bucket → OFFSET/LIMIT → response

LIMIT and OFFSET run last. Asking for page 3 of the buckets still makes the database read, group and sort every star in the repository, then throw most of the result away. The JSON response shrinks, but the work does not. For react/react, all 24 pages would each reprocess all ~251,000 stars, about 6 million rows read to get the history once.

The shipped design: filter by date first

indexed date range → group rows in that range → daily buckets → response

As an illustration (not GitHub's actual numbers), suppose page 1 covers the most recent 30 weeks. The index lets the database jump straight to stars created in that window, perhaps a few thousand, and count them by day. Page 2 covers the 30 weeks before that, and so on back through history. Each star is read about once over the whole history, about 251,000 rows for react/react instead of 6 million. A single request covers up to 30 weeks however popular the repository is. The response has the same shape as GitHub's existing weekly commit activity endpoint (GET /repos/{owner}/{repo}/stats/commit_activity).

Weekly paging also fits the way people use the data: they usually look at growth across weeks or around a known release date.

Side effects

What remains open

The design limits each request to a period of time, not to a number of stars. A week with a huge spike, such as a viral launch, can still produce a slow query. GitHub calls dense weeks a remaining scaling limit and does not say how it will handle them. Weeks follow calendar boundaries, which affects how data lines up. Because un-stars delete records, the curve shows today's stargazers by the date they starred, not a true record of past totals. GitHub says these details are part of the API contract.

The author's main lesson: when a query is expensive, look at the order of operations, not just the size of the output. Two more: start from what users are trying to do, and return only what that job needs, since any extra data is data someone else can misuse.