Replaying Traffic and Rolling the Root Key · Morning Gist
Wednesday, October 75 stories · 11 min
Replaying Traffic and Rolling the Root Key
Infrastructure teams test with real production load, rotate trust anchors and pull apart coupled designs, while a DNS root key change on Saturday tests every validating resolver.
Picked from 19 new articles across 34 sources · arrow keys for other days
Today in 30 seconds
30s left
Airbnb captures production MySQL traffic at ProxySQL and replays it in timed batches, catching regressions and nondeterministic queries before its 8.0 upgrade. story 1 ↓
Meta's nts.meta.com derives its NTS cookie key each day from a master secret, so servers share no key ring and keep no sessions. story 2 ↓
On October 11 the DNS root replaces KSK-2017 with KSK-2024 (tag 38696). Resolvers that don't trust the new key will fail all DNSSEC validation. story 3 ↓
GitHub is replacing its five-replica Spokes quorum. Durable data moves to Azure Blob Storage and reads come from cache-backed workers, so scaling reads no longer slows writes. story 4 ↓
Orosz's 2026 snapshot, mostly about AI: code review is struggling, reliability is down, CPUs are scarce and large migrations now run very fast. story 5 ↓
1/5Airbnb Engineering· Zuofei Wang· Oct 6
Airbnb replays real production MySQL traffic captured at ProxySQL
Captured traffic is replayed in timed five-minute batches. It caught latency regressions and nondeterministic queries before Airbnb's MySQL 5.7-to-8.0 upgrade.
Why it matters · Capacity planning and upgrade testing usually rely on synthetic benchmarks like sysbench. This is a worked design for capturing real traffic and replaying it with correct transaction order, timing and auto-increment IDs, along with the bugs it found.
The old setup: separate Java, Python and Ruby query loggers sending to Kafka, replaced by proxy-side capture.1 / 4
Airbnb runs hundreds of MySQL-compatible clusters that serve millions of queries per second. Its earlier capture system logged queries on the client side, separately for Java, Python and Ruby, and sent them to Kafka. It was costly to maintain, no team clearly owned it, and it lacked the transaction context needed to replay queries faithfully or check that results matched. The new system captures queries at ProxySQL, a proxy that already sits in front of every database, so no application has to change.
The pipeline
Capture. ProxySQL's built-in query logging is turned on per cluster through query rules, and only for as long as a test needs. It writes binary logs to local disk. A Log Mover sidecar ships those files to object storage.
Processing. ProxySQL runs as a pool of stateless Kubernetes pods, and each pod serves several backend clusters. An offline Log Processor decodes the logs, splits them by cluster, puts each transaction's statements back in their original order, and cuts the output into per cluster and pod. It also rewrites INSERTs to set the recorded in production explicitly. For example, becomes . This stops later reads that depend on that ID from failing on engines that assign auto-increment values differently. The cost is that replays no longer test the target's own auto-increment behavior.
2/5Engineering at Meta· Oct 6
Meta adds stateless NTS to its public time service; key derived daily
At nts.meta.com, servers re-derive the cookie-sealing key from a master secret and the day number, so they replicate no keys and keep no sessions.
Why it matters · The CA/Browser Forum is shrinking certificate lifetimes to 47 days, so certificate checks increasingly depend on a correct clock. Meta's derived-key design also shows how to make a crypto service stateless with no replicated key ring to keep in sync.
Image: Engineering at Meta
Meta's public time service now supports NTS (Network Time Security, RFC 8915) at nts.meta.com. The server, client and protocol code are open source in github.com/facebook/time. Plain NTP has been unauthenticated since 1985. A client sends 48 bytes, gets 48 bytes back and believes whatever arrives. That clock is what certificate notBefore/notAfter checks, token expiry and replay windows all rely on. The article says this gets more urgent as CA/Browser Forum ballot SC-081v3 cuts TLS certificate lifetimes from 200 days today to 47 days by March 2029. Renewal then runs roughly monthly and unattended over ACME, so a host with a skewed clock fails or reissues without anyone noticing.
How the protocol splits the work
Key establishment (NTS-KE) runs once over TLS 1.3 on TCP/4460. Client and server pick an AEAD: AES-SIV-CMAC-256 (mandatory to implement), AES-SIV-CMAC-512 or AES-128-GCM-SIV, and the client's preference wins. Two directional session keys come from the RFC 5705 TLS exporter. The server hands over eight cookies, names the NTP server the client should use, and closes the connection.
Authenticated NTP then runs over UDP/123. Each request carries a unique ID, a cookie and an authenticator. The server's reply includes fresh cookies, encrypted, so an observer can't follow a client from one network to another. Forged packets are dropped silently, with no NAK, so an attacker can't use failures to force the client into a new key exchange. Replayed responses carry the wrong ID and match no outstanding request.
Cookies without state
A cookie holds the session keys, sealed with AES-SIV. Rather than replicating a key ring, each server computes . Here counts whole 24-hour periods since the Unix epoch, so time zones and DST never come into it. The day number travels in the clear as the key ID, and servers accept keys from two days back to one day forward to cover clock skew at rotation. The KE host and the NTP responders never talk to each other, and adding capacity means adding a server. Plain NTP traffic runs through the same code at the same cost as before.
3/5Cloudflare· Sebastiaan Neuteboom· Oct 6
DNS root swaps its key-signing key on October 11, for only the second time
KSK-2024 (tag 38696) replaces KSK-2017. Cloudflare's readiness test uses the RFC 8509 sentinel, which lets you ask a resolver whether it trusts the new key.
Why it matters · If a validating resolver doesn't trust the new root key, it can't validate any DNSSEC-signed name under any TLD, so healthy sites go dark for its users. In 2018, resolvers lost learned trust anchors during routine upgrades and machine moves.
Chain of trust from the root through .com to cloudflare.com, starting at the resolver's trust anchor.1 / 3
On October 11, 2026, the DNS root will start signing its DNSKEY set with a new key-signing key, KSK-2024 (key tag 38696), in place of KSK-2017 (tag 20326). It's only the second root KSK rollover ever. Most site operators don't need to do anything. Anyone running a DNSSEC-validating resolver must make sure it already trusts the new key.
Why the root key is special
DNSSEC trust passes downward. Each parent zone publishes a Delegation Signer (DS) record holding a fingerprint of its child's key, so the chain runs from the root to .com to cloudflare.com. The root has no parent, so resolvers start from a preconfigured trust anchor. The root uses two keys. The KSK signs the DNSKEY set. The zone-signing key (ZSK) in that set signs everything else, including the TLD DS records. If a resolver lacks the new KSK, validation fails for every TLD at once.
RFC 5011 lets resolvers learn new anchors automatically. The root publishes the new KSK next to the old one, and the old KSK signs the set. A resolver has to keep seeing the new key in validly signed records for at least 30 days, then re-verify, before it accepts the key. KSK-2024 has been in the root DNSKEY set since January 11, 2025.
4/5GitHub Engineering· Brian Celenza· Oct 6
GitHub is moving Git off its five-replica quorum design to separate durability from scale
GitHub's Spokes design ties read capacity to write latency. The replacement puts authoritative data in Azure Blob Storage and serves reads from cache-filled compute workers.
Why it matters · It's a clear case of a common architectural ceiling: the replicas that make data durable also serve reads, so adding read capacity slows every write. Here is how a large Git host plans to remove that coupling.
Image: GitHub Engineering
GitHub is rebuilding its Git storage layer. The trigger is write volume. Git events on GitHub more than doubled in a year, from 218.2 billion to 473.3 billion a month. Pushes grew 4.9x to 3.35 billion a month, and commits hit 7.38 billion in September. The busiest single repository handled about a billion requests in August. Brian Celenza says much of the growth comes from automated workloads: CI, merge queues and fleets of agents committing on many branches of one repository at once.
Where Spokes hits its limit
Today every repository lives in Spokes, which keeps full copies on the local disks of several fileservers, five by default. A reference update runs a three-phase commit with a quorum, so CI, the web UI and the API all see a consistent state. Local disks keep reads fast, and the extra copies spread read load.
The flaw is that one mechanism handles both durability and scale. Each read replica is also a durable replica that takes part in every write. A push therefore completes only as fast as the slowest replica, and adding replicas to absorb reads makes writes slower. At the top of the activity curve, losing a replica cuts read capacity, and losing quorum stops writes entirely.
The pressure shows up in a few places:
Merge queues and trunk-based development push every merge through one hot ref. PR merges are up nearly 4x.
Each push fans out into thousands of fetches. Actions ran 3.26 billion times in September.
Compaction and GC run on the same hosts that serve live traffic, and their cost grows with write volume.
5/5The Pragmatic Engineer· Gergely Orosz· Oct 6
Orosz's 2026 industry snapshot: code review is broken, CPUs are scarce
His LDX3 keynote draws on data from GitHub, Linear and Uber, and covers what agent-written code is doing to review, reliability, migrations and team size.
Why it matters · Heads-up: this piece is mainly about AI, which your brief excludes. It's here for the parts that bear on engineering practice: review processes that no longer work, falling reliability, a CPU capacity squeeze and very fast large migrations.
Gergely Orosz's keynote at LDX3, an engineering leadership conference, lists what changed in the past year, what broke and what stayed the same. It is built on interviews at OpenAI, Anthropic, Uber and Ramp, plus unpublished data from GitHub, Linear and Factory AI. Most of it covers coding agents. The parts below are the ones that matter for how teams build and run software.
What broke
Volume assumptions. GitHub data shows lines of code and commit counts growing exponentially, not linearly. Agent-authored PRs grew ninefold in eight months, and in August 2026 they outnumbered human-authored PRs on GitHub. Orosz says this load helps explain GitHub's ongoing reliability problems.
Code review. An engineer at a mid-sized startup told him reviewers facing 5–10× more changes "just stamp everything with LGTM." The company is now formally phasing out parts of review. Linear data shows PRs reviewed only by agents are rising. Orosz calls most review today "theater."
Reliability. Mario Zechner says 98% uptime is becoming the norm, even for large services. Orosz links worse quality to heavier agent use.
Replay. An API server breaks each job into tasks, one per log file. A scheduler gives every task in a batch the same future "expected start time". Workers, which can scale to thousands, wait for that moment and then replay together, spacing queries by their original timestamps times a speed factor. This reproduces the original concurrency, not just the query mix. The scheduler stops sending new tasks if tasks start failing. There are two modes: replay only, at 1x, 2x, 3x or a target QPS, tracking errors and per-digest latency histograms; and replay and compare, which runs the same queries against two targets and diffs the results.
What it found
During the 5.7→8.0 upgrade, the team first replayed against 5.7 with and without the query cache, which 8.0 removes, and then compared cache-less 5.7 with 8.0. Separating the two effects this way exposed duplicate cacheable queries from an application that the cache had been quietly absorbing. The worst regression was one query pattern that went from 0.03 s to 2.6 s and read 273 MB per join instead of 5 MB. The cause was a change in how MySQL 8.0.20+ reads rows while sorting.
Replay-and-compare runs against snapshots of the same data on both versions turned up queries with no ORDER BY, or with an ORDER BY lacking a unique tiebreaker. Combined with LIMIT, these return different rows on the new engine. Owning teams added explicit ordering.
For capacity planning, replaying 80% more write traffic on one large cluster raised average commit latency from about 6 ms to 34 ms, which showed where its ceiling was. Airbnb says the fleet finished the 5.7→8.0 upgrade without a major production incident.
Remember
Capturing queries at the proxy needs no application changes and avoids keeping a separate client logger for each language.
Pinning last_insert_id and giving batches a shared start time keep replays faithful to production.
Replay found a 0.03→2.6 s sort regression in MySQL 8.0.20+ and LIMIT queries with no defined row order.
On the client, one chrony line does it: pool nts.meta.com nts iburst maxsources 5. This brings up five independent authenticated sources (time1–5.meta.com), each with its own session keys, over about 20 minutes. Each cookie is 68 bytes.
Limits
Meta lists what NTS does not solve:
An on-path attacker can still delay packets and shift the clock by up to half the added delay.
NTS-KE needs TLS, and TLS needs a roughly correct clock, so a cold device still has to bootstrap.
A correctly authenticated server can still serve the wrong time.
Wall clocks still step backward. Cloudflare's 2016 leap-second outage came from exactly that, so code that measures elapsed time should use monotonic clocks.
Meta argues the real gap is mobile. Android's platform time sync is plain SNTP, and Meta says it knows of no NTS support in Apple's timed/time.apple.com. Between them those two stacks set the clocks on billions of devices.
Remember
Cookie-sealing keys come from HKDF over a master secret and the epoch-day number, so no keys are replicated and no sessions are stored.
NTS stops forged time responses but not delay attacks, bootstrapping problems or wrong-but-authenticated servers. Multiple sources are still required.
Android and iOS still set the time over unauthenticated packets. Meta says mobile is the main gap in NTS adoption.
During the 2018 rollover, Cloudflare saw resolvers lose this learned state when their software was upgraded or moved to another machine. This time it added KSK-2024 to its resolver's built-in anchors in July 2024, so the key is there from startup. 1.1.1.1 and Gateway DNS users don't need to do anything.
Asking the resolver directly
In 2018, users had no way to check this. Now 1.1.1.1 supports RFC 8509 trust anchor sentinels. These are ordinary queries for specially labelled names under a signed domain (Cloudflare uses dnstest.dev). A supporting resolver validates the records first. It then answers root-key-sentinel-is-ta-38696 normally if it trusts the key, and returns SERVFAIL if it doesn't. not-ta-38696 works the other way round. The test at dnstest.dev/ksk-2024 adds three controls: a normal signed name, a deliberately broken one, and a sentinel query for the current key. The controls separate a real result from a resolver that doesn't support sentinels. If support can't be established, the result is "inconclusive", not "missing".
Same algorithm, more rollovers ahead
Both keys use RSA/SHA-256. The rollover still limits how long one private key stays in use, and it rehearses distributing anchors, a process that can fail even when the cryptography is fine. IANA's target interval is three years. ICANN blames the eight-year gap on pandemic disruption and upgrades to the hardware that protects the signing keys. KSK-2017 will be revoked and its private key deleted in 2027. A separate move to ECDSA P-256 has been proposed, and it isn't post-quantum. 1.1.1.1 already validates ML-DSA-44 signatures, but a post-quantum chain of trust will need yet another root rollover. Cloudflare argues that sentinel support should become standard so these rollovers can be checked before they happen.
Remember
KSK-2024 (tag 38696) starts signing the root DNSKEY set on Oct 11; validating resolvers must already trust it.
RFC 8509 sentinel queries (is-ta / not-ta) show whether a resolver trusts a specific root key.
The algorithm is still RSA; ECDSA P-256 and later post-quantum keys will each need more root rollovers.
Minimize coordination. Only the reference update needs agreement. Storing objects, checking connectivity and secret scanning are the expensive steps, but they can run in parallel with other writes and off the critical path. A push can be acknowledged once the small coordinated step finishes. Maintenance moves to separate workers that operate directly on durable storage.
Decouple storage from compute. Authoritative repository data moves to Azure Blob Storage, which handles durability and replication. Reads go to lightweight compute workers that cache data. Losing a worker then behaves like a cache miss rather than a full repository rebuild: a replacement starts serving immediately and fills its cache from storage. Workers can also scale up for a burst, such as a release, and scale back down afterward.
GitHub reports up to 35x higher write throughput in internal benchmarks. It gives no latency figures, methodology or consistency details. It also doesn't say how ref-update agreement works without the replica quorum, or what object-store latency does to cache misses. Those details are promised for a follow-up post. The migration will happen without a maintenance window, and branch protections, required reviews and audit logs stay unchanged.
Remember
In Spokes, every read replica also takes part in every write, so adding read capacity slows pushes.
In the new design, only the ref update is coordinated; object storage, connectivity checks and scanning run off the critical path.
Authoritative data moves to Azure Blob Storage with cache-backed compute workers. GitHub claims up to 35x write throughput but hasn't given methodology.
Capacity. Besides the GPU and memory shortages, CPUs are now scarce. Orosz says spot CPU capacity is nearly impossible to get without an established relationship with the cloud provider. Some regions reportedly turn away new tenants, and customers are paying now for capacity that comes online in December. His advice: if you rely on spot instances for dynamic workloads, lock in fixed capacity now, even at a higher price.
What changed, and what didn't
Large migrations have dropped from years to weeks:
Uber moved 600,000 JUnit 4 tests across 15M lines of code to JUnit 5 in 4 months.
Asana migrated 4,000 test files from Enzyme to React Testing Library in 2 weeks, against a five-year estimate.
Airbnb did a similar migration of 3,500 files in 6 weeks.
Bun was ported from Zig to Rust in 11 days, against an estimated 1.5 engineer-years.
Uber kept AI spending flat since May, even as usage grew, by switching to open models and routing requests to the cheapest suitable model. That cut per-token costs by 50% or more.
Individual projects at Anthropic are capped at about two engineers, because each person's agents already conflict with one another. But Katelyn Lesse says teams still own software and on-call as two-pizza teams. Bun creator Jarred Sumner says testing still takes about as long as writing the code. Orosz found no company where PMs or designers ship to production.
Charity Majors frames the open question: what would it take to ship code you haven't read? She notes that Ops and QA have done that for decades. Orosz predicts CI/CD will be rebuilt with automated agent checks built into the pipeline, and that observability for agent-built systems will become its own specialty.
Remember
Agent-authored PRs passed human-authored ones on GitHub in August 2026, and human review has mostly become rubber-stamping.
CPU capacity is tight: spot capacity is hard to get, reservations are booked months ahead, and some regions refuse new tenants.
Large test migrations now take weeks, such as Uber moving 600K JUnit tests in 4 months, yet two-pizza teams persist.
