Cloudflare stops rebuilding the whole DNS zone · Morning Gist
Monday, November 21 story · 4 min
Cloudflare stops rebuilding the whole DNS zone
One story today: how Cloudflare swapped brute-force DNS rebuilds for targeted ones without giving up a byte-for-byte correctness guarantee.
Picked from 67 new articles across 34 sources · arrow keys for other days
Today in 30 seconds
30s left
Cloudflare halved DNS build delay and cut memory use 85% by replacing many full rebuilds with targeted ones. story 1 ↓
Some zones exceed 20 million records; a new planner picks the smallest rebuild it can prove matches a full one, byte for byte. story 1 ↓
The write-up is a useful template for incremental work, covering the proof rule, queueing effects and a production comparison. story 1 ↓
1/1Cloudflare· Alex Fattouche· Nov 2
Cloudflare halves DNS build delay, cuts memory 85% with targeted rebuilds
Cloudflare changed its DNS pipeline to rebuild only what a change can affect, not whole domains. Peak memory fell 85%, the worst delay spike shrank 12-fold, and median time to write changes roughly halved.
Why it matters · It shows how to replace brute-force recomputation with incremental work while keeping correctness checkable, including the proof rule, the queueing effects, and a production comparison.
Know this first
Context from the editor, not the article · 37 sec
1 · DNS zones and authoritative servers
A DNS zone holds a domain's records, such as A records mapping names to IP addresses, plus provider settings. Authoritative servers give the final answer for that domain. At Cloudflare they read precompiled data rather than the customer database, so every change must be converted and shipped.
2 · Kafka partitions and offsets
Kafka is a durable message log split into ordered partitions. Consumers record a committed offset per partition, meaning everything before it is finished. After a restart, reading resumes from there. So one slow message can hold back the offset for all later messages, which then get replayed.
3 · Recompute versus incremental updates
Systems can recompute desired state from scratch or apply only the change. Recomputing is slow but needs no knowledge of what changed, which makes it robust. Incremental updates are fast but must identify everything a change touches; miss something and the output silently drifts from the truth.
The propagation pipeline: APIs and services feed Kafka, the Aggregator combines events, and the Zone Builder writes to Quicksilver.1 / 4
Remember
Keep recomputation as the fallback: unknown or vague events still get full builds, and smaller work must be proven equivalent.
Cheaper builds created headroom to shorten safety waits; those shorter waits, not build speed itself, halved the median delay.
Counts can mislead: full builds fell about 40% by count, yet memory fell 85% because large zones stopped rebuilding.
Cloudflare's DNS team has changed how its 1, the service that compiles a domain's records and settings for its servers, handles changes. Zone setting changes no longer rebuild zones with tens of millions of records. Alex Fattouche reports peak memory down 85% and median delay down from about 7.7 to about 3.3 seconds.
The problem
A DNS change produces an event on 2, a durable message log. The builder converts zone data into keys for Cloudflare's 3, which give the final answer for a domain. It compares those keys with 4, Cloudflare's distributed key-value store, and writes only the differences.
A full build still reads and compares every record in memory, even when one key changes. Some zones exceed 20 million records. Record changes got a shortcut in 2022, but settings did not, because a zone event may not say what changed and can affect data shared by several zones. Doing too little leaves wrong DNS data at the edge.
The mechanism
Events now carry changed fields where available. A planner picks one of five builds: specific names, names plus shared metadata, apex (the bare domain) plus metadata, purge, or full zone. Every key a targeted build writes must match a full build byte for byte. Anything uncertain gets a full build.
Because events describe the past, the builder rereads current state just before executing. That can upgrade, purge or skip the build. A single scheduler now merges queued work per zone, and merging only ever adds work.
This also helps Kafka. Its 5, the point before which all messages in a partition are finished, cannot pass an unfinished build, so one long build holds back everything behind it. Shorter builds reduce that exposure.
To check correctness, a mirror Zone Builder consumes production events but writes to a separate Quicksilver environment serving no customers. A comparer samples 20% of zones for differences.
The results
92.95% of builds now avoid the complete zone. The remaining full builds average 0.35 seconds, since they mostly affect new, small zones.
Average memory fell from 49 GiB to 7 GiB, and peaks from 79 GiB to 12 GiB.
The largest 6 delay spike, the time within which 99% of events finished, fell from 14.6 to 1.21 minutes.
Typical delay barely changed at first. The median halved only after spare capacity let the team shorten two safety waits. These figures exclude distribution across the network.
The caveat
Above a fixed record count, the builder never escalates to a full build, because it would not fit in memory and would block its partition. It rebuilds the apex, metadata and queued names instead, with manual rebuilds for rare cases. On those zones, feasibility beats the byte-for-byte guarantee.
Cloudflare's DNS team has changed how it turns a customer's DNS change into data its servers can use. Its 1, the service that compiles a domain's records and settings for the edge, no longer rebuilds huge zones when a setting changes. Alex Fattouche reports peak memory down 85%, the worst 2 delay spike (the time within which 99% of events finished) down from 14.6 to 1.21 minutes, and median delay down from about 7.7 to about 3.3 seconds.
Work that grows with zone size
A change lands in a database and produces an event on 3, a durable message log. A service called the DNS Kafka Aggregator combines events from several topics and passes them to the Zone Builder. The builder reads the zone data and converts it into keys for Cloudflare's 4, which give the final answer for a domain. It then compares those keys with 5, Cloudflare's distributed key-value store, and writes only the differences.
Record changes have grown from about 250 per second in 2022 to about 700, plus about 100 other events. Fattouche says event count was never the real problem. A full build reads, encodes and compares every record while holding the zone in memory, even if only one key ends up changing. Some zones now exceed 20 million records, and Cloudflare expects hundreds of millions. Work in 2022 gave record changes a shortcut, 150 times faster on average, but zone settings still forced full builds.
Why settings are harder than records
A record event names what changed. A zone event can come from many products, may not say why, and can alter data shared by several zones. 6, Cloudflare's automatic certificate feature, shows the difficulty. Toggling it updates metadata and can change 7 records, which tell certificate authorities whether they may issue for a domain, at the apex (the bare domain, such as example.com).
A targeted build must therefore know which names can change, whether shared metadata or the list of published zones can change, and what old state to remove. Doing too much wastes capacity. Doing too little leaves stale or wrong DNS data at the edge.
Choosing the smallest provable build
The team changed event producers to include changed fields, with old and new values where available, then traced each setting through the code that generates DNS output. Some database functions only bump a timestamp, which may be harmless or may mean data changed elsewhere. Those events, like unknown ones, still get a full build.
The planner chooses among five build types:
Resources only: rebuild specific DNS names, as for ordinary A, AAAA, TXT and CNAME changes.
Resources and metadata: the same plus shared metadata, for nameserver and wildcard changes.
Apex and metadata: for settings such as Universal SSL.
Purge: remove a zone that should no longer be published.
Full zone: when nothing smaller can be proven safe.
The rule is that every key a targeted build writes must be byte-for-byte identical to what a full build would write.
One name, several zones
During onboarding and other lifecycle changes, several internal zone objects can briefly exist for one public domain. One shared metadata key lists which zone IDs the servers may use. Suppose example.com has zones 101, 202, 303 and 404, and the key lists the first three. A TXT change in zone 202 touches only that zone's record keys. A wildcard change in zone 202 must also update the metadata key. The builder rereads the current list, rebuilds the whole key rather than patching it, and writes the zone's records before the metadata that refers to them.
Planning twice
An event describes the past, and a build may wait in a queue, so the event produces only a conservative work list. Just before execution, the builder reads a fresh snapshot: current settings, whether this deployment should publish the zone, and which same-name zones belong in the key. That can upgrade a targeted build to a full one, turn it into a purge, skip it, or reconstruct the metadata key.
The read comes from 8, read-only database copies, and Kafka can deliver an event before its change reaches one. The aggregator delays events to cover normal lag, and a final check rejects reads from a replica that is still behind.
One scheduler, and Kafka
Record work and full-zone work used to have separate schedulers that could not combine with each other. One scheduler now merges queued work per zone. An A record change at www.example.com and a wildcard change at *.example.com become one build covering both names and the metadata. Merging only adds work: any full requirement makes the build full, and a purge overrides everything. DNS work and pending Kafka messages are tracked separately, so messages stay attached until the result is written.
Kafka stores one 9 per partition, the point before which everything is known to be complete. Cloudflare uses 10: builds may finish out of order and be marked done in memory, but the offset moves only when all earlier messages finish. If message 2 is a long build, messages 3 to n cannot commit, and a restart replays 2 to n. Targeted plans shorten that exposure. A per-zone rebuild interval stops one busy zone from taking every worker, though old queued work can run early before it blocks a partition for too long.
Proving equivalence
Equivalence tests run full and targeted builds from the same database state and compare keys byte for byte. During rollout, a full build also ran beside every targeted build until the mirror comparison gave the same confidence. That mirror is a second Zone Builder that consumes production events and reads production data but writes to a Quicksilver environment serving no customers. A comparer samples 20% of zones. One dashboard capture shows a 0.31% diff rate across 234,313 zones; most differences are timing, so the team watches for new kinds of difference. Rollout hashed the zone name, not its ID, so two schedulers never wrote the same metadata key.
Results
Over 24 hours, 92.95% of builds avoided the complete zone: resources only 86.81%, resources and metadata 2.54%, apex and metadata 2.30%, purges 1.30%. Full builds were 7.04% and now average 0.35 seconds, since they mostly affect new, small zones. The two metadata types used to be full builds, so full builds fell by only about 40% by count.
Memory shows the real effect, because the avoided builds include the largest zones. Average 11, the memory actively in use, fell from 49 GiB to 7 GiB, and peaks from 79 GiB to 12 GiB. Kafka consumption now tracks arrivals of about 600 to 800 messages per second, instead of falling behind and catching up in bursts of up to 1,550.
The delay figures need careful reading. They run from the change to the Quicksilver write and exclude distribution across the network. Comparing production with the mirror over the same 24 hours, typical delay barely moved; the gain was in the tail. The median halving came later, when the spare capacity let the team shorten the rebuild interval on most plans and the aggregator's hold time. Early percentile readings were inflated by coarse histogram buckets. A 12, which picks fine-grained buckets automatically, puts p90 at about 5.5 seconds and p99 at about 12.
What remains open
Cloudflare dropped its 2022 goal of eliminating full builds, because first publication and zone-wide changes genuinely need them. It also made one deliberate exception to its safety rule. Above a fixed record count, the builder never escalates to a full build, since that build cannot fit in memory and its retries would block the partition. It rebuilds the apex, metadata and queued names instead. Zone holds, which stop a name being added to another account, reduce the risk, and rare cases get manual rebuilds. So on the largest zones, the byte-for-byte guarantee gives way to feasibility. The article gives neither the threshold nor any way of detecting drift there.
