Today's stories cover isolating noisy workloads, keeping storage metadata with its data, testing programs that never answer the same way twice, and compliance that changes products.
Picked from 65 new articles across 34 sources · arrow keys for other days
Today in 30 seconds
30s left
Netflix's Titus now puts allowlisted noisy or sensitive containers alone on right-sized EC2 VMs, and the shared fleet packs more densely as a result. story 1 ↓
The price of that isolation: slower launches, warm pools that no longer help, and fixed overhead on every node. story 1 ↓
Fly.io replaced its modified JuiceFS with SBD, a plain block device on object storage, after 5,000 test cycles left some Sprites unable to checkpoint. story 2 ↓
Hillel Wayne explains why comparing outputs fails for nondeterministic programs: even matching output sets can hide different pairings and frequencies. story 3 ↓
Cloudflare passed South Korea's 80-control security audit by turning off Google, Apple and GitHub sign-up there, because those flows skipped the required separate consents. story 4 ↓
1/4Netflix Tech Blog· Netflix Technology Blog· Oct 22
Netflix gives noisy, sensitive containers their own VM inside Titus
Netflix added a mode to Titus, its internal container platform, that runs one task per right-sized EC2 virtual machine. The aim is to stop interference between neighbours that cgroups can't fence, without application teams taking on host management.
Why it matters · Shows a concrete case where dedicating hosts improved fleet efficiency, and spells out the costs: launch latency, warm pools that stop working, and fixed per-node overhead.
Know this first
Context from the editor, not the article · 37 sec
1 · What Titus is
Titus is Netflix's internal container platform, built on Kubernetes. It runs millions of containers a week across tens of thousands of AWS EC2 instances. A central Compute/Platform team owns and patches the hosts. By default many teams' containers share each host and its single Linux kernel.
2 · What cgroups can't fence
Linux cgroups cap each container's CPU time, memory and disk use. Some resources are shared in hardware, though, and no cgroup covers them: the last-level CPU cache, memory bandwidth, and the kernel's network-processing time. One container can starve another of these without ever exceeding its own quota.
3 · Noisy-neighbour background
2/4Fly.io· Oct 22
Fly.io swaps its modified JuiceFS for a plain block device on object storage
Fly.io replaced the heavily modified JuiceFS setup behind its Sprites sandboxes with SBD, a daemon that exposes an object-storage bucket to Linux as an ordinary disk. That removed three separate records that could drift out of agreement, and Fly says reliability improved by orders of magnitude.
Why it matters · It shows a familiar storage failure: metadata kept apart from the data it describes, each made durable on its own schedule. Merging them under one source of truth fixed it.
Know this first
Context from the editor, not the article · 33 sec
1 · What Sprites are
Sprites are Fly.io's persistent Linux sandboxes, marketed as computers for coding agents. A Sprite can go to sleep, then wake on a different physical server, so its disk can't live on any one machine's local drive. It also needs fast checkpoints and restores.
2 · Object storage can't edit
Object stores such as S3 store whole blobs. To change one byte you have to rewrite the entire object. Linux programs constantly rewrite small ranges in the middle of files, so some layer has to turn small in-place writes into whole-object uploads.
3 · Block devices and filesystems
3/4Hillel Wayne· Oct 22
Hillel Wayne: why differential testing breaks for nondeterministic programs
Hillel Wayne looks at how to check that a rewritten program behaves like the original when both are nondeterministic. One mismatch no longer proves a difference, and "same behavior" gets harder to define.
Why it matters · Rewrites, ports and migrations are usually checked by comparing outputs, and real services have timeouts and races that make that comparison unreliable.
Know this first
Context from the editor, not the article · 34 sec
1 · Differential testing
You check a new implementation against a trusted one by feeding both the same inputs and comparing their outputs. Any mismatch is a bug in one of them. Teams use it for ports, compiler testing and database migrations, where a reference version already exists.
2 · Nondeterminism
A program is nondeterministic if the same input can give different outputs on different runs. Network timeouts, thread scheduling, clocks and random numbers all cause it. A web handler that usually returns 200 OK but sometimes times out with 408 is a typical case.
3 · Hyperproperties
Most correctness properties can be judged on a single run: "this run never crashed." Hyperproperties can only be judged across several runs together, such as "two runs with the same input always agree." They are much harder to test and verify.
4/4Cloudflare· Kabin Calan· Oct 22
Cloudflare passes South Korea's security audit by disabling social sign-up there
Cloudflare earned South Korea's ISMS security certification. To meet Korean consent rules, it rebuilt sign-up for Korean users around four separate consent checkboxes and removed social login, which skipped the consent form.
Why it matters · It shows a legal requirement turning into product changes: identity-provider login skipped the consent step, so it had to go, and the code needs IP location and user-declared country as two separate signals.
Know this first
Context from the editor, not the article · 37 sec
1 · Compliance certifications
Companies prove their security practices through independent audits. ISO/IEC 27001 is a global certification for an information security management system. SOC 2 is a US attestation report. Both typically list the products or systems they cover, and buyers check that list during vendor risk reviews.
2 · Korea's privacy regime
South Korea's Personal Information Protection Act (PIPA) is one of the stricter privacy laws. Korean practice expects consent to be itemized: users agree separately to terms, to data collection, to overseas transfer and to third-party sharing, with each item marked required or optional. A single combined checkbox doesn't meet that expectation.
Netflix has written before about narrowing this gap. One approach re-pins containers to CPU cores to cut cache interference. Another uses eBPF to measure each container's wait time in the scheduler queue. A third traced slow container startup to a global kernel lock. Each helped, but none removed interference completely.
Two containers within their CPU limits still contend for shared cache and memory bandwidth that no cgroup covers.1 / 4
What Netflix built
Titus is Netflix's Kubernetes-based internal container platform, and it packs many teams' containers onto shared hosts. It now has a single-tenant mode: an eligible task runs alone on its own right-sized EC2 virtual machine. Teams keep the same API and deployment path, the platform team still patches the host, and teams switch in or out with a labelling change.
Why sharing a kernel wasn't enough
Linux 1 cap each container's CPU, memory and disk. They can't divide resources that are shared in hardware, such as the last-level CPU cache and memory bandwidth. They also can't divide the kernel's own network work. Network shaping uses 2 (hierarchical token bucket), which serialises traffic through a global lock. One container sending floods of small packets can starve its neighbours and cause retransmits and tail-latency spikes, without any container exceeding its quota.
The worst-hit victims were stateful streaming jobs like Apache Flink. A hiccup there can force a checkpoint restore and minutes of catch-up.
Netflix rejected the alternatives:
3 (Firecracker, Kata): they need bare-metal or nested virtualization, a self-run hypervisor, and they lack GPU support.
4, a user-space kernel: it adds overhead on I/O paths.
Plain EC2: teams would have to own their hosts.
The authors argue those sandboxes solve a security problem Netflix doesn't have. What these workloads need is performance isolation.
How it works
The 5, which decides where jobs run, checks a platform-controlled allowlist and records the instance families allowed for the task. The autoscaler then launches the smallest instance that fits. Anti-affinity rules, a node 6, an exemption from preemption and disabled repacking keep the node to one task for its whole life.
Costs
Each task now waits tens of seconds for its own instance to boot, then pulls its image. Warm pools help less here: best-fit sizing creates many instance shapes, and a warm instance only helps a task of matching shape. Netflix keeps a small, curated warm pool. It also cuts launch time directly, with lazy image pulls and by disabling Kubernetes preemption logic that was leaving new nodes idle.
Fixed host overhead can eat tens of percent of a small node, so the mode suits large, long-running jobs.
Results
Netflix describes a benchmark that runs a reference app next to a deliberately abusive container and scores how much throughput and P99 latency it keeps. It publishes no scores. Instead it reports:
the Flink and pipeline teams see stable operation;
the shared fleet packs more densely, because sensitive jobs no longer need to be spread out;
patching is simpler, because draining a node moves only one task.
The news
Netflix has added a single-tenant mode to Titus, its Kubernetes-based internal container platform. An eligible task gets its own EC2 virtual machine, sized to fit, with nothing else scheduled on it. Everything else stays the same. Teams use the same Titus API and the same deployment pipeline, and the Compute/Platform team still owns and patches the host. Teams move a job in or out with a labelling change, so the decision can be reversed. Moving from Titus to plain EC2, by contrast, has historically been a one-way door.
Why shared-kernel isolation runs out
Titus packs many teams' containers onto large bare-metal hosts that share one Linux kernel. It isolates them with the standard tools:
1 cap CPU, memory and disk.
namespaces give each container its own view of the system.
seccomp and AppArmor restrict which system calls a container can make.
The iocost controller meters disk I/O, and traffic shaping meters network use.
These tools only partition what the kernel can partition. Cores share a 2 and a path to memory. A container streaming lots of data from RAM evicts a neighbour's cache lines and slows it down, even though both stay inside their CPU limits. No cgroup exists for cache residency or memory bandwidth.
The authors list a whole family of these shared resources:
CPU: run-queue contention, cache pollution and memory bandwidth.
Network: packets are processed in 3 context, kernel interrupt work that gets charged to whichever CPU happens to run it, often one a neighbour is using. A flood of small packets can exhaust that budget well before any bandwidth cap applies.
Storage: queue saturation and shared per-instance throughput ceilings.
A long tail: NUMA traffic, interrupts that flush other cores' address-translation caches (TLB shootdowns), power throttling, and global kernel locks.
The network has hurt most. Per-container shaping uses 4 (hierarchical token bucket), a Linux queueing discipline that serialises traffic through a global lock. When one container sends a flood of small packets, contention on that lock burns kernel CPU and starves neighbours, causing TCP retransmits and tail-latency spikes. In Netflix's words, "the mechanism meant to prevent noisy neighbors becomes one."
The victims that matter most are stateful streaming jobs, such as Apache Flink stream processors and data pipelines. If one falls behind, it may have to restore from a checkpoint and spend many minutes catching up, and someone gets paged. These teams report they never saw this on plain EC2.
Options weighed
Keep tuning shared-kernel isolation. This helps everyone, but the authors call it structurally limited, because the hardware stays shared.
5 (Firecracker, Kata Containers). These are lightweight VMs that boot in about 150 ms. They need bare-metal instances or nested virtualization, and nested virtualization wasn't ready on Netflix's instance types. There is no managed service, so Netflix would have to run the hypervisor itself. They also add overhead and don't support GPUs.
The authors argue that microVMs and gVisor solve a different problem: security between tenants who distrust each other. Netflix's teams need protection of performance and resources, not protection from a hostile co-tenant, and a dedicated VM provides exactly that with no emulation layer.
How it works
Placement logic lives in Titus's 7, which picks a cell and capacity for a job before it is scheduled onto a node.
Eligibility. The platform team keeps an allowlist. A team opts in by targeting a single-tenant workload group; there is no public API flag. Federation tags the task and records the instance families it may run on.
Provisioning. The autoscaler launches the smallest instance in those families that fits the task's requested CPU, memory and GPU, just in time.
Exclusivity. Pods get anti-affinity rules against all other pods and a priority class that exempts them from preemption. Nodes carry a 8, which only single-tenant pods tolerate. Defragmentation and repacking are turned off, so the task is never moved just to tidy the fleet.
Nodes vacated by a task are first offered to pending tasks before being terminated. New host images (AMIs) roll out either in place ("mutable") or only to new nodes ("immutable"), so many upgrades don't disrupt anything.
The allowlist is deliberate. As a free checkbox, everyone would tick it and the efficiency of multi-tenancy would erode. Netflix also does not silently resize jobs to fill their nodes.
The launch-latency tax
On a shared host, a new task starts in seconds because the host is already booted and its image layers are cached. A single-tenant task waits tens of seconds for an instance to boot, then for its image to be pulled.
Warm pools (instances booted in advance) work well for the near-uniform shared fleet. Here they work poorly, because best-fit sizing produces many instance shapes. A warm r7i.2xlarge does nothing for a task that needs an m7a.8xlarge. During a regional failover, a thin pool spread across many shapes drains immediately.
Netflix's response:
Keep only a small amount of warm capacity, and narrow the menu to a curated set of sizes.
Pull images lazily (stargz, SOCI), fetching only the bytes a container reads at startup.
Consider checkpoint/restore with CRIU for slow-starting processes.
Drop slow-booting metal instances.
Disable Kubernetes preemption logic, which stress tests showed was leaving fresh nodes idle while pods waited.
Measurement and results
Isolation is measured differentially. A reference app runs alone, then beside a "havoc" container that generates:
an XDP UDP flood,
pointer-chasing and STREAM to pollute the cache and saturate memory bandwidth,
fsync storms against the disk,
stress-ng against the scheduler,
GPU-burn on the accelerator.
Runs count only once 9, the kernel's measure of time tasks spend stalled on a resource, shows real stalling. Results are medians with bootstrapped confidence intervals. The score blends retained throughput with undisturbed P99 latency divided by P99 under attack. 100% means perfect isolation, and any single hard failure voids the run.
The post publishes no actual scores. Its outcomes are qualitative. Flink, streaming connectors and telemetry pipelines report stable operation, and GraphQL and edge services followed.
On efficiency, Netflix reports a counterintuitive result. Sensitive workloads used to be spread out with anti-affinity rules, and disruption-sensitive pods blocked defragmentation, so shared hosts sat partly empty. Moving those workloads out lets the shared fleet pack more densely. Theory favours multi-tenant packing, but AWS sizes are discrete, so large hosts leave sizeable bin-packing holes. Netflix says right-sizing one workload per instance captured more efficiency in practice. No figures are given. Patching is also easier: draining a single-tenant node relocates one task.
Trade-offs and what's open
Every host carries a roughly fixed overhead of CPU and a few GB of memory for sidecars, the kubelet and system services. On a small node that can be tens of percent; on the largest shapes it is about 2%. Add the gap between the request and the next instance size, and Netflix's guidance follows. The mode suits large, long-running, disruption-sensitive jobs sized close to a node shape. It is the wrong choice for small, bursty or elastic jobs.
More nodes also mean more load on the control plane and more EC2 launches. Netflix mitigates this with cells and staged rollouts. The shared fleet still gets work:
eBPF Earliest Departure Time shaping, which rate-limits by timestamping packets instead of taking a lock, to replace HTB;
Intel RDT and AMD PQoS, hardware partitioning of cache and memory bandwidth;
NVIDIA MIG for GPU partitioning.
Longer term, Netflix wants one intent-based Compute API covering both VMs and containers.
Remember
cgroups can't divide cache, memory bandwidth or kernel network locks; HTB's global lock was Netflix's worst source of interference.
Giving dedicated nodes to a few noisy or fragile jobs let the shared fleet drop spreading rules and pack denser.
Best-fit sizing creates many instance shapes, which undermines warm pools and makes launch latency the main cost.
The Linux kernel treats a disk as a block device: numbered fixed-size blocks you can read and write. A filesystem such as ext4 sits on top and turns those blocks into files and directories. If something provides the blocks, ext4 does the rest.
Image: Fly.io
Fly.io has replaced the storage layer under 1, its persistent Linux sandboxes, with the Sprites Block Device (SBD). The goal was reliability. Fly's argument is that infrastructure should be the plain thing that works, not the cleverest thing you can build.
Why the disk lives in a bucket
A Sprite's disk is rooted in object storage, not in the local drive of whichever server it runs on. That lets a Sprite sleep and wake on another machine with its files intact. But object stores can't change part of an object: editing one byte means rewriting the whole object. Linux programs edit small parts of files constantly, so something has to translate.
The old design: three ledgers
At launch that translator was a heavily modified 2. JuiceFS is a filesystem that stores data chunks in a bucket and keeps a separate metadata database recording which chunks make up which file. Fly's version kept that metadata in SQLite, replicated by 3, and stored a single file: a 100 GB disk image, mounted through a 4 (which makes a file look like a disk) and formatted as ext4. Checkpoints were 5 clones of that image, listed in a second SQLite database.
So the disk was three things: chunks, chunk metadata and a checkpoint list. Each was made durable separately, which meant they could disagree. For speed, JuiceFS ran in 6, recording a chunk in metadata before uploading it. If a host died between those two steps, metadata pointed at a chunk that didn't exist. ext4 then hit a block that failed its checksum and went read-only. Checkpoints didn't help, because the clones shared the same missing chunk.
A test harness put 400 Sprites through about 5,000 write, checkpoint, sleep, wake and restore cycles. A "nonzero number" of them permanently lost the ability to checkpoint.
The new design: just a disk
SBD presents a bucket to Linux as a block device using 7 (Network Block Device), a mechanism that lets a program answer the kernel's disk reads and writes. The Sprite formats it as ext4 as usual. Metadata now lives in the bucket next to the blocks it describes, so there's no separate database to keep in sync. A daemon, sbd serve, handles reads and writes from a local cache, uploads modified blocks in the background and flushes on demand.
A checkpoint is now a versioned snapshot of the whole volume, recorded in a single catalog. A restore unmounts the filesystem, rewinds the volume in place and remounts it, at about the cost of a restart. SBD checks every block it reads back against what it wrote, and if the daemon dies, the whole Sprite restarts instead of running on a disk it can't trust.
Fly says SBD is "orders of magnitude more reliable" but publishes no failure rates or performance numbers.
Fly.io has replaced the storage layer under 1, its persistent Linux sandboxes, with what it calls the Sprites Block Device (SBD). Users shouldn't see any difference except reliability. The post is about getting rid of a clever homemade design. The author's analogy is homemade pasta versus a $1 bag of spaghetti. Fly's view is that good infrastructure is the simple thing that works, not the thing you put the most effort into.
The fixed constraint: the disk lives in a bucket
One design choice is unchanged from the first version. The root of a Sprite's disk is object storage, not the NVMe (fast local flash) drive in whatever server the Sprite happens to be running on. That is what lets a Sprite sleep, wake on another machine and still find its files. In Fly's words, a Sprite's durable state is "more or less, a location in a bucket."
The catch is that object stores can't modify part of an object. Changing one byte means replacing the whole thing. Linux programs rewrite small pieces of files all the time, so something has to translate between the two.
Version 1: 2 holding one giant file
At launch, that translator was a heavily modified JuiceFS. JuiceFS is an open-source filesystem that normally stores file data as chunks in a bucket and keeps a separate database (the metadata) recording which chunks make up which file. Fly's version used it unusually:
The metadata backend was a rewritten SQLite database, kept durable with 3, a tool that continuously replicates SQLite to object storage.
JuiceFS stored exactly one file: a 100 GB disk image.
That image was attached through a 4 (a Linux feature that makes a file look like a disk) and formatted with ext4. Linux did all the ordinary filesystem work, and JuiceFS just held the image.
Checkpoints were 5 clones of the image. A clone shares the existing chunks and only stores new data when something changes. That made checkpoints fast, because taking one was mostly a matter of moving metadata around. A second SQLite database, checkpoints.db, listed the checkpoints and which image to mount for each.
Three ledgers, three ways to disagree
So a Sprite's disk was really three things:
chunks in the bucket,
a metadata database mapping chunks to each image,
a checkpoint database mapping checkpoints to images.
Each was made durable separately, so any one could be correct while the others were wrong.
The post describes a failure that "may or may not be based on lived experience". For speed, Fly ran JuiceFS in 6, which records a chunk in the metadata first and uploads the bytes afterwards. If the machine dies between those two steps, the metadata points at a chunk that doesn't exist. Here is that sequence as a worked example:
An application writes to a directory, and ext4 updates a directory block.
JuiceFS records "the image now uses chunk X" in SQLite, and Litestream replicates that record.
The host is killed before chunk X is uploaded.
On the next boot, JuiceFS asks the bucket for chunk X and gets a 404 (not found).
ext4 sees a directory block that fails its checksum and switches the filesystem to read-only.
Moving the Sprite to another machine doesn't help. Restoring a checkpoint doesn't either. The checkpoints were clones sharing the same chunks, so every one of them had the same hole. The safety net and the thing it protected depended on the same data.
The dogfooding result
Fly built a harness that drives fleets of Sprites through repeated write, checkpoint, sleep, wake and restore cycles and checks every byte that comes back. Run against 400 JuiceFS Sprites for about 5,000 cycles, it left "a nonzero number" of them permanently unable to checkpoint. The post doesn't give the exact count.
Fly first tried to fix the existing design with more reconciliation, repair tooling and cleverer ways to keep the three ledgers in agreement. The author's verdict is that "spaghetti that comes with a repair kit" is still the wrong recipe.
Version 2: SBD, just a disk
SBD presents a bucket to the Linux kernel as a block device using 7 (Network Block Device). NBD is a standard Linux mechanism that lets a program outside the kernel answer the kernel's block reads and writes. The Sprite formats that device as ext4 and mounts it like any disk.
The structural change is that metadata lives in the bucket alongside the blocks it describes, so there is no separate database to keep in sync. A daemon inside the Sprite, sbd serve:
answers the kernel's reads and writes from a local cache,
uploads dirty (modified but not yet uploaded) blocks to object storage in the background,
flushes on demand when something needs a guarantee that the bytes are actually stored.
ext4 still does all the filesystem work. The stack under it shrinks from a filesystem, a database and a disk image to one daemon and one bucket.
Checkpoints and restores
A checkpoint is now a block-level snapshot of the whole volume with a version number, recorded in SBD's own checkpoint catalog. That catalog is the single source of truth and replaces checkpoints.db. Because each checkpoint knows which volume version it belongs to, a restore unmounts the filesystem, rewinds the volume in place and remounts it. Fly says this costs about as much as a restart.
Failing loudly
SBD doesn't promise that nothing will break. It changes what happens when something does. SBD checks every block it reads back from object storage against what it wrote. If SBD itself dies, the whole Sprite restarts, on the reasoning that a clean restart is the only way back to a disk you can trust. A Sprite now stops and reports the problem instead of running on a disk it can't vouch for.
Results and open questions
Fly reports that continued dogfooding, an aggressive test harness and a few design partners show SBD to be "orders of magnitude more reliable" than the JuiceFS setup. SBD is now the Sprites filesystem, and Sprites created before October 6 can be migrated.
The post leaves several things unmeasured or unexplained:
Failure rates: no counts or rates for either system.
Performance: no numbers comparing SBD with the writeback JuiceFS setup it replaced.
Bucket layout: no detail on how block metadata and checkpoint versions are stored in the bucket.
Durability of acknowledged writes: background uploads of 8 raise the same basic question writeback mode did, namely what is lost if the host dies before an upload finishes. The post's answer is detection: a checksum mismatch or a daemon crash stops and restarts the Sprite instead of letting it run on a damaged disk. It doesn't say how much recent data can be lost when that happens.
Remember
Keeping metadata separate from data, with each made durable independently, let a single crash corrupt both the live disk and every checkpoint.
Copy-on-write checkpoints share chunks, so one missing chunk breaks the backups along with the live disk.
SBD's answer to corruption is to detect it, check blocks on read and restart the whole Sprite, not to repair it.
Hillel Wayne, a writer on formal methods and testing who now works at the testing company 1, asks a simple question in his latest newsletter. You've converted a Python program to Rust. How do you know its behavior didn't change?
The standard answer is 2: run the original and the new version on the same inputs and compare their outputs. Wayne shows that this breaks down when the programs are 3, meaning the same input can give different outputs on different runs.
When it works
Wayne writes A -> (x, y) to mean that on input A, the original produced x and the new version produced y. For deterministic programs, a single B -> (1, 2) proves they differ. Running the same input twice tells you nothing new, so you just try more inputs and gain confidence with each one that matches.
When it doesn't
Take a web endpoint that can time out:
A -> (200 OK, 408 timeout)
A -> (200, 200)
You can't call these programs different. The next run might give A -> (408, 200), which would show the original can time out too. Once the conclusion depends on comparing several runs instead of one, Wayne notes, you are dealing with 4: properties judged across a set of executions rather than any single run.
Matching outputs can still hide a difference
Suppose you've seen (200, 408), (200, 200) and (408, 408). Both programs have produced both status codes. But (408, 200) hasn't appeared yet. Maybe the new program never succeeds where the old one would have failed, which would make it less reliable. Wayne's rule: if you see (x, y), you need to see the reverse pair (y, x), not just some other mixed pair.
Then there are frequencies. If the new version times out 100 times as often, or a fair coin flip has become a biased one, the programs differ even after you've seen every combination of outcomes.
Where he lands
Wayne names two problems. One divergent run proves nothing, and "different" can mean different output sets, different outcome pairs or different probabilities. He also notes that even deterministic programs can regress without changing their output, for example by getting much slower. His practical advice is that showing both programs produce the same set of possible outputs is usually enough. That's still hard, but it avoids reasoning about probabilities.
The post doesn't say how many runs per input would be enough, or how to remove randomness during testing, for example by fixing random seeds or controlling the thread scheduler.
He also announces an Antithesis webinar on 5 for web apps. He argues that checking 6 about system state, rather than algebraic laws or round-trip properties, is what will make randomized testing useful to more developers.
The question
In a newsletter issue that also covers personal news, Hillel Wayne, a writer on formal methods and testing who now works at the testing company 1, asks a practical question. Say a Python program has been converted to Rust, for example by an automated translation tool. How do you know the conversion didn't change what the program does?
The usual answer is 2: run the original and the new version on the same inputs and compare the results. Wayne shows that this answer stops working once the programs are 3, meaning the same input can give different outputs on different runs.
The deterministic case
Wayne uses a small notation. A -> (x, y) means you gave input A to both programs, the original output x, and the new one output y.
A -> (1, 1): the programs agree on A.
B -> (1, 2): they disagree on B.
If both programs are deterministic, one disagreement proves they differ. Each input only needs testing once, because running it again would give the same answer. So the method is simple: try more and more distinct inputs. Every input that matches raises your confidence in the translation, and the first one that doesn't is a confirmed bug.
What changes with nondeterminism
Now imagine a web endpoint that sometimes times out. You might see:
A -> (200 OK, 408 request timeout)
A -> (200, 200)
Are the programs different? Wayne says you can't tell. If the original always returns 200, then yes. But a third run might give A -> (408, 200), which would show the original can time out as well. The divergent first run could just be bad luck.
He sums it up as a two-by-two grid:
observe A -> (x, y)
observe A -> (x, x)
Deterministic
definitely different
unknown
Nondeterministic
unknown
unknown
Only one cell gives you a firm answer, and it is the one that needs determinism. As soon as a verdict depends on comparing several runs rather than one, Wayne notes, you are dealing with 4: properties that can only be judged across a set of executions, not from any single one.
Matching output sets aren't enough
Next Wayne takes the observations:
A -> (200, 408)
A -> (200, 200)
A -> (408, 408)
Both programs have now produced both 200 and 408, so the sets of possible outputs look the same. Wayne argues this still isn't agreement. The pair A -> (408, 200) has never appeared. Perhaps the new program never succeeds on a run where the old one would have timed out, which would make the two programs differ in reliability even though they produce the same values.
The rule he draws: seeing A -> (x, y) is not balanced out by seeing some other mixed pair such as A -> (z, x). You need to see the reverse pair, A -> (y, x). So the target is no longer "do the outputs match?" but "do the patterns of joint outcomes match?" That takes many more observations to establish.
Frequencies matter too
Wayne adds one more layer. Suppose the new program returns 408 a hundred times as often as the old one. Or suppose a fair coin flip has turned into a biased one. You could see both A -> (H, T) and A -> (T, H) and the programs would still differ, because the probability of each outcome has changed. Catching that kind of difference needs statistical testing and a decision about how much skew counts as a bug, not just a comparison of outputs.
Two difficulties, and a practical compromise
Wayne names two separate problems:
You can't conclude much from one run. A single mismatch no longer proves the programs differ.
"Different" is poorly defined. You might mean different sets of outputs, different pairs of outcomes, or different probabilities, and each needs a different testing strategy.
He points out that deterministic programs have a quieter version of the second problem. Identical output can still count as a regression if the new version is much slower. Which differences matter depends on how you define correctness.
His practical suggestion is that in most cases it is enough to show both programs produce the same set of possible values for each input. That is still hard, since it means sampling each input enough times to be confident you've seen every possible output. But it avoids having to reason about probabilities.
What the post leaves open
This is a short essay, not a method. Wayne doesn't say how many repetitions per input would be enough, how to control the sources of randomness (for example by controlling thread scheduling or injecting faults on purpose so that runs can be replayed), or how to choose a threshold for frequency differences. One obvious approach is to make both programs deterministic during testing by fixing the random seed and controlling the scheduler. That turns the problem back into the easy deterministic case, and it is roughly how deterministic-simulation tools like Antithesis's work. Wayne doesn't discuss it here, though.
The same issue also announces Wayne's upcoming Antithesis webinar on 5 of web applications. Property-based testing generates random inputs and checks that stated rules hold for all of them. Wayne argues that it becomes most useful when the rules are 6 about the system's state, such as "an account balance never goes negative," rather than algebraic laws or round-trip properties like "decode(encode(x)) == x." He discloses that the webinar covers open-source tools, and that Antithesis runs it partly to build interest in property-based testing that may help its business.
Remember
With nondeterminism, a single mismatched run is evidence of a difference, not proof of one.
Matching output sets can hide a difference: seeing (x, y) means you also need to see (y, x).
Changed probabilities, like more frequent timeouts, count as divergence even when every outcome combination has appeared.
With "Sign in with Google/Apple/GitHub", the site redirects the user to that identity provider, which authenticates them and sends back a verified identity. This is typically done with OAuth or OpenID Connect. The site's own sign-up form never appears, and the provider's screens can't be customized.
Image: Cloudflare
Cloudflare has earned South Korea's Information Security Management System (1) certification under the ISMS-P framework, which adds personal-information requirements. The final audit took ten days. Auditors appointed by the Korea Internet & Security Agency (2) worked through 80 controls. The full effort took about 18 months.
What ISMS checks
ISMS covers access control, encryption, incident response, risk management and disaster recovery. It judges them by how personal data actually flows through a company and against Korea's Personal Information Protection Act (3). Certificates last three years, with annual re-audits. The author argues it isn't harder than the 4 family Cloudflare already holds, just more local: the same principles plus Korean regulatory requirements.
Consent, itemized
Elsewhere, one line covers terms, privacy and cookies. Korean users see four separate items, each linked to its full Korean text, plus an "agree to all" box. Three are required: terms of service, collection and use of personal data, and overseas transfer. One is optional: sharing data with marketing partners. The sign-up button stays disabled until all three required boxes are checked.
Dropping social sign-up
The hard part was sign-up through Google, Apple or GitHub. Those flows send the user to the identity provider and bring them back authenticated, so Cloudflare's own form, where consent lives, never appears. Cloudflare had two options: build a separate consent step after login for one country, or remove the bypass. It removed the bypass. Social sign-up isn't offered in South Korea, so every account there goes through the one form that enforces consent.
Two country signals
The default form comes from the country code Cloudflare's edge derives from the client IP. That code is exposed as the CF-IPCountry header, or as request.cf.country in a 5 (Cloudflare's edge-run serverless code). If the code is "KR", the Korean form appears. But IP geolocation is an estimate. A Korean resident abroad, on a VPN, or behind a carrier's 6 (many users sharing one IP) may not look Korean. So the country the user selects at sign-up overrides the IP guess. As the post puts it, the IP tells you where a request came from, and only the user can say where they live.
The audit and its scope
Auditors rejected written descriptions of data flows. They wanted diagrams they could interrogate, and they tested every access-control layer from data collection to deletion. They also checked the exact wording of each consent item. Part of the delay came from audits covering a single annual period: preparation crossed into a second year, so evidence had to be redone.
Unlike 7 or ISO 27001, which name the products they cover, the ISMS certificate defines scope as a boundary: the organization, the infrastructure and the data flows. A product is covered if it is delivered through that certified service, whether or not it is named. The post doesn't detail the controls, the findings or how customers verify the delivery path.
The news
Cloudflare has been certified under South Korea's Information Security Management System (1), a national security certification run within the ISMS-P framework (the variant with "-P" adds personal-information requirements). The final audit lasted ten days. Auditors appointed by the Korea Internet & Security Agency (2) went through 80 controls with Cloudflare staff. The whole effort took about a year and a half. Most of the post is about compliance, but the interesting engineering is how a legal requirement forced a change to the account-creation path.
What ISMS is
ISMS covers familiar ground: security policy, access control, authentication and permission management, encryption, incident response, risk management and disaster recovery. The difference is what it measures against. Controls are judged by how personal data actually moves through the company's systems, and against Korea's Personal Information Protection Act (3) and its regulatory guidelines. The legal basis is Article 47 of Korea's network act, the Act on Promotion of Information and Communications Network Utilization and Information Protection. The criteria come from its Enforcement Decree and a notice issued by the Ministry of Science and ICT (MSIT). KISA is a government institution affiliated with MSIT. After the audit, a special committee reviews the findings. A certificate lasts three years, with an annual audit to keep it.
Cloudflare already holds 4, 27701, 27017 and 27018, which are international standards for security management, privacy management and cloud security and privacy. The author's view is that ISMS doesn't replace these. It localizes the same principles and adds Korean regulatory requirements on top. The author calls it "not a harder certification" than the others but "a more nuanced one".
The visible change: itemized consent
Globally, one line above the sign-up button covers everything: "By continuing, I agree to Cloudflare's terms, privacy policy, and cookie policy." Korean users instead see five checkboxes: an "agree to all" box plus four separately labeled items, each linking to its full Korean text:
[Required] Terms of Service
[Required] Collection and use of personal information
[Required] Transfer of a limited amount of personal information overseas
[Optional] Sharing a limited amount of personal information with marketing third parties
The three required items are hard gates. The sign-up button stays disabled until all three are checked, so no path through the form creates an account without an affirmative action on each one. Leaving the optional item unchecked only opts the user out of marketing messages.
The hard part: flows that never show the form
Itemizing consent on a form you own is easy. The problem was the sign-up paths that skip that form. Signing up with Google, Apple or GitHub sends the user to the identity provider and brings them back already authenticated. Cloudflare's own sign-up page, where the consent items live, never renders. Cloudflare can't change the provider's screen, so there was no step in that handoff where it could demand three separate affirmative consents before creating the account.
There were two options:
Build a separate consent gate that runs after authentication, for one country only.
Remove the paths that bypass the gate that already existed.
Cloudflare chose the second: social sign-up isn't offered in South Korea. Every Korean account now goes through the one form that enforces consent. The trade-off is friction for users who prefer social login. In return, there is one enforcement point instead of two code paths that must be kept in sync. The article doesn't say whether existing Korean users who signed up socially were affected.
How the branch decides: IP versus declared country
The default comes from the same geolocation signal every Cloudflare customer gets. Cloudflare's edge derives a two-letter country code from the client IP. It exposes this as the CF-IPCountry header at the origin server and as request.cf.country inside a 5, Cloudflare's serverless code that runs at the edge. No extra lookup or round trip is needed. The post's illustrative sketch is a string comparison: if country === "KR", render the form with locale ko-KR, consent set to itemized and social login turned off. Otherwise, render English, combined consent and social login turned on.
That snippet isn't the whole trigger, and the post explains why. IP geolocation is an estimate. Address blocks rotate and change owners. A Korean resident abroad, on a VPN, or behind a mobile carrier's 6 (network address translation, where many users share one public IP) may not show up with a Korean address. So the IP only picks which form appears by default. The country the user selects at sign-up overrides it and drives everything after that. In the post's words: "The IP address tells us where a request came from; only the user can tell us where they reside." Keeping the two signals separate is what makes the consent actual rather than assumed.
As an illustration, a Seoul resident on a VPN exiting in Japan would first see the global form. Once they select South Korea as their country, they get the itemized consent and lose the social-login option.
What the auditors examined
Auditors started from data flow diagrams of every path by which employees could access personal data Cloudflare collects, processes or stores. The flows weren't Korea-specific. The format of the evidence was: auditors refused narrative descriptions and wanted diagrams they could interrogate. For each flow, they tested every access-control layer across internal systems, public websites, event pages and the dashboard, covering the data lifecycle from first contact to destruction. The diagrams aren't published. Auditors also checked the exact wording, structure and separation of each consent notice.
Why it took 18 months
An ISMS audit covers a single annual period, and all evidence must reflect that period. Cloudflare's preparation ran past one audit year into the next. That meant retranslating updated policies, re-collecting and reassessing evidence, republishing the privacy policy, and absorbing website design changes made in the meantime.
Scope is a boundary, not a product list
This is the operationally useful part for buyers. 7 reports and ISO 27001 certificates name the services they cover, so a procurement team checks that its product is on the list. KISA instead defines scope as a service boundary in three parts:
the organizational perimeter: the entities, teams and processes that run the service
the infrastructure perimeter: the network and platform it runs on
the information assets and personal-data flows inside that boundary
A product is covered if it runs on the certified platform, is operated by the certified teams and is governed by the same ISMS, whether or not it is named. The certificate is held by Cloudflare, Inc. and listed on KISA's public register.
What's left open
The post gives no detail on the 80 controls, findings or remediations. It doesn't say whether a post-authentication gate was seriously costed, how the country override is stored or audited later, or how many products actually fall inside the boundary. For a vendor reviewer, the boundary model moves the burden onto confirming the delivery path instead of finding a product name. The post asserts this but doesn't show how a customer would verify it.
Remember
Consent the law needs to enforce must have one choke point; any login path that bypasses it has to be gated or removed.
Use IP geolocation for the default only; let the user's declared country of residence override it for legal decisions.
ISMS scope is a service boundary rather than a product list, so buyers confirm coverage by how a product is delivered.
6. This is a user-space kernel that intercepts system calls. It slows the I/O-heavy paths that most need protecting, and the authors say it is effectively maintained by a single vendor.
Plain EC2. Teams would take back instance selection, patching and scaling.
