A certificate-authority failure Cloudflare fixed in about 26 hours, AI agents keeping a compiler port in sync, and what it takes to keep GPUs and agents running reliably.
Picked from 27 new articles across 23 sources · arrow keys for other days
Today in 30 seconds
30s left
Weak keys in eight eMudhra certificates allowed forgery of any site's certificate. Cloudflare fixed its systems in about 26 hours. Non-browser clients need manual trust-list updates. story 1 ↓
Cloudflare's tsrs is an experimental Rust port of the TypeScript compiler. Early tests show it up to 8x faster than Go-based TypeScript 7.0, using about 70% less memory. story 2 ↓
Meta made its Prometheus supercluster 99% less prone to interruption. The cluster spans buildings up to 100 km apart and is growing toward one million GPUs. story 3 ↓
IBM's ALTK-Evolve added self-written rules to a GPT-4.1 agent's prompt. It reached 67.5% Pass^4 on τ²-bench retail, 6.1 points above a fine-tuned model. story 4 ↓
DeepSeek open-sourced its in-house inference engine, while the US federal government banned agencies from buying DeepSeek. story 5 ↓
1/5Cloudflare· Christopher Patton· Oct 20
Weak keys in eight eMudhra certificates let attackers forge any website's certificate
The private keys behind eight widely trusted eMudhra intermediate certificates were compromised, so anyone could forge a certificate for any domain. Cloudflare patched its software and network within about a day. Operators must patch their own systems.
Why it matters · If your services call external APIs or connect to backends over TLS, their trust settings may live in places that browser fixes never reach.
Know this first
Context from the editor, not the article · 35 sec
1 · What a certificate proves
When you connect to a site over TLS, the encryption layer behind HTTPS, the server shows a certificate saying "this key belongs to example.com." A certificate authority (CA) signs it. Your software accepts it only if the signature traces back to a CA it already trusts.
2 · Roots, intermediates, trust stores
CAs keep their root keys offline and sign day-to-day certificates with intermediate certificates. A trust store is the list of roots a browser, operating system or library accepts. If an intermediate's private key leaks, whoever holds it can sign a valid-looking certificate for any domain.
3 · Revocation is slow by default
2/5Cloudflare· Boshen Chen· Oct 20
Cloudflare releases tsrs, an experimental Rust port of the TypeScript compiler
Cloudflare, VoidZero and Owner.com have ported the TypeScript compiler to Rust. The port, tsrs, checks code up to 8x faster than TypeScript 7.0 and uses about 70% less memory. It is built to run efficiently in WebAssembly.
Why it matters · It shows AI-assisted ports of a huge, subtle codebase becoming practical, and it tests whether agents can keep a port faithful to an upstream that keeps changing.
Know this first
Context from the editor, not the article · 35 sec
1 · What a type checker does
TypeScript is JavaScript with type annotations. Its compiler, tsc, reads a whole project and checks every value against its declared type before the code runs. On large codebases this check can take minutes and a lot of memory, and developers wait on it in editors and build pipelines.
2 · TypeScript already went native
Microsoft's TypeScript team rewrote the compiler from TypeScript itself into Go. That version ships as TypeScript 7 and is roughly 10x faster than the old one. Other teams have since tried their own ports in Rust or C++, with mixed faithfulness to the original.
3 · WebAssembly and garbage collection
3/5Engineering at Meta· Oct 19
How Meta made its Ohio GPU supercluster 99% less prone to interruption
Meta describes Prometheus, its Ohio cluster of hundreds of thousands of GPUs that trained its Muse models. Hardware spares, faster firmware fixes and detailed failure tracking cut job interruptions by 99% from their mid-2025 peak.
Why it matters · At this scale, training speed depends less on peak chip specs than on how often the job stalls. This is a rare, number-heavy look at how one lab keeps those stalls down.
Know this first
Context from the editor, not the article · 40 sec
1 · Why big training is fragile
Training a large model splits the work across thousands of GPUs that move forward in lockstep and swap results at every step. If one chip, cable or switch fails, every GPU waits. The job then rolls back to its last saved state, called a checkpoint, and the work done since that save is lost.
2 · Scale-up versus scale-out
Inside a server or rack, GPUs talk over very fast dedicated links. NVIDIA's version is called NVLink. This is the "scale-up" domain. Racks talk to each other over ordinary network cards and switches, which is "scale-out". Scale-out is much slower per GPU, so engineers keep the most frequent communication inside the scale-up domain.
3 · Data-parallel replicas
4/5Hugging Face· Oct 20
IBM's ALTK-Evolve gets to No. 1 on a customer-service agent benchmark by adding learned rules to the prompt
IBM Research's ALTK-Evolve has an AI agent review its own practice runs, write the lessons up as short rules, and add them to its prompt. That took it to first place on τ²-bench retail and airline without any retraining.
Why it matters · It shows that a cheap, automatic layer of prompt-level memory can improve agent reliability more than switching models or fine-tuning. The costs and the right amount of memory are both measured.
Know this first
Context from the editor, not the article · 42 sec
1 · LLM agents and ReAct
An agent is a language model that runs in a loop, calling tools such as find_user_id_by_name_zip or a refund function. ReAct is a simple, common loop: the model reasons about what to do, takes one action, reads the result and repeats. Its behaviour is shaped mostly by the system prompt, the standing instructions it receives on every call.
2 · Prompting vs fine-tuning
There are two ways to improve a model on a task. Fine-tuning retrains its weights on examples, which is costly and tied to that one domain. Prompting leaves the weights alone and changes the text the model reads. Text that repeats on every call can be served through prompt caching, which makes it cheaper.
3 · Measuring agent reliability
5/5Interconnects (Nathan Lambert)· Florian Brand· Oct 20
DeepSeek open-sources its in-house inference engine as US and China tighten rules
Interconnects' October roundup of open-weight AI: DeepSeek released DeepInfer, the software it uses to serve its own models, plus a cheap new model. Meanwhile US and Chinese regulators moved to control open models.
Why it matters · Serving software, licenses and procurement rules now shape which open models you can afford to run, and are allowed to run, as much as benchmark scores do.
Know this first
Context from the editor, not the article · 42 sec
1 · Open-weight models
An open-weight model is one whose trained parameters (the weights) anyone can download and run on their own hardware. That differs from a closed model like GPT, which you can only use through the maker's API. The license decides what you may legally do with the weights. MIT and Apache 2.0 are permissive licenses that allow commercial use.
2 · Reading model names
Names like 552B-A16B describe a mixture-of-experts model. It has 552 billion parameters in total, but only about 16 billion are active for each token, because a router sends each token to a few specialist sub-networks. The result is the knowledge of a big model at roughly the compute cost of a small one.
Taking back a bad certificate is called revocation. In practice, most software outside browsers only changes what it trusts when someone ships a new trust store and restarts the program. Browsers ship small revocation lists that they can update quickly, such as Chrome's CRLSets.
Image: Cloudflare
What happened
On October 19, 2026, Cloudflare learned that the private keys behind eight 1 from eMudhra, a 2, had been compromised. Browsers, operating systems and Cloudflare itself all trusted these certificates. Cloudflare says the keys "are weak," so an attacker could sign a forged certificate for any domain, pose as that site's server, and read or change its traffic. The article doesn't say how the keys became weak. Cloudflare found no sign that anyone had used the flaw against its services.
Why it's serious
HTTPS runs on 3 (Transport Layer Security). TLS clients trust a server's certificate if a chain of signatures leads back to a root certificate authority in their 4, the list of authorities they accept. Roots sign through intermediates, so a forged certificate signed by one of these eMudhra intermediates passes every normal check. Encryption still works, but authentication, the step that proves you reached the right server, does not. The fix is to distrust the eight intermediates everywhere trust is decided. That means OS certificate bundles, language runtimes, HTTP libraries, container images and custom stores, and each program usually needs a restart before the change takes effect.
Cloudflare's response
Cloudflare first patched services that verify remote servers, such as its proxies connecting to customers' 5 (the customer's own backends) and its apps calling external APIs. At the same time it updated shared trust stores and base images, then checked apps with custom stores one by one. Updates for WARP, its VPN-like client app, shipped about 9 hours in; Apple's app review is still holding up the Cloudflare One Agent for iOS. 6, its tunnel daemon, followed after about 16 hours, and fixes across the content delivery network (CDN) and Zero Trust products finished after about 26 hours. Cloudflare says more work remains.
What you should do
Look beneath the SDK. Your API client may hand verification to a library, runtime or the OS. Updating the SDK alone may change nothing.
Upgrade the packages on Mozilla engineer Dennis Jackson's crowdsourced list of patched packages, including indirect dependencies. Then rebuild images and restart.
Test the path the app actually uses. A passing browser or curl test doesn't prove a background service is safe.
Never clear the resulting certificate errors by turning off verification.
The bigger point
Browsers fixed this almost instantly because they use fast revocation channels, such as Chrome's 7. Most server software treats its trust store as a static file. Cloudflare argues for projects like upki, which treat the trust store as something that can be updated quickly with revocations and 8 keys (keys for the public logs that record every certificate issued).
What happened
On October 19, 2026, at about 17:00 UTC, Cloudflare learned that the private keys for eight 1 issued by eMudhra, a 2, had been compromised. Browsers, operating systems and Cloudflare's own systems all trusted these certificates. Cloudflare says "the keys themselves are weak," so an attacker could sign a forged certificate for any domain. With one, an attacker in the network path could pose as a real server and read or change its traffic. The article doesn't say how the keys became weak or who found the problem.
Cloudflare patched its critical services before the public disclosure. Its security review found no sign that anyone had used the flaw against Cloudflare services.
Why one bad intermediate matters so much
The 3 is the public system of certificate authorities (CAs) and trust lists that secures HTTPS. In it, any trusted CA can vouch for any domain. A certificate authority signs statements like "this public key belongs to bank.example." 4 (Transport Layer Security, the protocol behind HTTPS) clients check that signature against a 5, a list of root CAs they accept. Roots usually sign through intermediate certificates, so a chain like root → eMudhra intermediate → forged bank.example certificate passes every check. The verifier can't tell the signing key was stolen or guessable. The encryption still works. What fails is authentication, the step that confirms you reached the right party. Cloudflare stresses this point: TLS's encryption algorithms are not weakened.
The fix is to stop trusting those eight intermediates. That sounds simple, but trust decisions live in many places. These include the operating system's certificate bundle, a language runtime such as Java or Python, an HTTP library, a Docker base image, or a store compiled into an application. Each one needs its own update, and the program usually needs a restart before the change takes effect.
How Cloudflare responded
Cloudflare worked in priority order:
Services that authenticate remote endpoints with public certificates. These were the most exposed. They include Cloudflare's proxies connecting to customers' 6 (the customer's own backend behind Cloudflare) and internal applications calling external APIs. A forged certificate could impersonate the other end of either connection.
Shared foundations. In parallel, teams updated shared trust stores and base images, so every service that inherits trust from them got the fix. Reloads, restarts and redeployments followed to activate it.
Odd cases. Applications with bundled or custom trust stores were checked and updated one by one.
Timeline:
The first client fix shipped about 9 hours after Cloudflare learned of the problem, and network-wide fixes finished about 26 hours in. Customers using Gateway, Cloudflare's filtering proxy within its Zero Trust remote-access suite, together with an up-to-date client are protected automatically. Cloudflare says remediation is still ongoing and that "there's much work still to be done."
What engineers should do
The most useful advice is aimed at developers: look beneath the SDK. An API client may hand certificate checks to an HTTP library, which may use the language runtime, which may use the operating system. Bumping the SDK version alone may change none of that. Concretely:
Upgrade affected packages. Mozilla's Dennis Jackson keeps a crowdsourced list of patched packages. Many will be indirect dependencies, so check the full dependency tree.
Rebuild container images, then reload, restart or redeploy. An updated file on disk does nothing until the process reads it again.
Test the real path. Jackson's test site checks browsers and explains how to test other software. Cloudflare warns that a passing browser test says nothing about a background service. A passing curl test doesn't either, if the app uses a different verifier.
Don't turn off verification to clear the certificate errors that may appear after patching. That removes the very protection the patch restores. The errors may mean a legitimate upstream server was using one of the bad chains.
Customers running their own origin servers and trust configurations are the most likely to need to act.
The structural lesson
Cloudflare's closing point is that the incident hits some systems much harder than others. Browsers fixed it "with a flip of a switch." That is partly because each browser is one application rather than hundreds. It is also because they use quick-revocation channels like Chrome's 8, small revocation lists pushed to the browser outside the normal release cycle. Cloudflare uses a similar mechanism for some products. Most server software instead treats its trust store as a static file that only changes when a package update ships.
Cloudflare points to upki as a better model. Upki is a project built on the idea that a trust store should be a live feed, not a fixed list of root CAs. The feed would carry revocations and 9 keys (certificate transparency is a system of public logs that record every certificate issued). The article doesn't describe how upki works or how widely it is used.
Open questions
The article doesn't explain why the keys were weak, which domains or chains eMudhra had legitimately issued under them, or whether forged certificates were used anywhere outside Cloudflare. It also gives no count of how many internal services needed custom-store fixes, and no date for when remediation will finish.
Remember
A leaked intermediate key breaks authentication for every domain. TLS encryption itself is unaffected.
Trust lives in many layers: OS, runtime, library, image and custom stores. Each needs updating and a restart.
Browsers fixed this almost instantly with fast revocation lists. Most server software has no such channel.
WebAssembly (Wasm) is a portable bytecode that runs in browsers and sandboxed servers. Go manages memory with a garbage collector, which has to be bundled into every Wasm build. Rust frees memory at compile-time-known points, with no collector, so its Wasm builds are leaner.
Benchmark at equal worker counts: tsrs against TypeScript 7.0 and 7.1 beta on speed and memory.1 / 2
Cloudflare has released an experimental version of 1, a Rust port of the TypeScript compiler. It was built with 2, the company behind the Vite and 3 JavaScript tools, and with Max Schwenk of Owner.com. The project sits inside Oxc, a Rust toolchain for JavaScript, and Cloudflare says it is working closely with Microsoft's TypeScript team.
Not a fork
TypeScript adds types to JavaScript, and its compiler checks those types across a whole project. Cloudflare stresses that tsrs is not a fork. The TypeScript team keeps sole control of the language, and tsrs aims to copy the behavior of the official compiler exactly. The team sometimes spends years prototyping a feature before settling on a design. Cloudflare argues that a port which drifts from those decisions would eventually become incompatible and fall apart.
The numbers
4 is Microsoft's own rewrite of the compiler in Go, already about 10x faster than the original. Against it, using the same number of parallel 5, tsrs is up to 8x faster and uses about 70% less memory. The TypeScript team agreed the comparison uses the same configuration. Cloudflare says that matters because other port benchmarks often used bigger machines or non-default settings. The upcoming TypeScript 7.1 beta is itself up to 2.5x faster than 7.0, partly thanks to optimizations taken from tsrs. By default, tsrs adjusts its settings to the machine, using more memory to go faster still.
Why Rust if Go is fast?
The main reason is 6, a portable format that runs in browsers and sandboxes. A Go program compiled to Wasm must carry Go's 7, the runtime component that frees unused memory. Rust needs no collector, so its Wasm builds are smaller and pass memory more directly to the host program. Cloudflare wants full development toolchains running cheaply in sandboxes on 8, its serverless platform, for AI agents to use. The same builds could also run in a browser or on small devices.
Keeping up with upstream
Cloudflare says frontier AI models can now port huge codebases in days. It plans to keep tsrs current with agents, as it does for 9, its version of the Next.js web framework. Each morning an agent reviews upstream changes and files issues. Each night the upstream test suite runs against the port, and agents propose fixes for any gaps.
Caveats
This is an experimental release (npm install tsrs). It still differs from upstream in places, such as the order in which it prints errors. The speed figures are "up to" results from early benchmarks.
Cloudflare has published an experimental release of 1, a port of the TypeScript compiler to Rust. 2, the company behind the Vite build tool and the 3 toolchain, worked on it with Cloudflare, along with Max Schwenk of Owner.com. tsrs is being incubated inside Oxc, a set of Rust-based JavaScript tools. Cloudflare says it is working in close collaboration with Microsoft's TypeScript team. Anyone can try it with npm install tsrs and npx tsrs.
The problem: waiting on the type checker
TypeScript is JavaScript plus type annotations. Before code runs, the compiler, tsc, checks every expression in the project against its declared types. On a large codebase that check can take a long time and a lot of memory. Developers wait on it in their editors, in pre-commit hooks and in CI (continuous integration, the automated build-and-test run on every change).
Microsoft has already attacked this once. It rewrote the compiler, originally written in TypeScript, in Go. That rewrite ships as 4, and the article credits it with about a 10x speedup and much lower memory use. Cloudflare says this was the only successful port among many attempts over the years, because the compiler is so complex.
Why porting a type checker is hard
A type checker isn't defined by a short specification. In practice, TypeScript's behavior is its implementation, with thousands of edge cases in how types are inferred, narrowed and reported. A port therefore has to match the reference code decision by decision. Cloudflare also notes that the TypeScript team sometimes spends years prototyping a feature before choosing an implementation. A port must follow each of those choices or it will drift into incompatibility. That is why the company stresses that tsrs is not a fork. The TypeScript team keeps sole authority over the language, and tsrs aims to reproduce the official compiler faithfully.
Why now: AI-assisted porting
Cloudflare's answer to "why now" is that frontier AI models can now carry out large-scale ports "almost independently in a matter of days." It notes a recent wave of TypeScript ports to Rust and C++. It describes them as "slopforks" (careless AI-generated forks), faithful ports and full re-architectures, and reads the wave as evidence of demand for a faster checker. The article implies, but does not spell out, how much of tsrs itself was written by AI.
Results
The headline benchmark compares tsrs with TypeScript 7.0 using the same number of 5, the parallel threads that check files. On that basis tsrs is up to 8x faster and uses about 70% less memory. Cloudflare says it worked with the TypeScript team to make sure both sides ran the same configuration. It contrasts this with other published port benchmarks that, it says, inflated their results with larger machines or non-default options.
Two more data points:
TypeScript 7.1 beta is up to 2.5x faster than 7.0. Cloudflare says some of those optimizations came from tsrs, and it plans to keep contributing performance work back to the Go codebase.
tsrs's default mode adapts to the machine, trading more memory for even more speed. The article shows this in a chart but gives no figure in the text.
The text does not say which projects were benchmarked or on what hardware. The "up to" framing means typical gains may be smaller.
Why Rust, if Go is already fast?
Cloudflare argues the point is not mainly speed. It is about 6, a compact bytecode that runs in browsers and in sandboxed server runtimes. Go manages memory with a 7, a runtime component that periodically finds and frees unused memory. Compiling Go to Wasm means shipping that collector inside every module. Rust frees memory at points fixed when the program is compiled, so it needs no collector. Cloudflare says this also lets a Wasm module and the program hosting it exchange memory simply.
An analogy (ours, not the article's): a Go Wasm module is like a guest who brings their own cleaning crew to every room they stay in. A Rust module tidies up as it goes and needs no crew.
The target is Cloudflare 8, its serverless platform that runs code in sandboxes across its data centers. Workers has long supported Wasm, and Cloudflare recently added native Rust support through a new Emscripten target for wasm-bindgen, the tool that connects Rust code to JavaScript hosts. When build and deploy systems run in tight loops, a small memory footprint saves money quickly. Cloudflare's stated aim is to run a whole development toolchain, end to end, in a sandbox that AI agents can drive. The same Wasm build could also run client-side in a browser, or on embedded and IoT (internet-connected) devices.
Keeping the port faithful
Maintenance has two parts. First, Cloudflare is working with Theo Browne to merge tsrs with a similar project, tsc-rs, into one package covering all of TypeScript's features. That includes the 9, the standard way editors get errors, completions and go-to-definition from a language tool. The article also mentions "content mappers" but doesn't explain them.
Second, it will reuse the agent workflow it runs for 10, its reimplementation of the Next.js web framework:
Each morning an agent reviews upstream changes, fetches the diffs and opens tracking issues for anything that could affect the port.
Each night the upstream test suite runs against the port and regenerates a compatibility matrix.
When a test or issue reveals a gap, agents find the matching change in both codebases, build a reproduction, port the relevant tests and propose a fix.
Cloudflare promises "extreme care" not to introduce behavior that diverges from upstream.
Trade-offs and open questions
Known differences remain. One example: tsrs prints diagnostics (errors and warnings) in a different order from upstream. Cloudflare aims to minimize such differences before a stable release.
A second implementation must track a moving target. The agent pipeline is promising, but it rests on upstream's test suite catching every behavior difference. For a type checker, that suite is extensive but not exhaustive.
Upstream is catching up. TypeScript 7.1's 2.5x gain narrows the gap, and the TypeScript team is experimenting with its own memory improvements. tsrs's lasting advantage may be Wasm portability more than raw speed.
Integration plans. The Oxc team is refactoring the code for speed and maintainability. Cloudflare wants to use tsrs for type-aware lint rules in Oxlint, its Rust linter, which need type information that a plain linter lacks.
Remember
The fair comparison is with the same worker count: up to 8x faster and ~70% less memory than Go-based TypeScript 7.0.
Rust was chosen mainly for WebAssembly, because Go's Wasm builds must bundle a garbage collector.
Faithfulness depends on nightly upstream test runs plus AI agents that triage diffs and propose fixes.
A common way to split training is to make several full copies (replicas) of the model. Each copy trains on different data, and the copies average their updates. Within one replica, traffic is heavy and sensitive to delay. Between replicas, traffic is lighter and can tolerate more delay.
Chart of GB200 interruption rates, showing the trend behind Meta's reported 99% reduction.1 / 4
Meta has published the engineering behind Prometheus, its AI training supercluster in New Albany, Ohio. The first phase is finished and trained Meta's Muse family of models. It already holds hundreds of thousands of 1 (the graphics processors used for AI math), and Meta plans to grow it to about 1 million, all running as one machine.
The real problem: keeping it running
When thousands of GPUs train one model in lockstep, one bad part can stall them all. Meta tracks this with two numbers. 2 is the share of time spent on useful training. 3, mean time between interruptions, is measured in GPU-hours. In mid-2025, Meta saw nearly 1,000 interruptions per 8,000 GPUs per day. After a year of work it reports a 99% reduction. Pre-training of its Muse Wonder model ran at over 85% goodput, and reinforcement-learning runs (a later training stage) at over 90%. MTBI has since passed 54,000 GPU-hours. The goal is about 100,000.
Bigger racks, bigger blasts
Meta uses NVIDIA's 4 systems, where 72 GPUs share 5, NVIDIA's fast chip-to-chip links, and act as one large accelerator. The previous generation tied together only 8, so one failure now hits nine times as many GPUs. Meta's answer is to run training on 64 GPUs per rack and keep 8 as spares in the same rack. If a GPU dies, a spare takes over and the job resumes from its last save without waiting for a repair. Meta also holds about 2% extra racks for whole-rack failures. A job using 32,768 GPUs needs 523 racks in total, about 13% overhead.
When a fault stays inside one copy of the model, a mechanism called 6 (Parallelism-Aware Fault Tolerance) drops just that copy while the others keep training. Larger faults roll the whole job back to a 7, which is saved in the background.
Where the failures came from
NVLink signal integrity: each rack has 5,184 links running at 200 Gbps, and the cluster has over 600,000 NVLink connections. Weak links are found with stress tests before they carry real work.
Memory errors: these grew on the newer GB300 chips. Meta and NVIDIA fixed them with firmware changes and by retiring chips that showed early warning signs.
Firmware rollout: fixes now reach more than 100,000 GPUs in a few hours. The slowest-10% upgrade time for one rack group fell from over 7 hours to 2.
Network and power
Buildings up to 100 km apart are joined by a dedicated Ethernet network. Training traffic is arranged so the delay-sensitive parts stay within a building. When thousands of GPUs start or stop together, power draw jumps by 30–40%. Software that gradually lowers GPU power caps, plus batteries, smooths out those swings. The article is a vendor-friendly showcase, and it does not report how much useful training time the remaining interruptions still cost.
Meta's engineering blog has published a detailed look at Prometheus, its gigawatt-scale AI training supercluster in New Albany, Ohio. Two years ago the site ran ordinary web traffic. Now it holds hundreds of thousands of NVIDIA 1, the processors that do the matrix math of AI training, spread across several buildings and wired together to act as one machine. Phase one is finished. It trained Meta's Muse foundation models, including the frontier "Muse Wonder" models, and Meta plans to grow the cluster to about 1 million GPUs. The most useful part of the post is about reliability, not raw compute.
Why reliability is the whole game
In synchronous training, every GPU finishes a step and swaps results before any can start the next. One failing host can stall or crash the whole job. Meta tracks two numbers:
2: the share of time spent on productive training.
3 (mean time between interruptions): measured in GPU-hours. An MTBI of 50,000 GPU-hours on a 10,000-GPU job means a stall roughly every five hours (illustrative arithmetic).
In mid-2025 the cluster saw a peak of nearly 1,000 interruptions per 8,000 GPUs per day. Meta reports a 99% reduction after a year of work. Muse Wonder pre-training reached over 85% goodput, and its reinforcement-learning runs (a later training stage) over 90%. MTBI was about 47,000 GPU-hours during those runs and is now above 54,000. The stated goal is two interruptions per 8,000 GPUs per day. That works out to roughly 100,000 GPU-hours (8,192 × 24 ÷ 2 ≈ 98,000).
The hardware generations
Meta builds its own rack designs around NVIDIA chips:
The 72-GPU failure domain
In the H100 generation, NVLink joined only 8 GPUs. NVL72 joins 72, so one fault can now take out nine times as many GPUs. Training splits work most cleanly into groups sized as powers of two, so Meta runs 64 GPUs per rack and keeps 8 as in-rack spares. When a GPU fails, a spare takes its place. The job restarts from its last 9, and the rack stays in the job instead of waiting for a technician. For rarer whole-rack failures, Meta keeps about 2% extra racks in reserve.
Worked example from the article: a 32,768-GPU run needs 512 racks of 64 active GPUs, plus spares, for 523 racks or 37,656 GPUs. That is about 13% above the active count. Meta points out that 11% of the margin comes from the 64-of-72 layout itself, not from reliability spares.
The physical wiring is part of the risk. One NVL72 domain has 5,184 differential signal links running at 200 Gbps. Each link passes through GPU circuitry, circuit boards, cable cartridges and switch connectors, and any of these can degrade.
Two recovery paths
10 (Parallelism-Aware Fault Tolerance): training runs as several data-parallel replicas, meaning full copies of the model that each train on different data. If a fault stays inside one replica, only that replica goes offline. The others repeat the step and continue, and the damaged replica restarts on spare hosts and catches up. A full halt becomes a brief dip in throughput.
Checkpointing: larger faults roll the whole job back. Checkpoints save model weights, gradients and optimizer state in the background to tiered storage. Meta uses delta checkpoints (saving only what changed) and stages them in memory first, so training barely slows.
Breaking down the 99%
Meta classified every interruption by root cause and shared the triage with NVIDIA, whose staff work on site. NVIDIA owns firmware, signal diagnostics and memory handling. Meta owns scheduling and infrastructure. The main categories:
NVLink signal integrity, the top problem on GB200. Across more than 600,000 NVLink connections, small amounts of signal degradation set off frequent link-recovery cycles. The fixes: continuous telemetry, stress tests that catch weak links before production, tools that separate passing glitches from persistent faults, and a validation step before repaired systems return.
Memory (HBM and the CPU's LPDDR), which grew with GB300's larger memory. The fixes: HBM firmware tuning, memory scrubbing, better error correction, and retiring chips whose error patterns predict failure. Meta says it is still measuring the impact.
NVLink switches, where one bad switch can take down a whole rack domain. Health is monitored at millisecond resolution, and traffic is rerouted automatically.
Back-end network links, rerouted in hardware across parallel network planes.
Fixing things quickly mattered as much as finding them. Meta can push firmware to more than 100,000 GPUs in a few hours. The 90th-percentile upgrade time for one pod (a group of racks) fell from over 7 hours to 2.
Network across 100 km
Alongside the normal data-center network, Meta runs a dedicated Ethernet 11 used only for GPU-to-GPU traffic. Each building fabric connects up to 20,736 GPUs. It began as a "scheduled" fabric, where switches spread packets across paths and reorder them. That design avoids collisions between the few large, long-lived flows that training produces. Once network cards could reorder packets themselves, Meta moved to simpler 51.2 Tbps switches running its own 12 switch software. The network links the buildings over 2 to 100+ km using deep-buffered Arista switches, with several petabits per second between region pairs. Meta places delay-tolerant traffic between replicas on these long links and keeps delay-sensitive traffic inside buildings. It also needed dedicated tuning to cope with speed-of-light delay. In total the cluster uses millions of miles of fiber and millions of optical transceivers.
Power and cooling
When tens of thousands of GPUs switch between computing and communicating at the same moment, cluster power swings by 30–40%. Starting or stopping a job can move tens of megawatts in seconds. Software called 14 gradually lowers GPU power caps as the cluster nears its electrical limit, instead of halting jobs. Rack batteries, capacitor trays and facility battery storage absorb the spikes. Cooling varies by building: air-assisted liquid cooling in older and rapid-deployment buildings, and facility liquid cooling in the new AI-specific building.
What's left open
This is a company showcase, written partly to promote the Muse models. Its claims are not independently checked. It does not say how many GPUs were in each Muse run, how much wall-clock time recovery cost, or how much the 100 km BAG links slow pre-training. Some numbers also don't line up cleanly: a 99% cut from 1,000 interruptions per day still leaves about 10 per 8,000 GPUs per day, while 54,000 GPU-hours MTBI implies fewer than 4. That suggests the figures come from different jobs or time windows. Meta says the lessons feed into Hyperion, its next supercluster in Louisiana.
Remember
Moving from 8-GPU to 72-GPU NVLink groups makes each failure hit nine times as many GPUs.
Running 64 of each rack's 72 GPUs leaves 8 spares, so failures need no repair wait. Total overhead is about 13%.
Most early stalls came from weak NVLink signals and memory errors, found by tracing every interruption to a root cause.
Agents are nondeterministic: the same task can pass on one run and fail on the next. Benchmarks therefore run each task several times. The average success rate measures capability. Pass^k, the share of tasks solved on all k attempts, measures whether you can trust the agent every time.
Accuracy and Pass^4 by setting and domain: reliability gains outpace average-accuracy gains.1 / 3
IBM Research's ALTK-Evolve took a plain customer-service agent running GPT-4.1 to first place on the retail leaderboard of 1, a customer-service agent benchmark. GPT-4.1 is an older OpenAI model with no extended reasoning mode. The agent scored 67.5% 2. The model was never retrained. Its only change was a list of rules, which the agent wrote itself, appended to its 3.
The benchmark
τ²-bench, from the company Sierra, pairs an agent with a customer played by another 4 (large language model). The customer has a hidden goal. The agent has tools, such as refunds and flight rebooking, and a written policy it must follow. A task passes only if the final database state is exactly right. Pass^4 runs each task four times and counts it only if all four runs succeed. It measures whether an agent is reliable every time, not just on average. The tests cover the retail and airline domains.
How the playbook is built
The agent is the benchmark's stock 5 agent, a loop that alternates thinking and taking one tool action. It first attempts a set of training tasks. The same model, GPT-4.1, then reads those transcripts and distills short rules. Examples: authenticate by name and zip code if there's no email, and get explicit approval before changing the database. No human labels anything. Each rule has a 6, the number of training tasks it came from. IBM compared three settings:
no rules
sup3, only rules with support of 3 or more (31 retail rules, 14 airline)
7, every rule (292 retail, 101 airline)
Results
These results are on held-out test tasks: 40 retail and 20 airline. In the best setting for each domain, Pass^4 rose 10 points:
Retail (lossless): 57.5% → 67.5%
Airline (sup3): 35% → 45%
Average accuracy rose 4.4 points in retail and 7.5 in airline. The two domains wanted different amounts of guidance: retail needed every rule, while airline was most reliable with its 14-rule core.
The cost is input tokens. Retail with lossless rose from $0.084 to $0.139 per conversation (+65%). Conversations didn't get longer: on tasks solved both with and without rules, the rules trimmed about 0.3–0.4 agent turns. IBM argues 8 can recover much of the extra input cost.
With GPT-5.5, a stronger model, on airline, the rules lifted Pass^4 from 55% to 70%. On the full leaderboard task set, that agent scored 82%, ahead of every frontier-model entry (66–72%). On retail, GPT-4.1 beat RAFT-30B-A3B, a model fine-tuned on retail data, by 6.1 points.
Caveats
The test sets are small. In airline, one task is worth 5 points of Pass^4. The leaderboard score also includes the training tasks the rules were learned from.
IBM Research reports that ALTK-Evolve, its system for giving AI agents learned memory, lifted a plain agent built on GPT-4.1 to the top of the retail leaderboard on 1, a customer-service benchmark. GPT-4.1 is an older OpenAI model with no extended reasoning mode. The agent reached 67.5% 2, beating every retail entry in the October 12, 2026 snapshot. That includes RAFT-30B-A3B, a model fine-tuned on synthetic τ²-bench retail data, which scored 61.4%. The model's weights never changed. The only change was a list of behavioural rules appended to the 3, and the agent wrote those rules itself from its earlier attempts.
The problem: policy-bound conversations
τ²-bench, built by the company Sierra, pairs an agent with a customer played by another 4 (large language model). The simulated customer has a hidden goal. The agent gets domain tools, such as order lookup, refunds and flight rebooking, plus a written policy it must follow. Grading is strict. A task counts as solved only if the final database state matches the expected one, and many actions, like a refund, can't be undone. The agent has to draw out the right details from the customer, stay within policy and call the right tools in the right order. IBM uses the benchmark's two customer-service domains, retail and airline.
Consistency is the hard part. A model can handle a return correctly three times in four, and that still leaves a quarter of real customers let down. For that reason the leaderboard reports Pass^4: each task runs four times and counts only if all four succeed. IBM also reports Mean Accuracy, the average success rate over those four runs, as its measure of capability.
How ALTK-Evolve builds a playbook
The setup is deliberately stock. It uses τ²-bench's own 5 agent, a loop that alternates THINK and ACT steps, with GPT-4.1 as the agent and GPT-5.5 playing the customer. ALTK-Evolve runs the agent on the benchmark's training tasks. GPT-4.1 then reads the successful and failed transcripts and distills them into short rules. The same model plays both roles, and no human labels anything. The rules are appended after the policy as a markdown list, then applied unchanged to held-out test tasks. Examples include:
Authenticate the user before any order action. If there's no email, verify by full name and zip code.
Summarise intended changes and get explicit approval before any write to the database.
Never combine a tool call and a message to the user in one turn.
Split a handle like daiki_johnson_9523 into first and last name before looking the user up.
Each rule has a 6: the number of distinct training tasks it was extracted from. A high count suggests the rule generalises; a count of 1 may be a one-off. IBM tested three settings:
baseline: no rules.
sup3: only rules with support of 3 or more. That left 31 rules for retail and 14 for airline.
7: every rule kept after merging duplicates. That left 292 for retail and 101 for airline.
Results on held-out tasks
The test sets are 40 retail tasks and 20 airline tasks. In the table, "pp" means percentage points.
Domain
Config
Mean Accuracy
Pass^4
Retail
baseline
78.1%
57.5%
Retail
sup3
78.8%
60.0%
Retail
lossless
82.5% (+4.4 pp)
67.5% (+10.0 pp)
Airline
baseline
57.5%
35.0%
Airline
sup3
60.0%
45.0% (+10.0 pp)
Airline
lossless
65.0% (+7.5 pp)
40.0%
Reliability improves more than average accuracy. The two domains also disagree about how much guidance to give. Retail needs the full 292-rule set for a real gain. In airline, the 14-rule core is the most reliable setting even though lossless scores higher on accuracy. IBM concludes that the support threshold should be tuned per domain, not fixed in code.
What it costs
The rules make the prompt longer, and that longer prompt is sent on every call. Cost per conversation, counting both the agent and the simulated customer:
Retail: $0.084 → $0.095 with sup3 (+12%), and $0.139 with lossless (+65%).
Airline: $0.081 → $0.097 with sup3 (+19%), and $0.107 with lossless (+32%).
Conversations did not get longer. IBM compared only runs that both the baseline and the guided agent solved, on the same scenario and random seed. On those, lossless trimmed the conversation from 10.59 to 10.16 agent turns in retail and from 9.15 to 8.82 in airline. No setting added turns. Each turn is a sequential model call, so fewer turns means lower latency. The rule block is identical on every call, so IBM argues 8 can recover much of the extra input cost. The article gives no numbers for the cached case.
Does it help a stronger model?
One objection is that the rules might only patch a weak model's gaps. So IBM reran airline with GPT-5.5 as both the agent and the rule extractor, mining a fresh set of rules. No rule reached support 3, so the high-confidence core became sup2.
That is a 15-point reliability gain from a higher starting point, larger than GPT-4.1's 10 points on the same tasks. IBM also notes that the baseline's Pass^4 varied between reruns, while the lossless setting reproduced exactly.
Leaderboard placement
τ²-bench requires leaderboard submissions to be scored on the full "9", meaning training and test tasks together. These figures therefore differ from the held-out ones above.
Retail, GPT-4.1: 67.5%, ranked first.
Airline, GPT-4.1 with sup3: 44%, ranked eighth. That beats GPT-5.2 without reasoning and nearly matches Claude Sonnet 4.5.
Airline, GPT-5.5: 82.0%, ahead of every frontier-model entry (66–72%).
Caveats
Small samples. In airline, one task out of 20 is worth 5 points of Pass^4, so several of the differences above are just one or two tasks.
Training tasks in the leaderboard score. The base split includes the training tasks the rules were mined from. IBM reports no test-set leakage, and its held-out numbers support the gains. Still, the leaderboard figures blend in tasks the agent effectively studied.
Simulator not compared. The article doesn't say whether other leaderboard entries used the same GPT-5.5 customer simulator, which could affect comparability.
Two domains only. The study doesn't test whether a 292-rule prompt holds up in domains with more tools or longer policies.
The method is open source at github.com/AgentToolkit/altk-evolve. IBM's argument is that a self-built playbook can beat fine-tuning for this kind of work, because it is cheap and automatic and can be rebuilt for each new domain.
Remember
The rules raised Pass^4, the all-four-runs reliability score, more than they raised average accuracy, in both domains.
The extra cost was input tokens (up to 65% more), not more turns; conversations got slightly shorter.
The best number of rules varied by domain, so the support threshold needs tuning per domain.
Inference means running a trained model to produce output. Serving it cheaply at scale takes specialized software that batches requests, manages memory and runs fast GPU kernels. vLLM and SGLang are the main open-source serving engines. Big labs are known to run faster tools internally that they keep private.
Performance versus price on the DeepSWE coding benchmark, where DeepSeek models sit on the frontier.1 / 4
DeepSeek, the Chinese AI lab, has publicly released DeepInfer, the in-house software it uses to serve its own models. Florian Brand, writing in Interconnects' roundup of 1 AI, calls it one of the biggest open-source AI artifacts of the past few months. Open-weight models are ones whose trained parameters anyone can download.
Why serving software matters
Running a model to produce answers is called 2. Its cost depends heavily on serving software, which batches requests and squeezes performance out of GPUs. The common open-source engines are 3 and 4, which are general-purpose. Brand expects DeepInfer, built for DeepSeek's models, to be far more efficient on them. He cites estimates that DeepSeek earns up to 90% 5 (revenue minus serving cost) on its tokens.
Big labs are known to keep better tools private, and Brand says this is the first time a leading lab has shared its own. Hosting companies will adopt it, so he expects DeepSeek-model prices to fall, or providers' margins to rise. The article gives no benchmarks for DeepInfer itself.
Governments move in
United States: The Department of Homeland Security barred DeepSeek's company and models from federal civilian procurement. It's unclear whether this covers models trained on DeepSeek outputs, such as NVIDIA's Nemotron.
China:6, the Ministry of Commerce, proposed requiring pre-approval before frontier AI is shared abroad. Brand calls the reaction overblown, since open-source releases would be approved automatically. Still, it creates a framework for future control.
Notable models
Sizes like 552B-A16B describe 7 models: 552 billion parameters in total, about 16 billion used per token.
Kimi K3.1 is rumored to be imminent.
Interconnects publishes a regular roundup of 1 AI releases. Open-weight models are ones whose trained parameters anyone can download and run. In issue #25, Florian Brand argues that the biggest open-source artifact of recent months isn't a model at all. It is DeepInfer, the 2 software DeepSeek uses to serve its own models, which the company has now released publicly. The release lands in a week when governments in both Washington and Beijing moved toward controlling open models.
The regulatory backdrop
Three items set the scene.
China.3, China's Ministry of Commerce, published a policy proposal. It is not yet law. Under it, frontier AI models and some other technologies would need government pre-approval before being shared internationally. Brand calls the reaction "overblown", because the document says open-source technologies would get automatic approval. He notes that geopolitics watchers still see it as significant: it creates a framework that could be used to regulate China's AI industry later. Other sources he trusts think the real target may be deals like the sale of Chinese startup Manus to Western companies, not model releases.
United States. The Department of Homeland Security (DHS) issued an order that bars DeepSeek, both the company and its models, from being a vendor in federal civilian procurement. Brand points out an open question the order doesn't settle. What happens to models fine-tuned from DeepSeek, or trained on its outputs? NVIDIA's Nemotron models, for example, have long leaned heavily on DeepSeek models in their training recipes.
Existing controls. Zhipu AI, the maker of the GLM models, is already on the 4. That list is run by the Commerce Department's Bureau of Industry and Security (BIS) and restricts access to American technology.
Brand reads the DHS move as expected. He says it strengthens the case for a domestic US open-model ecosystem.
Why an inference stack matters
Training a model is only half the cost. Every response has to be generated on GPUs, a step called inference. How cheaply that runs depends on serving software that batches many users' requests together, manages GPU memory and uses fast low-level kernels. The standard open-source options are 5 and 6. They are general-purpose engines that support many model architectures.
DeepInfer is built for DeepSeek's own models, so Brand expects it to be "far more efficient" on them than vLLM or SGLang. The article cites outside estimates that DeepSeek makes up to 90% 7 selling tokens on this stack. Gross margin is revenue minus the direct cost of serving. The article doesn't give benchmarks for DeepInfer, describe how it works internally, or say where the 90% figure comes from.
Brand's point is about who benefits. Top labs are known to have inference tools much better than the public ones, and he says this is the first time a leading lab has shared its own. Third-party inference companies, which host open models and sell API access, can adopt it. He expects one of two results: prices for DeepSeek models fall across providers, or providers keep prices and widen their margins.
As an illustration (not from the article): if a provider's cost to serve a million tokens dropped by half while competitors could do the same, competition would push the price down toward the new cost.
The model picks
DeepSeek-V4.1-Flash-1017. DeepSeek shipped two iterations since the last roundup, at a new size: 552B-A16B. That means 552 billion total parameters, about 16 billion active per token, in a 8 design. It also accepts image inputs and keeps the MIT license. Brand calls that notable at a time when licenses are becoming "more bespoke and exclusionary". On 9, which the article calls a favorite agentic software-engineering benchmark, DeepSeek dominates the Pareto frontier of performance against price. The Pareto frontier is the set of models that no other model beats on both score and cost at once.
MiMo-V2.6-Pro-RL (Xiaomi). This model reuses the base models from MiMo V2.5 but puts far more compute into 10 (RL). RL is post-training in which the model attempts tasks and gets rewarded for good outcomes. Xiaomi released a detailed technical report and some of its RL environments, which Brand says is rare at this scale. It also ran a live training dashboard. The final V2.6-Pro is currently the top open model on Artificial Analysis, an independent benchmarking site.
Beam-1.0-501B-A23B (Reflection). Brand calls it the strongest open American model under a permissive license (Apache 2.0). It isn't the smartest open model, but it is very token-efficient: it reaches its answers using few output tokens, second only to GPT-6-Astra on the article's chart. Brand reads this as a sign of a mature post-training stack. Fewer tokens per answer means lower cost and latency.
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4. This is the mid-sized Nemotron update, with vision support (VL means vision-language). 11 is NVIDIA's 4-bit floating-point format, which shrinks the model enough to run on a high-end laptop or an RTX Spark. As usual, NVIDIA released most of the training data and a detailed report.
clef (Cloudflare). This is a "decision model", a category the article says the startup Typesafe introduced. A decision model is a general classifier that outputs a few tokens along with their probabilities. You get a label and a confidence score, not free-form prose. Many open fine-tunes of the idea have followed.
Other releases
Ternary-Bonsai-2-27B (prism-ml): a ternary version of Qwen3.8-27B, meaning each weight is restricted to roughly three values (−1, 0, +1). It ships in 12, a file format used by local runtimes such as llama.cpp, and targets on-device use.
Ornith-1.5-397B (ornith-ai): fine-tunes of Qwen3.5 and Gemma 4 that local-AI users noticed for beating those already strong base models.
Ling-3.0-flash-VL-fp4 (inclusionAI, part of Ant Group): a 124B-A5.5B model.
Muse-Spark-Open-816B-A42B (Meta): an open version of Meta's Muse model.
Kimi K3.1 is rumored to be close, with its metadata already appearing in public libraries.
What stays open
The article leaves several questions unanswered. It doesn't measure DeepInfer against vLLM or SGLang, or say whether it supports any models besides DeepSeek's. It doesn't settle whether the DHS order covers derivatives such as Nemotron. And the MOFCOM text is only a proposal. The source is also cut off partway through its list of releases, so this summary doesn't cover the whole issue.
Remember
Serving software is a competitive moat: DeepInfer reportedly lets DeepSeek earn up to 90% gross margin on tokens.
A US federal ban on DeepSeek leaves it unclear whether models trained on DeepSeek outputs are covered.
Token efficiency is now a headline metric: Reflection's Beam ranks well by using few tokens per answer.
Updates ship for Cloudflare One Client / WARP (its VPN-like client app, also called 1.1.1.1) on all platforms except Cloudflare One Agent on iOS, which awaits Apple's app review
Oct 20, 09:15
Update for 7 (Cloudflare Tunnel, a daemon that connects private servers to Cloudflare)
Oct 20, 19:20
Fixes fully deployed across the CDN (content delivery network) and server-side Zero Trust products
Catalina (GB200 4): 72 Blackwell GPUs joined by 5, NVIDIA's fast chip-to-chip links, so they act as one accelerator. Meta chose the two-rack "Ariel" variant, with one Grace CPU per GPU and two 400G network cards per GPU, which gives 800 Gbps of 6 bandwidth (traffic leaving the rack). Meta's reasoning: compared with the older H100, GB200 raised compute, memory bandwidth and NVLink size, but standard scale-out stayed at 400G. Meta expected that mismatch to cost performance. Specs: 13.4 TB of 7 (high-bandwidth memory stacked next to the GPU), 720 PFLOPS at 8 (an 8-bit number format), 130 TB/s NVLink bandwidth, and about 120 kW across the two racks.
Clemente (GB300 NVL72): 288 GB of memory per GPU (1.5x), roughly double the acceleration for attention, and 800G networking built in. The whole 72-GPU domain fits in one rack at about 135 kW. Meta reports 5–10% better performance than GB200.
San Miguel (Vera Rubin NVL72), coming next: 288 GB of HBM4 per GPU, 2.52 EFLOPS at FP4 per rack, 216 TB/s NVLink, over 170 kW per rack, and a claimed 10–20% better performance per watt than GB300.
13
DeepSeek-V4.1-Flash-1017 (552B-A16B, accepts images, MIT license): it leads on performance per price on 8, an agentic coding benchmark.
Xiaomi MiMo-V2.6-Pro: the same base model as before, but with much more 9 (training by reward on tasks). It is now the top open model on the Artificial Analysis rankings, and Xiaomi released its technical report and some training environments.
Reflection Beam-1.0 (Apache 2.0): the strongest permissively licensed American open model. It isn't top-scoring, but it is very token-efficient, reaching answers with few output tokens.
NVIDIA Nemotron-3.5-Super (120B-A12B): uses 10, NVIDIA's 4-bit number format, and runs on a high-end laptop. NVIDIA released most of its training data.
Cloudflare clef: a "decision model", a classifier that outputs a few tokens with probabilities.