Give the Noisy Container Its Own Machine · Morning Gist
Friday, October 235 stories · 16 min
Give the Noisy Container Its Own Machine
Netflix isolates containers with plain VMs, AI models find PHP engine bugs, Microsoft trains agents to succeed every time, and two infrastructure layers are being built for agents.
Picked from 45 new articles across 23 sources · arrow keys for other days
Today in 30 seconds
30s left
Netflix now gives noisy or sensitive containers their own right-sized EC2 VM. It's a reversible label, a simpler fix than microVMs or gVisor. story 1 ↓
GPT-5.6 Sol (used by Calif) and Gemini (used by Wiz) each found the same two critical PHP engine bugs. Coordinating the fixes took two months. story 2 ↓
Microsoft trained a 27B Qwen model with database-state rewards. Tasks it passed on all 20 attempts rose from 38 to 107, while tasks solved at least once fell slightly. story 3 ↓
Cloudflare turned on WebMCP agent tools at the edge for Commerce.com's 100,000 store domains. Each merchant can opt out. story 4 ↓
Modal's CEO says GPU contracts don't fit how AI workloads run. Code sandboxes, a product Modal nearly shut down, now bring in about a third of revenue. story 5 ↓
Anthropic is donating Agent Skills to the Agentic AI Foundation. More than 70 platforms, including Cursor, Copilot and VS Code, already support it.
1/5Netflix Tech Blog· Netflix Technology Blog· Oct 22
Netflix gives noisy or sensitive containers their own virtual machine
Netflix added a mode to Titus, its in-house container platform, that runs one container alone on its own right-sized cloud machine. This stops interference from neighbors that per-container limits can't block, and the packing of the remaining shared machines can improve.
Why it matters · It shows where container isolation physically runs out, and why a plain virtual machine can be a cheaper fix than microVMs, gVisor or more kernel tuning.
Know this first
Context from the editor, not the article · 37 sec
1 · Multi-tenant container hosts
Container platforms pack many containers from different teams onto one large machine. That keeps the machine busy, which saves money. But all of those containers share one Linux kernel (the core of the operating system) and the same physical CPU caches, memory bus, disks and network card.
2 · How containers are fenced
Linux limits containers using cgroups, which cap each one's CPU time, memory and disk I/O, and namespaces, which give each container its own view of processes and files. These are software quotas. They can't split up hardware that is shared at a lower level, such as the CPU's last-level cache or memory bandwidth.
3 · Kubernetes placement controls
2/5Latent Space· Oct 22
AI models found two critical bugs in PHP's core engine
Two security teams using frontier AI models separately found memory-corruption bugs inside PHP itself. Attackers can use them to take over WordPress, Drupal and Joomla servers without logging in. The fix shipped October 21 after two months of private coordination.
Why it matters · A model-assisted find in a core language runtime, with working exploits, shows these models now do real security work. It also shows the bottleneck has moved from finding bugs to deploying fixes.
Know this first
Context from the editor, not the article · 41 sec
1 · PHP and its reach
PHP is a server-side language that runs inside web servers and builds pages on request. WordPress, Drupal and Joomla are content management systems (CMSs) written in it, and together they power a large share of websites. A flaw in PHP's engine reaches every app built on it, so a plugin update alone can't fix it.
2 · Memory corruption and RCE
PHP's engine is written in C. Mistakes in C code can let attackers read or overwrite memory they shouldn't touch. That can leak secrets, or escalate to remote code execution (RCE): running the attacker's own commands on the server. "Pre-authentication" means the attacker doesn't need to log in first.
3 · Coordinated disclosure
3/5Hugging Face· Oct 22
Microsoft turns database-state grading into rewards for training AI agents
Microsoft and Hugging Face used ThinkingBox, a test suite that grades agents by the database records they leave behind, as a training reward. A 27B Qwen model became far more consistent, with slightly narrower reach.
Why it matters · For agents that write to production records, repeatability matters more than average accuracy, and this shows a cheap, checkable way to train for it.
Know this first
Context from the editor, not the article · 40 sec
1 · Agents and tool calls
An AI agent is a language model that acts by calling tools, such as "look up booking" or "issue refund", in a loop until it decides it is done. It can report success while leaving the wrong data behind. Grading only its final message or its sequence of calls can miss this.
2 · Reinforcement learning post-training
After pretraining, a model can be improved with reinforcement learning (RL): it attempts tasks, a reward function scores each attempt, and its weights shift toward the higher-scoring behavior. RL works best when rewards are cheap, automatic and hard to game, which is why code tests and math answers became popular reward sources.
3 · pass@k metrics
4/5Cloudflare· Jenny Lea· Oct 21
Cloudflare turns on WebMCP agent tools across Commerce.com's 100,000 store domains
Cloudflare added a script at its network edge that gives AI shopping agents named functions, like "search products" and "add to cart", on nearly 100,000 Commerce.com stores. Agents no longer have to guess which button to click.
Why it matters · It shows one likely path for making existing websites usable by agents: add a structured tool layer at the proxy, without rewriting any front-end code.
Know this first
Context from the editor, not the article · 40 sec
1 · How agents use websites now
Most AI agents that act on websites read the raw HTML or screenshots, guess which element does what, click, and retry when they're wrong. This burns tokens (the units of text a model processes, which cost money). It also breaks easily: a redesign, a cookie banner or a bot check can stop it.
2 · MCP and tool calling
The Model Context Protocol (MCP) is an open standard for describing tools to an AI model. Each tool has a name, typed inputs and an output, such as search_products(query, max_price). The model then calls the function directly instead of driving a user interface. Usually an MCP server runs on a backend machine.
3 · Cloudflare's edge and zones
5/5Latent Space· Basil Chatha· Oct 22
Modal's CEO on why GPU contracts don't fit how AI workloads actually run
Modal CEO Erik Bernhardsson explains how his company rents GPUs on long fixed contracts and resells them in small, short slices. Combining thousands of customers' uneven demand makes that work, and fast startup makes it practical.
Why it matters · If agents start running every command or test in a throwaway sandbox, the time it takes to start an environment decides what kinds of software are practical to build.
Know this first
Context from the editor, not the article · 43 sec
1 · Serverless compute
In serverless computing you hand the platform your code and it starts machines only when requests arrive, then stops them afterward. You pay for the seconds used, not for idle servers. The hard part is a cold start, the delay before a fresh copy of your code is ready. AI models make cold starts worse because they are gigabytes in size.
2 · How GPUs get bought
GPUs are the chips that run AI models, and they are expensive and scarce. Big buyers usually rent them through reservations: a fixed number of GPUs for a fixed term, often years, sometimes paid up front. Spot capacity is cheaper leftover capacity that the cloud provider can take back at short notice.
Kubernetes schedules small groups of containers, called pods, onto machines, called nodes. Anti-affinity rules keep certain pods apart. A taint on a node turns pods away unless they explicitly tolerate it. Autoscalers add or remove nodes to match the pods waiting to run.
Two containers within their CPU limits still contend for the unfenced shared cache and memory bandwidth.1 / 4
Netflix runs most of its services on 1, its in-house container platform built on Kubernetes (the standard open-source system for scheduling containers onto machines). By default, containers from many teams share each 2 instance (a rented virtual machine in Amazon's cloud). Netflix has added a single-tenant mode. An eligible task gets its own right-sized EC2 machine, but keeps the same Titus API and deployment path, and Netflix's platform team still owns and patches the host.
The problem quotas can't fix
Linux fences containers with 3 (caps on CPU, memory and disk). But some hardware sits below anything a cgroup can reach: the CPU's 4, memory bandwidth, and the kernel's own locks. A neighbor streaming data from RAM can evict your cached data and slow you down while both of you stay within quota.
Netflix says the network hurt most. Its bandwidth shaper, 5 (hierarchical token bucket), serializes traffic behind a global lock. One container flooding the host with small packets could starve its neighbors and cause latency spikes. Stateful streaming jobs such as Apache Flink suffered worst: falling behind can mean restoring from a checkpoint and pausing for minutes.
Why a plain VM won
Netflix rejected four alternatives:
More kernel tuning, which narrows the gap but can't close it.
6 (Firecracker, Kata). These need bare-metal hosts, mean running a self-managed hypervisor, add overhead and don't support GPUs.
7, a user-space kernel. It slows I/O-heavy jobs.
Plain EC2, which pushes patching and scaling back onto app teams.
The authors argue that microVMs and gVisor defend against hostile tenants. Netflix only needs performance isolation, and a dedicated VM gives that without any emulation.
How it works
Titus's federation layer, which decides where jobs run, checks an allowlist and tags eligible tasks. The autoscaler then boots the smallest instance that fits the task. The scheduler keeps the node to one task: other pods are kept away (8), unrelated work is repelled (taints), preemption is off, and the task is never moved to repack the fleet. Opting in or out is a label change.
Results and costs
Netflix built a benchmark that runs an app beside a deliberately hostile "havoc" container and scores how much throughput and 9 latency (the latency 99% of requests beat) it keeps. The post publishes no scores, only reports from teams of stable operation. Netflix also argues that fleet efficiency improved. The shared hosts no longer have to stay partly empty to keep sensitive jobs apart, and right-sized instances wasted less capacity than big shared hosts did. No figures are given.
The costs are real. Each task waits for a fresh machine to boot. Warm pools of pre-booted machines help less because tasks need many different instance sizes. Fixed host overhead can eat tens of percent of a small instance. Netflix's guidance is to use the mode for large, long-running, disruption-sensitive jobs, not small bursty ones.
Netflix runs most of its services on 1, its in-house container platform built on Kubernetes (the widely used open-source system for scheduling containers onto machines). Titus starts millions of containers a week on tens of thousands of 2 instances (rented virtual machines in Amazon's cloud). By default it is multi-tenant: containers from many teams share each host. Netflix has now added a single-tenant mode. An eligible task gets its own right-sized EC2 machine with nothing else on it, while keeping the same Titus API, the same deployment pipeline, and the same arrangement where Netflix's Compute/Platform team, not the application team, owns and patches the host.
Why quotas aren't enough
Titus fences containers with the standard Linux tools. 3 cap CPU, memory and disk. Namespaces give each container its own view of the system. seccomp and AppArmor restrict which system calls a container may make. The iocost controller meters disk I/O, and traffic shaping meters network use. The authors argue that these tools partition only what the kernel controls. Several cores share a 4 (the largest, slowest on-chip CPU cache) and a path to RAM. A neighbor that streams lots of data from memory evicts your cached data and slows you down, even when both containers stay inside their CPU limits. No cgroup exists for cache space or memory bandwidth.
The article lists other ways neighbors interfere through shared hardware: contention for the CPU run queue, packet processing that runs in softirq context (deferred kernel work that gets charged to whichever CPU happens to be handling it), disk queues and per-volume throughput limits on cloud storage, NUMA traffic between sockets, TLB-shootdown interrupts, thermal throttling, and global kernel locks.
Netflix says the network has hurt most. Its per-container bandwidth shaping uses 5 (hierarchical token bucket), a Linux queueing discipline that serializes traffic behind one global lock. When one container floods the host with small packets, contention on that lock burns kernel CPU and starves its neighbors. The result is TCP retransmits and tail-latency spikes for services that did nothing wrong. The worst-hit workloads were stateful streaming jobs such as Apache Flink stream processors. When one of these falls behind, it may restore from a checkpoint and pause for many minutes, which pages an on-call engineer.
Four alternatives rejected
More shared-kernel tuning. Netflix keeps doing this, but it narrows the gap without closing it, because the hardware is still shared.
6 (Firecracker, Kata Containers): lightweight VMs that boot in about 150 ms. They need hardware virtualization, which means bare-metal instances, and nested virtualization wasn't ready on Netflix's instance types at the time. There is no managed service for them, so Netflix would have had to run its own hypervisor. They also add CPU and I/O overhead and don't support GPUs.
The central argument is about the threat model. microVMs and gVisor are built to protect against hostile tenants trying to break into each other. Netflix teams don't need that. They need performance isolation, and a dedicated VM provides it with no emulation layer on top.
How it works
Three layers cooperate:
Federation, the Titus layer that picks which cell and which capacity a job uses, checks an allowlist. A team opts in by targeting a single-tenant workload group; there is no public API flag. Federation tags the task and records which instance families it may use.
The autoscaler launches, just in time, the smallest instance from those families that fits the task's requested CPU, memory and GPU. Freshly vacated nodes can be reused before they are torn down.
The scheduler enforces one task per node through several mechanisms. Pod 8 keeps other pods off the node. A priority class exempts the task from preemption. A taint on the node repels everything that isn't single-tenant. Repacking (defragmentation) is disabled, so the task is never moved to consolidate the fleet.
Jobs keep the resources they asked for and are not silently resized. Moving in or out of the mode is a labeling change.
Results
Netflix measures isolation with a differential benchmark. A reference app runs alone, then runs again beside a "havoc" container designed to be a bad neighbor. The havoc container uses an XDP UDP flood (XDP is a fast in-kernel packet path), pointer-chasing and STREAM to thrash cache and memory bandwidth, fsync storms on disk, stress-ng against the scheduler, and GPU-burn on the accelerator. A run counts only once Linux's Pressure Stall Information shows tasks are actually stalling on a resource. Results are reported as medians with bootstrapped confidence intervals, because the latency distributions are heavy-tailed. The score blends two ratios:
Throughput efficiency: the share of operations per second the app keeps under attack.
Latency efficiency: undisturbed 9 latency divided by P99 under attack. (P99 is the latency that 99% of requests beat.)
A perfectly isolated system scores 100%. The post does not publish any measured scores. Its evidence is qualitative: the Flink, connector and logging teams report stable operation where they used to see intermittent interference.
The efficiency argument is the most interesting part. Before this mode, sensitive workloads were spread across hosts with anti-affinity rules, which left hosts partly empty. Disruption-sensitive stateful pods also blocked off-peak repacking. Netflix describes the pattern as "a few perpetrators, a lot of collateral damage." Moving the noisy and fragile jobs onto dedicated nodes lets the shared fleet pack more densely. Netflix also found that EC2 sells only discrete instance sizes, so big shared hosts always left bin-packing holes. Picking a close-fitting instance for each workload captured more efficiency in practice than multi-tenancy's higher theoretical ceiling. No figures are given. Patching also gets easier: draining a single-tenant host means moving one task, not evacuating many tenants with disruption budgets (limits on how many replicas can be down at once).
Trade-offs
Launch latency. On a shared host the machine is already booted and image layers are cached, so a task starts in seconds. A single-tenant task waits for its own instance to boot, which takes tens of seconds before the image pull even begins. Warm pools (pre-booted idle machines) help less here because right-sizing produces many instance shapes. A warm r7i.2xlarge is useless to a task needing an m7a.8xlarge. Netflix therefore keeps a small warm pool, narrows the list of allowed shapes, and works on cutting launch cost directly:
lazy-pulling image snapshotters (stargz, SOCI) that fetch only the bytes a container reads at startup
checkpoint/restore with 10, which snapshots a warmed-up process so it can be restored instead of cold-started
avoiding slow-booting metal instances
disabling Kubernetes preemption logic, which had left new nodes sitting idle under load
Fixed overhead. Sidecars, the kubelet (Kubernetes' node agent) and system services take a roughly fixed amount of CPU plus a few GB of memory. On a small dedicated node that can be tens of percent of the machine; on the largest shapes it is a couple of percent. The gap between the request and the next instance size is also pure waste. Netflix's guidance: use the mode for large, long-running, disruption-sensitive workloads sized close to an instance shape, not for small, bursty or highly elastic ones. That is why the mode is allowlisted.
Open questions
Rolling out many small nodes adds load on the Titus and EC2 control planes, especially during regional failover. Netflix is watching launch-rate metrics as adoption grows. For the shared fleet, it plans eBPF-based Expected Departure Time shaping (rate-limiting by timestamping packets instead of contending on a lock) to replace HTB, hardware cache and bandwidth partitioning (Intel RDT, AMD PQoS), and NVIDIA MIG for GPU partitioning. The longer-term goal is one intent-based Compute API covering both VMs and containers.
Remember
Cache space, memory bandwidth and kernel locks are shared below what any cgroup quota can partition.
Isolating a few noisy jobs can raise packing density on the shared hosts left behind.
Fixed host overhead makes dedicated nodes wasteful for small instances; the mode suits large, long-running jobs.
When researchers find a serious flaw, they usually tell the maintainers privately. A fix is prepared and released at an agreed time, before details go public. Heartbleed (OpenSSL, 2014) and Log4Shell (Java's Log4j, 2021) are the standard examples of flaws in widely shared components that forced the whole internet to patch at once.
Image: Latent Space
The news
On October 21, 2026, researchers disclosed PHP2Shell: two bugs in the core engine of 1, the server-side language that runs WordPress, Drupal and Joomla. The bugs are tracked as CVE-2026-109618 and CVE-2026-109619; 2 is the standard numbering system for public security flaws.
Both are 3 bugs: an attacker can make the program read or write memory it shouldn't. They sit in how PHP internally handles out-of-memory conditions and fatal errors, not in a plugin. That's why plugin updates can't fix them. The researchers report two kinds of attack:
Pre-authentication 4: running their own commands on a server without logging in. They demonstrated this against all three CMSs.
"Heartbleed-style" memory leaks: claimed against any PHP application.
These are researcher-reported demonstrations. They don't show that every deployment is exploitable the same way.
Who found it
Two teams found the bugs separately:
Calif. Ryan Austin of the security firm Calif found them in August 2026 using OpenAI's GPT-5.6 Sol. He was working in OpenAI's "Patch the Planet" program, with model access paid for by 5, OpenAI's $1B subsidy for defenders.
Wiz. Wiz, now part of Google Cloud, found the same bugs using Gemini 4 Argon, GPT-5.6 Sol and its own tool. It credits Gemini with the discovery, building the exploit, writing detectors and validating the patch.
The response
Because the flaw is in the runtime itself, the fix needed a major engine change. It took about two months of private work among PHP maintainers, researchers, security vendors and governments.
Before the 14:00 UTC patch release, Cloudflare pushed 6 rules (filters that block known attack patterns) to all customers, free tier included. Its CTO called them a stopgap. The NSA told operators to update immediately or take systems offline if they couldn't. OpenAI released a 7 to help operators check whether they're exposed.
Several viral claims are uncorroborated: a pre-patch leak, about 1.5 million unpatched servers, and outages at Wikipedia and Archive.org. Basic facts such as fixed version numbers and severity scores are also missing from the record.
The hard part is deployment
PHP hides in managed hosts, old container images, background workers and vendor appliances. Each copy must be found, updated, restarted and checked. A WAF doesn't help if traffic reaches the server without passing through it. And a patch closes the hole without proving nobody already got in.
The judgment
OpenAI calls this a "Defender's Window": models found the flaw before attackers did, "to our knowledge." Latent Space concedes the disclosure is partly marketing and that its severity compared with Log4Shell is debatable. Still, it calls the coordinated response the community at its best. It expects a year of similar finds, and calls this the biggest security audit software has ever had.
What was disclosed
On October 21, 2026, researchers published two vulnerabilities in the core engine of 1, the server-side language behind WordPress, Drupal, Joomla and much of the web. The bugs are called PHP2Shell and tracked as CVE-2026-109618 and CVE-2026-109619. 2 (Common Vulnerabilities and Exposures) is the industry's numbering scheme for public flaws.
Both are 3 bugs: the program can be made to read or write memory it shouldn't. The researchers place them in how PHP handles out-of-memory conditions and fatal errors internally. They are not in a plugin and not in any application's login code.
That location is why they're serious. The researchers report pre-authentication 4, meaning they ran their own commands on servers without logging in, against WordPress, Drupal and Joomla. They also claim "Heartbleed-style" memory leaks against any PHP application. The newsletter's headline says the bugs affect 70% of the web, but the body never sources that figure. Latent Space also cautions that these are researcher-reported demonstrations. They don't prove that every deployment has an identical, ready-to-exploit path.
Who found it, and with what
Two parties found the bugs independently:
Calif/OpenAI. Ryan Austin of the security firm Calif found them in August 2026 using GPT-5.6 Sol, working within OpenAI's "Patch the Planet" program.
Wiz/Google. Wiz, a security company recently acquired by Google Cloud, found the same bugs using Gemini 4 Argon, GPT-5.6 Sol and its own tool, Wiz Atlas. A Wiz-affiliated account credits Gemini with the discovery, building the exploit chain (the sequence of steps that turns a bug into a takeover), writing detectors and validating the patch.
The newsletter separates two OpenAI program names that are easy to mix up:
Patch the Planet is the remediation effort.
5 for Frontline Defenders is the access route: a $1B subsidy, announced about two months earlier, that gives vetted defenders access to OpenAI's cyber-focused models.
Why the fix took two months
When a bug is in an application, its developer ships an update. When it's in the language runtime, every distribution, host and container image has to change too. OpenAI says the patch needed a major engine change and coordination among PHP maintainers, researchers, security vendors and governments. It also says agencies got confidential advance notice. A participating researcher says they worked with the PHP Foundation to harden the patch. These are accounts from participants, not an independently audited timeline.
Disclosure day
Operators were warned to expect the patch at 14:00 UTC on October 21 (23:00 in Japan). Before that, Cloudflare deployed 6 rules for all customers, free tier included. A WAF (web application firewall) sits in front of a site and blocks requests that match known attack patterns. Cloudflare's CTO said the rules covered known exploit vectors, were being updated as "early attacks" appeared, and were only a stopgap until operators patched. He gave no counts of attacks or compromises.
The NSA advised operators to prepare for an immediate update and to take systems offline if they couldn't mitigate in time. That was the day's most-engaged post, at about 80,000 interactions. OpenAI then published a 7, a packaged set of instructions for its coding agent, that walks operators through checking whether they're exposed.
Much of the rest is unverified:
A leak. The Rundown reported that details leaked before the patch. What leaked, and when, isn't established.
Exposure and outages. A viral compilation claimed about 1.5 million servers (attributed to the scanning firm Censys) were running unfixed versions. It also listed shutdowns at Dutch institutions, Wikipedia, Archive.org, Asana and GOG. None of these claims is backed by an underlying advisory or status page, and nobody has established whether each outage was a precaution, routine patch downtime, an unrelated failure or a compromise.
Why patching is the hard part
Latent Space argues that the emergency is as much about distribution and ownership as about discovery. Here is an illustrative case. A company runs PHP:
on a managed host,
in a Docker image built six months ago from a pinned base image,
in a background job worker,
inside a vendor appliance.
Each copy has to be found, given a fixed package, redeployed, restarted, and checked to confirm the running build is the new one. The container keeps running old PHP until someone rebuilds it. The appliance waits on its vendor.
Three warnings follow:
WAF rules can give false confidence. Traffic that reaches the origin server without passing through the proxy skips the filter entirely.
Patching closes the hole, but it doesn't prove nobody got in. Credential and data exposure need a separate investigation.
Auto-patching through the exploit is a bad idea. Some posters suggested using the RCE to patch other people's servers. The newsletter calls this unauthorized access, with risks of outages, botched fixes, destroyed evidence and legal liability.
Its recommended sequence: get the official advisories, inventory exposed runtimes, apply temporary protections, deploy authenticated fixed packages, verify the running processes, and investigate separately.
What AI changed, and what it didn't
The case for AI here is stronger than a benchmark score. There were working exploit chains, detectors and patch validation. OpenAI calls this a "Defender's Window": models found the flaw before attackers did, "to our knowledge," which is not proof. Credit is shared across humans, two model vendors, PHP maintainers and Cloudflare. Google's separate claim that Gemini 4 Argon finds more than twice as many vulnerabilities as baseline models publishes no dataset, budget or false-positive accounting.
Researchers also argued over access policy. One credits Daybreak with finds in PHP, GitLab, Chrome and WordPress. Another says Anthropic contributed "zero value," which the newsletter treats as opinion. Anthropic reportedly delayed broad access to its Mythos-class models by four months to strengthen cyber safeguards. Investor Sarah Guo reports that new CVEs rose from about 4,000 a month to 15,000 since April. The count isn't verified, and more disclosures doesn't mean more successful attacks.
The newsletter's verdict
Latent Space concedes that every high-profile disclosure is partly marketing. It also says whether PHP2Shell belongs alongside Log4Shell or 9 is debatable. Still, it calls the coordinated response the developer community "in its most ideal form." It expects at least a year of similar critical finds. It argues the right reading is not to blame the messenger: critical software is getting "the biggest security audit it has ever had."
Remember
The bugs are in PHP's error and out-of-memory handling, so every app on an affected runtime needs the engine patched.
Two competing AI vendors' models found the same bugs independently, which weakens any single vendor's claim to credit.
WAF rules and patches limit exposure, but they don't show past compromise; that needs a separate investigation.
Run each task k times. pass@1 is the average success rate across all attempts. pass@k counts a task as solved if any attempt succeeds. A stricter measure counts tasks that pass every attempt. A model can score well on the first two and still be unreliable on the third.
Hotel task: the base model broke the approval policy in 15 of 20 runs; the trained model never did.1 / 4
Microsoft and Hugging Face have released ThinkingBox-Training, code that fine-tunes AI agents with 1 using a database check as the reward. After 50 training steps, Qwen3.8-27B, an open 27-billion-parameter model, completed 107 of 507 benchmark tasks correctly on all 20 attempts, up from 38.
From grader to reward
ThinkingBox is a test sandbox. Each task starts a real backend and gives the agent tools over 2 (Model Context Protocol, a standard tool interface). A simulated user, played by another model, holds context the agent must ask for. At the end, code checks the final database records, not the agent's steps. An earlier post showed that models often pass a task once but fail to repeat it. A check that returns 1 or 0 from database state can also serve directly as a training reward.
Picking tasks that teach
Training used 3 (Group Relative Policy Optimization). It runs eight attempts per task and reinforces attempts that beat the group average. If all eight fail, or all eight pass, nothing beats anything and the model learns nothing. Each attempt is costly: a fresh backend, a live tool session, a model playing the user.
So the team tested candidate tasks against the base model before training. A task that succeeded on 3 of 8 tries got a probability estimate, a Beta(3.5, 5.5) distribution, that placed it in a "difficult" band and kept it. Tasks almost never or almost always solved were dropped. That left 187 training tasks, none of them from the benchmark.
Results
Across 20 attempts on each of the 507 tasks:
Passed all 20 attempts: 38 → 107 tasks
Average success (4): 51.70% → 60.78%
Solved at least once (5): 453 → 446 tasks
In one hotel task, the base model changed a booking without the manager approval policy required in 15 of 20 runs. The trained model never did. Estimated cost per reliably solved task fell from $24.36 to $8.00. That excludes training costs.
Caveats
The authors describe the effect as "6": the model got more consistent, not more capable, and lost a few tasks it used to solve occasionally. The benchmark's domains and most tools were unseen in training, but both sets share the same framework and grading style. This is a single model and a single short run. The training pool is private.
Microsoft's Copilot Studio team and Hugging Face have released ThinkingBox-Training, the code they used to fine-tune an AI agent with 1. The reward comes straight from checking a database. On Qwen3.8-27B, an open-weights model with 27 billion parameters, 50 training steps raised average success on their benchmark from 51.70% to 60.78%. The number of tasks it completed correctly on all 20 attempts nearly tripled, from 38 to 107.
The grader that became a reward
ThinkingBox is a sandbox for testing agents on business workflows. Each of the 507 tasks in ThinkingBox-Bench, its benchmark, starts a real backend. It gives the agent tools over 2 (Model Context Protocol, a standard way to expose tools to a model) and adds a simulated user, another language model that holds private context the agent has to ask for. When the episode ends, executable checks inspect the final database state. They don't look at which tool calls the agent made, so any path that produces the right records passes.
An earlier post ran 18 models through this benchmark 20 times each and found a gap that headline accuracy hides. Kimi-K3 solved 93.89% of tasks at least once but only 68 of them (13.41%) on all 20 attempts. Claude Opus 5.5 beat Claude Opus 5 by 0.66 points on 3 (average success) yet passed exactly the same 241 tasks every time. The team then noticed that a check returning 1 or 0 from the database state can serve as a reward function as well as a grade.
Why choosing tasks is the hard part
The training algorithm is 4 (Group Relative Policy Optimization). It runs a group of attempts, called rollouts, on the same task, and scores each one against the group's average. Rollouts above the average are reinforced and those below it are discouraged. If all eight rollouts fail, the rewards are [0,0,0,0,0,0,0,0]. If all succeed, they are all 1s. Either way, no attempt beat any other, so the group produces no learning signal. Only a mixed group such as [0,0,0,1,1,1,1,1] teaches the model anything.
For agents, a wasted group is expensive. Every rollout provisions an isolated backend, runs a real tool session, and calls a separate model to play the user. Most candidate tasks are either already solved or out of reach for the current model. The authors call this "policy-relative difficulty": a task is hard only for a given model (the policy), and only until that model learns it.
How tasks were selected
Before training, a stronger model (GPT-5.4) first screened candidates to check they were solvable. The base Qwen model then attempted each candidate n times, scoring k successes. Rather than treat k/n as the true rate, the team estimated it with a 5, Beta(k+½, n−k+½). This is a standard way to express uncertainty about a success probability from a handful of trials. They split the 0–1 range into five bands, with boundaries chosen as hyperparameters: rarely solved [0, 0.08), difficult [0.08, 0.40), intermediate [0.40, 0.60), frequently solved [0.60, 0.90), and nearly always solved [0.90, 1]. Each task went to the band holding most of its probability mass. The two extremes were dropped.
Worked example from the article: a task succeeded on 3 of 8 probes, giving Beta(3.5, 5.5). That put 54.0% of its mass in the "difficult" band, so it was kept. In training it later produced the mixed group [0,0,0,0,1,1,1,1], which gives GRPO a contrast to learn from. This selection ran once, offline. The team deliberately did not re-estimate difficulty as the model improved. The final pool held 187 tasks, and none came from the benchmark.
The training run
Each update ran 18 groups of 8 full episodes (144 in total) on 24 H100 GPUs, with full-parameter training, a 1e-6 learning rate, and 50 steps. Groups that still came back all-0 or all-1 were replaced, in the style of 6 (a GRPO variant that resamples such groups). Backend timeouts were refilled rather than scored as failures, because an infrastructure error is not a wrong answer. Most rewards came from deterministic checks. A minority of narrow requirements that the database can't capture used an LLM judge (GPT-5 Chat) answering a yes/no rubric question. The authors stress that training and evaluation must parse model output identically, including reasoning, tool names and argument types. Otherwise the reward measures a different agent from the one being evaluated.
Results
The benchmark's domains, databases and most tools were absent from training. The results across all 10,140 attempts:
pass@1 and the all-20 count rose in all five domains. Retail's all-20 share rose by 24.5 points. pass@20 fell in retail, travel and auto insurance, by 7.0 points in auto insurance.
In one hotel task, policy required manager approval before a room downgrade. The base model changed the booking and issued refunds without approval in 15 of 20 attempts. The trained model left booking and payment state intact in all 20.
Rankings depend on the metric. GPT-5.6 Sol beat the trained Qwen on pass@1 (61.91% vs 60.78%) but passed only 82 tasks every time, against Qwen's 107. Claude Opus 5.5 still leads at 241.
The team also estimated cost by dividing the price of the full 507×20 evaluation by the tasks passed every time. On that measure, cost per reliably solved task fell from $24.36 to $8.00, close to GPT-5.4 ($6.80) and Claude Opus 5.5 ($7.80). That figure is a comparative serving index. It excludes the cost of training and probing.
Caveats and open questions
The authors read the results as "8": probability shifted toward paths that work repeatedly, and some rare successful paths disappeared. The model got more consistent, not more capable. The transfer claim is limited, because the training and benchmark sets share the framework, the simulated-user setup, the grading style and some tools. This is one model and one 50-step run, and the 187-task training pool is private, so outsiders can't reproduce the exact result. The article doesn't test whether re-estimating difficulty during training, the path not taken, would do better. It also doesn't say how longer training affects the loss in coverage. The authors recommend checking the difficulty spread of your own task set before training, verifying that training and evaluation parse output the same way, and reporting both coverage and repeatability.
Remember
For GRPO, task selection matters most: groups that all pass or all fail produce no learning signal.
Average accuracy and every-attempt reliability rank models differently. Report both.
Training and evaluation must parse model output identically, or the reward trains a different agent.
Cloudflare is a reverse proxy: requests to a site pass through Cloudflare's servers near the user (the "edge") before reaching the site's own server (the "origin"). A zone is a domain's configuration on Cloudflare. Cloudflare for SaaS lets one platform serve thousands of customers' own domains from a shared zone.
custom hostname metadata
Key-value data a platform attaches to each customer domain, readable by Cloudflare rules at the edge.
The two tool calls an agent makes to find and cart a shoe, instead of simulating clicks.
The news
Cloudflare has turned on 1 for Commerce.com, an e-commerce platform with nearly 100,000 store domains. WebMCP is an experimental browser standard from Google. It lets a web page offer AI agents named functions, or "tools", such as search_products. Cloudflare adds them by inserting a small script into pages as they pass through its network, so merchants change nothing.
The problem
AI agents shopping on the web today mostly read a page's HTML, guess which button does what, click, and retry when they're wrong. That wastes 2 (the units of text a model processes and bills for) and breaks easily: cookie banners, size drop-downs, bot checks and site redesigns all trip it up.
How it works
3, the Model Context Protocol, is a standard way to describe functions an AI model can call. WebMCP brings it into the browser. A page registers tools on document.modelContext, and an agent calls them directly. In the article's example, "find a waterproof trail shoe under $140, size 10, add it to my cart" becomes two calls: search_products(...) and add_to_cart(...).
The tools run inside the shopper's own browser session on the store's domain. The agent sees the real cart and prices, and no login credentials leave the browser. The shopper still clicks checkout and pays themselves. Cloudflare's script acts as a bridge: it fetches tool definitions from Commerce.com's existing backend MCP server and passes calls through unchanged.
What had to be built
WebMCP was a single on/off switch per 4, Cloudflare's configuration unit for a domain. Commerce.com hosts thousands of merchant domains on shared zones through 5, so one switch would cover every store. Instead of a custom allowlist, Cloudflare used two existing features. 6 are conditional settings checked on each request. Custom hostname metadata is per-domain data the platform sets. When a merchant opts out in the Commerce.com dashboard, a rule sees webmcp_opt_out set to true and disables WebMCP for that store. The data is already at the edge, so Cloudflare says opted-out stores get no injected script and no added latency.
Two smaller fixes came from real use. Commerce.com's MCP server lives at /api/mcp, not the assumed /mcp, so Cloudflare added a configurable path. And if the MCP server is down or the browser lacks WebMCP support, the script exits and the page loads normally.
Caveats
The post gives no numbers on reliability, token savings or adoption. It doesn't discuss abuse or security of the tools. WebMCP itself still only runs behind Chrome flags, origin trials and a few agent browsers.
The news
Cloudflare and Commerce.com, an e-commerce platform, have turned on 1 across Commerce.com's storefronts, nearly 100,000 domains. WebMCP is an experimental browser standard proposed by Google. It lets a web page offer AI agents a set of named functions, or "tools". Cloudflare delivers those tools by inserting a small script into pages as they pass through its network, so merchants don't change their sites at all. To make this work for a platform rather than a single site, Cloudflare had to add a per-merchant opt-out and support for custom server paths.
The problem: websites are built for eyes, not agents
An agent shopping for you today usually works from a page's HTML. It looks at the page, guesses which button adds an item to the cart, clicks, and tries again if the guess was wrong. Cloudflare's post names the usual failures: cookie banners get in the way, size and color drop-downs get misread, and bot challenges built to filter out automated traffic block the agent. Every attempt costs 2, the units of text a language model processes and bills for. A simple theme change can break an agent that had learned the old layout.
This is hard because the page's structure is the only interface the site offers. The agent has to work backwards from layout to intent, and nothing guarantees the layout stays the same.
What WebMCP changes
3, the Model Context Protocol, is a standard way to describe tools to an AI model: a function name plus typed parameters. Normally an MCP server runs on a backend and agents connect to it over the network. WebMCP moves the idea into the browser. A page registers tools on an object, document.modelContext, and an agent running in the browser can find and call them directly.
The article's worked example is the request "Find me a waterproof trail running shoe under $140 in size 10 and add it to my cart." With WebMCP, the agent makes two calls:
There's no clicking and no guessing at the layout. The tools run inside the shopper's own browser session, on the store's own domain. That has three effects. The agent sees the same prices and cart the shopper sees. No login credentials go to a third party. The visit still counts as a visit to the merchant's site. Checkout stays with the human: the shopper clicks the checkout button and enters payment details themselves.
WebMCP is in early preview. It works behind a Chrome flag (chrome://flags/#enable-webmcp-testing), in Chrome 4 (Google's program for testing experimental features on real sites), in some agent-focused browsers, and in developer environments such as Cloudflare Kitesurf. The article doesn't describe Kitesurf beyond that.
How Cloudflare delivers it
In August, Cloudflare released a beta that does this at the edge. When a site turns it on, Cloudflare adds a script tag to pages as they pass through its proxy. The script loads "5", bundles of tools. Commerce.com uses the mcp-server-client toolpack, which acts as a bridge. It fetches tool definitions from Commerce.com's existing backend MCP server, offers them to the browser, and sends the agent's calls back to that server unchanged. The tool logic stays on Commerce.com's side, and Cloudflare only carries it.
Pototska, a Commerce.com product director, says the browser tools and the existing server-side MCP serve two kinds of shopping. Some agents connect to stores directly, such as an assistant comparing several sites. Others run inside the shopper's browser. Both use the same set of tools, so merchants manage one set.
Problem 1: an on/off switch built for one domain
WebMCP was designed as a single switch per 6, Cloudflare's configuration unit for a domain. Commerce.com hosts thousands of merchants' own "vanity" domains on shared zones through 7, Cloudflare's product for platforms that serve customer domains. Turning WebMCP on for the zone would turn it on for every merchant, with no way for one store to leave.
Cloudflare didn't build a separate allowlist. It connected WebMCP to two features that already existed. 8 are conditional settings evaluated per request. Custom hostname metadata is a small bag of key-value data the platform attaches to each customer domain. Commerce.com turns WebMCP on for the zone, then adds this rule:
When a merchant flips the opt-out switch in their Commerce.com dashboard, the platform sets that metadata flag, and the rule disables WebMCP for that hostname. The metadata is already on hand at the edge when Cloudflare handles the request, so it needs no remote lookup. Cloudflare says an opted-out store gets zero extra latency and no injected script.
Problem 2: not every MCP server lives at /mcp
The bridge assumed each site's MCP server was at /mcp. Commerce.com's is at /api/mcp. Cloudflare added a zone-level setting for the MCP URL and passes it to the injected script as an attribute:
The tag also loads a c2pa pack. C2PA is a standard for content provenance (a record of where content came from), but the article doesn't say what this pack does here.
Failing safely
If the store's MCP server is down, or the browser doesn't support WebMCP, the script exits. The page loads as usual, and human shoppers notice nothing. This matters because the feature is injected into pages the merchant didn't write. If it can't fail quietly, it becomes a new way for the storefront to break.
What the article doesn't say
The post gives no measurements. It reports no task success rates, token savings, latency figures, or numbers on how many stores or agents use the feature. "Faster, cheaper, and far more reliable" is stated, not shown. It also doesn't cover how tools are secured against misuse, for example an agent or malicious page running add_to_cart in ways the shopper didn't intend, or how bot management treats tool calls. WebMCP is still a proposal and only works behind flags and trials, so the shoppers who can use it today are mostly developers and early adopters. Cloudflare says its next step is helping agents find a site's tools more easily, which suggests discovery is still unsolved.
Remember
WebMCP swaps guessing at page layouts for typed function calls, run in the shopper's own logged-in browser session.
Cloudflare built the per-merchant opt-out from existing rules and per-hostname metadata, not a new allowlist system.
The post reports no numbers on reliability, token savings or adoption, and WebMCP still only runs behind flags.
A sandbox is an isolated, disposable computer environment where untrusted code can run without harming anything else. AI coding agents write code and then need somewhere safe to execute it, check the result and try again. Reinforcement learning, which trains models by letting them attempt tasks, needs huge numbers of these environments.
Image: Latent Space
Modal, a company that runs developers' Python code on cloud GPUs, reports $1 billion in annualized revenue and nearly three billion 1 started. On the Latent Space podcast, CEO Erik Bernhardsson explained the business, and the engineering, behind a simple mismatch.
Demand isn't a rectangle
GPU contracts usually give you a fixed number of chips for a fixed term. Plot that against time and it's a rectangle. Real AI workloads spike, cycle through the day and disappear. Buy for the peak and you pay for idle chips. Buy less and you miss demand.
Modal signs the rectangular contracts itself and resells capacity in small, short slices. Bernhardsson calls this a "duration transformation", similar to a bank turning short deposits into long loans. It works because thousands of customers' uneven demand adds up to a fairly smooth total that can be kept busy. Modal treats its capacity worldwide as one global pool rather than separate regional clusters.
Buying the supply
Supply has become harder to get. Modal now works with almost 25 clouds and often has to sign three-year deals with large upfront payments. Four people work full time on procurement, and about ten engineers deal with problems on physical servers. Small blocks of GPUs that big labs ignore, such as 32 nodes in Armenia, suit Modal fine.
Bernhardsson worries less about prices rising than about them collapsing, which would leave anyone with locked-in contracts overpaying. He also warns that falling prices, falling usage and a vanishing resale market would all hit at once.
Sandboxes: almost cancelled
A sandbox is an isolated, disposable environment for running untrusted code, such as code an AI wrote. Modal built one in a day or two in 2023, and early use was weak enough that the team discussed shutting it down. Then coding apps, background agents and 2 (training models through repeated attempts at tasks) took off. Sandboxes now bring in about a third of revenue. Serverless functions, mostly used for running models, bring in the other two-thirds.
Why startup speed matters
Bernhardsson argues that agents should start a sandbox in about 100 milliseconds, run a few commands, snapshot it and shut it down, rather than keep a cloud dev machine running. His example is 3, the automated tests run on every code change. Today, each parallel shard of a test suite repeats the slow setup. Do the setup once, snapshot it, and clone it into a thousand sandboxes, and each test can run on its own.
For GPUs, Modal speeds up startup with a buffer of spare GPUs, a custom filesystem, 4 and GPU memory snapshots. The interview gives no latency or utilization figures.
Modal is an AI infrastructure company. Developers write ordinary Python, and Modal runs it on cloud GPUs and CPUs that it manages behind the scenes. The company reports that it has passed $1 billion in annualized revenue and raised $750 million at a $16 billion valuation. It says it has started nearly three billion 1. On the Latent Space podcast, CEO Erik Bernhardsson explained the economics and engineering behind those numbers. The most useful part for engineers is his account of the gap between how GPUs are sold and how AI workloads actually use them.
The rectangle problem
A typical GPU contract gives you N GPUs for T years. Plot capacity against time and you get a rectangle. Real demand is not shaped like that. An inference service, meaning a deployed model answering requests, follows daily traffic cycles. A batch job comes in a burst. A research team needs 64 GPUs for a week and then none. If you buy for the peak, you pay for idle chips. If you buy less, you turn away demand at the peak.
Bernhardsson describes Modal's business as a kind of duration transformation. A bank does something similar when it takes short-term deposits and makes long-term loans. Modal signs long, rectangular contracts with suppliers and sells short, uneven slices to customers. This works because demand from many thousands of customers, added together, comes out much smoother than any single customer's demand. Here is an illustrative example with made-up numbers. One customer might swing between 0 and 100 GPUs over a day. A thousand customers whose peaks fall at different times might together swing only between 40,000 and 55,000. A smooth total can be planned for and kept busy.
This is why, he says, Modal has never wanted to become a "2", one of the newer companies that mainly rent out GPUs in bulk. Selling rectangles is the neocloud business. Modal's product is elasticity.
Where the capacity comes from
The supply side has changed. Until late last year, most of Modal's GPUs were 3 or one-year reservations on 4, the largest cloud providers (AWS, Google Cloud, Azure), plus Oracle. Spot capacity can be taken back at short notice. Modal could live with that because it pooled capacity across hundreds of regions. Early on it made a decision that turned out to matter a great deal: it treats all its capacity worldwide as one global pool rather than a separate cluster per region.
Growth and a tighter market pushed Modal to work with almost 25 clouds. Getting capacity now often means three-year deals with large upfront payments, which lock in today's prices. Modal has four people working full time on procurement. It has about ten engineers handling bare-metal problems, meaning physical servers it rents directly, where network faults, failing drives and GPUs "falling off the bus" (dropping off the system's connection) have to be dealt with by hand.
New vendors get a spec sheet listing minimum storage speed (IOPS, input/output operations per second), CPU scores and network bandwidth. Where possible, Modal runs a single test node for a while before committing. Even so, Bernhardsson admits that commitments in the tens or hundreds of millions of dollars can involve "a leap of faith". Modal keeps an informal internal ranking of vendors based on reliability and how fast they fix problems. Small blocks of GPUs suit it fine. He cites 32 nodes in Armenia and 128 in Indonesia. The big labs tend to skip blocks like these because large training runs need huge, tightly connected clusters.
The financial risk
Locking in prices means Modal effectively holds a bet that GPU prices will stay high. Bernhardsson says rising prices don't worry him much, since scarcity also lets Modal charge more. A price collapse would hurt anyone holding expensive long-term supply. Over the long run, hardware prices nearly always fall. He also points to correlated risk: if prices drop, utilization falls and the resale market disappears all at the same moment. That is why "just buy everything and resell the excess" is not a safe plan. He cites the Hunt brothers' failed attempt to corner the silver market around 1980. Modal has chosen to rent rather than own, because buying hardware means either issuing a lot of new shares or taking on debt that needs guaranteed long-term revenue to repay.
He also describes how neoclouds grow so fast. They pre-sell multi-year capacity to a large buyer, borrow against that contract, and then buy the hardware. Modal's customers mostly sign up and pay as they go, with few long-term contracts, so it cannot use this playbook as easily.
Sandboxes: nearly cancelled, now a third of revenue
Modal has two main products. Functions are serverless functions, mostly used for inference, and they bring in about two-thirds of revenue. Sandboxes bring in about a third. Sandboxes started over lunch: Bernhardsson and his cofounder Akshat Bubna noticed that language models were writing code that would need somewhere to run. Bubna built a prototype in a day or two on top of 5, a Google tool that isolates containers by intercepting their system calls. Adoption was weak, and the team talked about shutting it down. (The transcript's dates for this period are garbled.)
Then demand arrived from three directions. Vibe-coding apps such as Lovable needed to run the apps they generate. Companies such as Ramp used sandboxes for background agents. Research labs ran 6, which trains a model through repeated attempts at tasks, inside sandboxes. Modal has since added full virtual machines for workloads that need complete Linux compatibility, along with snapshotting.
Why startup speed is the product
Bernhardsson's argument is that today's agent setups mostly "lift and shift" a developer's machine into the cloud. A better model, he argues, is a long-running agent that starts a sandbox in about 100 milliseconds, runs a few commands, saves a snapshot and shuts down. Agents could then fork environments and run hundreds of branches in parallel.
7 (continuous integration, the automated tests that run on every code change) makes the point concrete. If you split a test suite into shards today, every shard repeats the slow installation and setup. As 8 says, the part you can't parallelize then limits the overall speedup. If you do the setup once, snapshot the container and clone it, each test could run in its own sandbox, perhaps a thousand at a time.
For GPU inference, Modal's write-up lists the machinery behind fast startup: a buffer of spare GPUs, a custom filesystem, 9 (saving a running program's state so it can resume) and GPU memory snapshots. Each piece cuts the time needed to bring another copy of a model online.
Other notes and open questions
Bernhardsson thinks 2027 may finally be the year non-NVIDIA chips catch on. He notes that some teams already split inference across hardware: NVIDIA chips for prefill (processing the prompt) and Cerebras chips for decode (generating the output). He also says strong open-weight models such as Kimi, DeepSeek and GLM have made fine-tuning optional rather than necessary. Modal reports reaching four to five nines of uptime (99.99% to 99.999%) recently. The interview gives no utilization figures and no startup latency numbers, so the central claim that pooling smooths demand is argued but not quantified.
Remember
Combining many customers' demand turns long fixed GPU contracts into elastic capacity, but it leaves the provider exposed if GPU prices fall.
Modal's sandbox product nearly got shut down; agents and reinforcement learning grew it to about a third of revenue.
Snapshot-and-clone startup could let CI run each test in its own sandbox without repeating setup work.
7: a user-space kernel that intercepts system calls. That interception taxes I/O-heavy paths, which are exactly the jobs needing protection. The authors also say the open-source version lags the internal one and effectively depends on a single vendor.
Plain EC2: this hands instance choice, patching and scaling back to the application team, and teams treated the move as a one-way door.
Japan. SemiAnalysis separately reports that Japan's prime minister urged operators to patch or disconnect.
Basic facts. Exact fixed versions, 8 scores (the standard 0–10 severity rating) and the conditions an attack needs are all missing from the material.
