Agents Under Test, and a Bacterial Vault · Morning Gist

Thursday, November 55 stories · 15 min

Agents Under Test, and a Bacterial Vault

Today's posts show agents doing lab work, debugging clusters and fixing themselves, though several vendors publish far stronger claims than evidence.

Picked from 35 new articles across 24 sources · arrow keys for other days

Today in 30 seconds

NVIDIA's workflow turns agent failure logs into tests; a one-paragraph instruction change took its demo mail assistant from 7/12 to 12/12. story 1 ↓
Anthropic's lab solved the first bacterial vault structure using Claude, which searched 15 billion sequences; purification fell from three days to 90 minutes. story 2 ↓
NVIDIA described six read-only agents that troubleshoot its GPU clusters, released as experimental open source, with no accuracy figures published. story 3 ↓
Google's open-source Agile Deliberation uses borderline images to define subjective classifiers; it beat three baselines in 18 live sessions, but figures are unpublished. story 4 ↓
OpenAI's Images 2.6 adds GIFs and SVG export, plus Flare for speed and Sunburst for consistency across edits; quality evidence is thin. story 5 ↓

1/5NVIDIA Technical Blog· Patrick Moorhead· Nov 4

NVIDIA publishes a workflow that turns agent failure logs into tests and fixes

NVIDIA's tutorial uses NeMo tools to mine an agent's run logs for recurring failures, generate tests, and check fixes on hidden tasks. The evidence is promising but small: twelve runs in one demo, vendor-reported results in another.

Why it matters · The trace-to-test loop is a useful discipline for anyone shipping agents, and the tutorial shows the concrete steps, including held-out tasks that stop a coding agent overfitting its fixes.

Know this first

Context from the editor, not the article · 27 sec

1 · What an agent harness is
The harness is everything around the model: system instructions, tool definitions, routing rules, memory and control-flow code. Many agent failures originate here, so swapping or tuning the model may not fix them.
2 · Traces versus evaluations
A trace is the recorded log of one agent run, including its tool calls. An evaluation is a repeatable test with a pass or fail check. Traces help explain what went wrong; evaluations measure whether a change helped.
3 · Why tasks get held out
If whoever writes a fix can see every test, the fix may simply fit those tests. Keeping some tasks hidden, as with a machine learning test set, gives a fairer check that the change generalizes.

2/5Anthropic Research· Nov 4

Anthropic's lab solves first bacterial vault structure using Claude

Anthropic's biology team used Claude to find, purify and solve the 3D structure of a vault protein shell from a bacteriophage — 96 subunits, 9.5 megadaltons — in under four months.

Why it matters · A concrete, auditable account of a model running an end-to-end research workflow — including the errors and the human corrections — rather than a benchmark score.

Know this first

Context from the editor, not the article · 23 sec

1 · Why structures matter
Protein function is often inferred from shape. Determining how a large protein complex assembles in 3D usually requires cryo-electron microscopy plus weeks of image processing in specialist software, and considerable expertise.
2 · Prokaryotes vs eukaryotes
Prokaryotes are simple cells like bacteria and archaea; eukaryotes have nuclei and include fungi, plants and animals. Bacteria grow in hours, making them far faster experimental systems than mammalian cells.
3 · Labs running on models
Anthropic has been publishing findings from its own wet-lab biology team, framed as demonstrations that Claude can drive real discovery workflows rather than just summarise literature.

3/5NVIDIA Technical Blog· Shara Tibken· Nov 4

NVIDIA details the AI agents that help troubleshoot its GPU clusters

NVIDIA described and released as experimental open source the framework behind its internal cluster-operations agents, which trace failures across hardware, network and scheduler layers and recommend fixes to engineers. It gives no performance results.

Why it matters · It is a concrete reference design for putting LLM agents near production infrastructure, showing how to scope tools and credentials, though without evidence that the diagnoses are accurate.

Know this first

Context from the editor, not the article · 30 sec

1 · Why GPU-cluster debugging is hard
A large training job runs in lockstep across many GPUs, so one faulty part can slow or stop all of it. The evidence sits in separate scheduler, hardware, network and application tools, which engineers must correlate by hand.
2 · Model Context Protocol
Model Context Protocol (MCP) is an open standard for connecting models to external tools. An MCP server publishes named tools with descriptions, and the model chooses and calls them, so limiting a server's tools limits what the agent can do.
3 · The NVIDIA agent stack

4/5Google Research· Nov 4

Google's Agile Deliberation uses borderline images to build subjective AI classifiers

Google Research built an open-source system that shows people borderline images and rewrites an AI classifier's prompt from their feedback. In 18 live sessions, it reportedly beat three alternatives and reduced user stress.

Why it matters · Anyone building moderation filters or policy classifiers faces shifting definitions; this offers a concrete workflow for finding edge cases, while showing how little a vendor's summary can prove.

Know this first

Context from the editor, not the article · 28 sec

1 · Labels assume a fixed definition
Supervised machine learning assumes that whoever labels examples already knows what the category means. That works for stop signs, but less well for "healthy food", where people often discover their rules only after seeing difficult cases.
2 · Prompt-driven image classifiers
A vision-language model accepts both images and text. Given a written definition, it can decide whether an image fits. The prompt then acts much like the classifier's rulebook, so changing the wording changes its decisions.
3 · Active learning
Active learning is a training method in which the system chooses which examples a person should label next, usually those the model is least sure about. The goal is to get more model improvement from fewer labels.

5/5OpenAI· Nov 4

OpenAI releases Images 2.6 with GIFs, SVG export and a complexity control

OpenAI's Images 2.6 generates images a claimed 50% faster and adds GIF animation in ChatGPT, Codex and the API. Developers also get two models, SVG export, region-targeted edits and a setting controlling planning effort.

Why it matters · For anyone building on image APIs, the separate quality and complexity settings change how to tune cost and latency, though OpenAI's post offers little evidence for its quality claims.

Know this first

Context from the editor, not the article · 25 sec

1 · Why consistency is hard
Image models synthesize pixels from a prompt, so a follow-up edit or a new frame can subtly change a face, logo or product. Keeping details stable across outputs has long been a weak spot.
2 · Sprite sheets and animation
A sprite sheet is one image holding an animation's frames in a grid. Software cuts it into panels and plays them in order, so a GIF can begin as a single generated picture.
3 · Pixels versus vectors
Most generated images are raster: fixed grids of pixels that blur when enlarged. SVG describes shapes mathematically, so it scales cleanly and suits logos, icons and print layouts.

End of edition

You’re caught up.

The next edition lands tomorrow at 06:00.

Models Lose the Plot, Laptops Read Weights All digests

NVIDIA has published a tutorial and a runnable reference project, the , for improving AI agents using records of their own runs. The loop is: find a recurring failure, write a test for it, change the agent, and measure on tasks the fixer never saw.

The problem

Generic benchmarks rarely explain why a domain-specific agent fails. Causes often sit in the , the prompts, tools, routing, memory and control code around the model. Reading every run log doesn't scale, and without a repeatable test you can't tell whether a fix worked.

The demo

The example agent is an enterprise assistant for a fictional company, built on , an agent framework from Nous Research. Asked to "draft a message with project updates," it drafted the message and then called send_message without asking.

Finding the bug. analyzes , the recorded logs of agent runs, and produces a prioritized list of issues linked to specific runs. Given 42 traces and , a short file stating how the assistant should behave, it flagged the unconfirmed sends.

Writing the test. gives a coding agent, such as Codex or Claude Code, the skills to generate evaluations from those findings. It produced four tasks like "draft a response," plus a , the code that fails any run calling send_message without human confirmation.

Making the fix. The change was a short addition to the agent's system prompt: requests to draft a message "authorize preparation only," and a token returned by a preparation tool doesn't establish the user's consent.

Measuring it. Each task ran three times, and two tasks were hidden from the coding agents proposing changes:

Evaluation set	Baseline passes	Updated passes	Tool calls
Development	3/6	6/6	10 to 6
Held out	4/6	6/6	28 to 15

Rerunning Compass showed the finding had gone and surfaced a new one: the agent sometimes found an answer but didn't ground its response in the evidence it retrieved.

A larger example

NVIDIA also ran the loop on , a Dassault Systèmes engineering assistant that proposes and simulates car door designs. Compass found it misplacing tool arguments, which produced an empty error it couldn't recover from. A coding agent added a validation step, and malformed calls fell from 142 to zero with accuracy unchanged. An unattended run of about 12 hours reported a 28% higher evaluation score and 35% fewer tool calls per run across six scenarios, with no reported regressions.

The caveats

This is a vendor tutorial with small numbers. Twelve runs per condition, all aimed at one bug, can't show a fix is reliable; NVIDIA itself says production suites often need hundreds to a few thousand examples. Its claim that Compass beats a coding agent at spotting anomalies comes with no figures. The LEO score's definition isn't given either. The method is still worth copying, particularly the rule that a change must also improve on hidden tasks.

NVIDIA has published a tutorial for improving AI agents from their own production records. It comes with the for agent optimization, a reference project you can run, and it walks through one full cycle: find a failure in recorded runs, write a test that reproduces it, change the agent, and compare before and after.

The problem

Generic benchmark scores often can't explain why a domain-specific agent fails. It might pick the wrong tool, take the wrong workflow branch, miss context, use a needlessly expensive model, or finish a task without meeting a domain requirement. The causes are spread across the , meaning the prompts, tool descriptions, routing, memory, skills and control-flow code around the model. A single unclear tool description can derail a whole task.

The hard part is scale and proof. Reading every run log by hand doesn't scale, and without a repeatable test you can't tell whether a fix worked or just moved the failure elsewhere.

The demo agent and its bug

The example is built on , an agent framework from Nous Research, running in NVIDIA , a sandbox for running agents. It plays an enterprise assistant for a fictional company, connected to chat, email, files and tickets through , a standard interface between agents and external tools.

Asked to "draft a message with project updates," the agent drafted the message and then called send_message without asking the user. Nobody asked it to send anything.

Step 1: find what to fix

analyzes stored , the recorded logs of agent runs, and any evaluation evidence, then reduces them to a prioritized backlog of issues with supporting examples. In the demo it reads 42 traces, stored as one JSON object per line, plus , a short file stating how the assistant should behave.

Its findings draw on three kinds of signal: signs of unusual behavior, patterns in how the agent uses tools, and divergence from intended behavior. Each signal links to trace IDs, so an engineer can check the actual runs. Here, comparing actions against ETHOS.md exposed the unconfirmed sends.

NVIDIA says that in internal benchmarks on thousands of traces, the full set of signals caught anomalies a coding agent given a similar task missed. It gives no numbers, so that claim can't be assessed.

Step 2: write the test

provides skills and workflows that a coding agent, such as Codex or Claude Code, uses to generate evaluations. You give it the Compass findings, raw traces, ETHOS.md and repository access. It can output two formats: , an agent evaluation framework, and NVIDIA .

In the demo it produced four Harbor tasks resembling "draft a response." It also created a , the code that decides whether a run passed. This one fails any run that calls send_message without human confirmation. NVIDIA tells users to review and tune what gets generated, since a generated test may not capture the real requirement.

Eval Author can also work before production data exists, using ETHOS.md, tool definitions and existing workflows, and then expand the suite as real failures arrive.

Step 3: change one thing

The fix went into , Hermes's equivalent of a system prompt:

Requests to write, draft, compose, prepare, or suggest a message authorize preparation only. Provide the draft for review and stop before sending it. A token returned by a preparation tool is a technical capability to execute an action. It does not establish the user's consent to that action.

The last two sentences are telling. They imply the agent had been treating a token from a preparation tool as permission to act, though the article doesn't spell out that diagnosis.

Step 4: measure

Two of the four tasks were hidden from the coding agents proposing changes, and each task ran three times.

Evaluation set	Baseline passes	Updated passes	Tool calls
Development, 2 tasks × 3 runs	3/6	6/6	10 to 6
Held out, 2 tasks × 3 runs	4/6	6/6	28 to 15

NVIDIA's acceptance rule is that a change must improve both the development and held-out sets. If results are inconclusive, or a fix causes a regression, you run Compass on the new traces and let the coding agent refine the change, keeping held-out cases hidden throughout.

Compass was then rerun on the updated agent's traces. The unconfirmed-send finding was gone, and a new one surfaced: for some queries the agent found the answer but didn't ground its response in the retrieved evidence. That becomes the next round's target.

The engineering example

NVIDIA also applies the loop to , an engineering "Virtual Companion" from Dassault Systèmes that proposes car door design changes and tests them in simulation. Here evaluations have to measure engineering outcomes as well as how efficiently the agent uses tools.

Compass found the agent repeatedly putting QueryModel arguments outside the expected options field. The tool returned an empty error message, so the agent couldn't recover. A coding agent added a validation step that catches misplaced arguments and names in tool calls. Malformed tool calls fell from 142 to zero, accuracy was unchanged, and no regressions appeared in the remaining scenarios.

The first pass ran overnight without human intervention. In about 12 hours, the reported results showed a 28% relative improvement in the workflow's overall evaluation score and 35% fewer tool calls per run across six scenarios, again with no reported regressions. The article doesn't say how the score is calculated or what the baseline was.

How much to trust this

The workflow is sensible, and the held-out step is its best feature. The evidence is thinner. The workplace demo rests on 12 runs per condition across four tasks, all aimed at one bug in a fictional company's data, so its held-out tasks test a narrow slice of behavior. Going from 4/6 to 6/6 is suggestive at that sample size, not conclusive. NVIDIA acknowledges the suite is small and says production suites often hold hundreds to a few thousand examples.

The tutorial also doesn't compare Compass and Eval Author with simpler alternatives, such as asking a coding agent to read the traces directly, in any measurable way. And in my judgment, an instruction in a prompt is a soft control for an irreversible action like sending a message. Passing six of six doesn't prove it will always hold, and a confirmation gate in code would enforce it. NVIDIA notes that harness changes can extend to middleware and plugins, but doesn't make this comparison.

The blueprint's third library, NVIDIA NeMo Relay, is named but its role isn't explained. To run the demo, NVIDIA recommends Ubuntu 24.04 running Docker natively, 8 vCPUs and 32 GB of RAM, plus OpenShell 0.1.2, Harbor 0.22.0 and an NVIDIA API key.

Anthropic's in-house biology lab has published the first 3D structure of a from a prokaryote — and spent most of the write-up explaining how Claude did the work.

Vaults are hollow, ribbed protein barrels first seen in 1986, present in almost every cell in the human body and across the eukaryotic tree. In mammals they are built from 78 copies of one protein, roughly three times the mass of a ribosome. Forty years on, nobody knows what they do. Part of the reason: they are absent from the standard genetic workhorses — yeast, fruit flies, nematodes — and mice bred without vault components look fine.

What they found

Using to search a database of about 15 billion predicted proteins, the team found vault relatives in 29 bacterial phyla and four archaeal ones, but in under 1% of sequenced species. They picked 44, had Claude convert each sequence into an orderable DNA construct, screened them in a cell-free system, moved 24 into E. coli, and got 15 soluble proteins. Seven formed vault-like particles under an electron microscope; confirmed true vaults for four.

The detailed structure is of L05, a protein. It assembles 96 copies (48 per half) into a 9.5-megadalton shell, 54 nm long by 45 wide — shorter and fatter than the mammalian 67×40, yet enclosing about 25% more volume. Its wall uses seven stacked repeat segments instead of nine. The end caps close in three tiered steps: in each trio of neighbouring copies, identical along the wall, one stops at the first step, one at the second, one reaches the tip.

They then tested the assembly rules. Of 15 engineered variants, single changes near the cap left open, rosette-like particles instead of sealed shells. Duplicating two wall repeats to reach nine — the human number — produced a larger vault with roughly 20% more volume, a plausible mechanism for how vaults changed size over evolution.

The workflow, and its failures

Claude argued against the faster purification route (, where big particles exit first in the "") because misfolded clumps elute there too; pushed ahead, it then suggested loading raw cell lysate directly onto the column. That cut purification from three days to about 90 minutes. For structure determination, Claude drove the open-source package in a multi-agent harness across hundreds of GPUs: a first model in 34 hours, final maps in about a week, versus weeks of manual work.

It also erred. A protocol specified a growth medium additive that blocks getting DNA into cells, costing two weeks; a skipped cryo-EM step produced a map blended with its mirror image. The team's instruction: "Please let's do normal things unless they don't work." Lab notes went back to the researchers who train Claude.

An unidentified small molecule sits in a pocket on each of the 96 copies; Claude noted it resembles coenzyme A. The vaults have so far only been made in E. coli, and the function remains unknown.

Anthropic's biology research lab has published what it says is the first three-dimensional structure of a from a prokaryotic cell — and the more interesting half of the write-up is the account of how Claude was used at every step, including where it was wrong.

The 40-year-old puzzle

A vault is a large, hollow, ribbed protein shell, named in 1986 by Leonard Rome and Nancy Kedersha after the vaulted ceiling of a cathedral. It is found in nearly every cell in the human body and across most of the eukaryotic tree, from slime moulds to humans. At roughly three times the mass of a ribosome, it is one of the largest known molecular complexes in eukaryotes. In mammals, it is 78 copies of the , arranged as two identical halves joined at the waist.

Nobody knows what it does. Proposals include shuttling molecules between nucleus and cytoplasm, protecting cells against toxins such as cancer drugs, and supporting innate immunity; none is confirmed. Two facts make progress hard. Vaults are missing from the organisms geneticists lean on most — baker's yeast, fruit flies, nematode worms — so the usual knock-out-and-look-for-a-defect playbook is unavailable. And in mice, which do have vaults, animals bred without a vault partner protein or the vault RNA develop normally. Yet evolution has kept this expensive object around, which is the standard argument that it must be doing something.

Bacterial relatives of human proteins have cracked open problems like this before: bacterial versions of cGAS–STING signalling, gasdermins and viperins revealed the microbial origins of human immune defences. In those cases the human function was already known. For vaults it isn't, which is what makes a simple, fast-growing bacterial system attractive.

Finding candidates

Bioinformaticians had already spotted distant vault-protein relatives in prokaryotic genomes and in — viruses that infect bacteria — but nobody had shown they assemble into vaults.

The team used to search a database of about 15 billion predicted proteins. The relatives turned out to be rare but widespread: present in 29 bacterial phyla (major groups) and four archaeal ones, but in fewer than 1% of species with a sequenced genome. The humans picked 44 proteins; Claude turned each sequence into DNA codon-tailored to the host, added the control sequences that switch a gene on, and prepared the designs for ordering. A cell-free screen identified the best performers; 24 went into living E. coli, and 15 produced soluble protein.

A three-day step done in 90 minutes

Isolating the particles was the bottleneck. The conventional eukaryotic method spins samples through a gradient of increasingly dense sugar and takes about three days. The team wanted to try instead: the sample flows through a column of porous beads, small molecules get held up in the pores, and anything too big to enter the pores — like a vault — comes straight out in the so-called .

Notably, Claude argued against this, because clumps of misfolded protein also come out in the void volume and would contaminate the result. The team went ahead anyway, and Claude then helped make it work, recommending a specific column plus assays to track the vault protein and confirm particles had formed. It went further and proposed loading the contents of ruptured cells directly onto the column with no prior purification — something the team was reluctant to try, since raw E. coli lysate can clog or contaminate an expensive column. It worked. Others have isolated vaults on sizing columns before, but the optimised route takes about 90 minutes from cells to microscopy grid.

Under an electron microscope, seven of the 15 proteins had formed vault-like particles. Cryo-electron microscopy (cryo-EM) — flash-freezing many copies, imaging them with an electron beam, and combining the images into one 3D model — confirmed genuine vaults for four: two from phages, two from bacteria.

The structure

The detailed model is of L05, a phage protein chosen because it formed complete, closed particles most reliably. It assembles from 96 copies, 48 per half, with a combined mass of about 9.5 megadaltons; the mammalian version has 39 per half. The phage vault is shorter and wider — roughly 54 nm by 45, against 67 by 40 in mammals — but because it is more spherical it encloses about 25% more volume. Its wall subunit is built from seven repeating segments stacked end to end rather than nine, and its end caps are flatter. It is among the largest protein shells identified in prokaryotes.

The cap geometry is the elegant part. A wall 48 subunits around has to narrow to a pole, and it does so in three steps, like a tiered dome: in every group of three neighbouring copies, one ends at the first step, one at the second, and one continues to the tip. The three chains are identical sequences that take identical shapes along the wall but three different conformations in the cap.

Because they could make and isolate the particle quickly, the team probed the assembly rules with 15 engineered variants, most altering the interfaces where neighbouring copies touch. Single changes near the cap were enough to stop the shell closing: instead of sealed vaults, open rosette-like particles appeared. Going the other way, duplicating two of the seven wall repeats to give nine — the human number — produced larger vaults with about 20% more enclosed volume. That is direct evidence that vaults could have changed size over evolutionary history by duplicating pieces of their own wall without breaking assembly, and it suggests a route to tunable nanocapsules for, say, enzyme delivery.

The harness, and the mistakes

Structure determination normally means weeks or months of manual work in specialised software. Here Claude ran the open-source package itself, inside an internal multi-agent harness with access to hundreds of GPUs in parallel, with humans advising on analysis decisions. A first full model was ready 34 hours after processing began; final maps about a week.

Claude also flagged something unexplained: a pocket partway up each of the 96 copies holds a small molecule that appears clearly in the map but is unidentified. Claude noted its shape resembles coenzyme A, a common carrier molecule cells use to shuttle chemical groups — which, if confirmed, would be an early hint at function.

The failures are reported plainly. A Claude-written protocol specified a growth medium containing an additive that interferes with getting DNA into cells, setting the project back two weeks. Early in cryo-EM processing it skipped a standard step and produced a flawed map blended with its mirror image, prompting the instruction "Please let's do normal things unless they don't work." Errors in areas the team knew well were caught fast; those in unfamiliar areas cost time. Lab notes were shared with the ML researchers who train Claude, feeding into training data.

What's open

The whole project took under four months, run by a team with limited structural-biology experience. Anthropic claims most of it is reproducible by a typical lab with existing Claude products; only the GPU-parallel cryo-EM step needs unusual compute. The caveats are real: these vaults have so far only been produced in E. coli, not in the bacteria and phages where the genes naturally live, and the function question is untouched. Rome, brought on as an adviser, says the breadth of the findings "suggests that it is going to be a complicated function and an interesting ride."

NVIDIA has described the AI Factory Operations Agent, a multi-agent system that helps operate its internal GPU clusters. It has also released the orchestration framework as experimental open source. The post is an architecture overview: it includes no accuracy, speed or incident figures.

The problem

When training slows, the cause could lie in compute, network, hardware or scheduling. , who keep production systems running, often have to compare several dashboards to find it. NVIDIA wants agents to do that correlation behind a single chat interface.

How it works

A central runtime, built on NVIDIA's blueprint, routes each question to a specialist agent:

Job Recovery spots failed jobs and falling , the rate of useful training progress, then analyses causes across layers.

Agents reach infrastructure through servers, which expose a limited set of approved tools to a model. Teams can add their own servers to fit their schedulers and monitoring. Inference runs on locally served open models, so the system can stay on-premises.

The security design

Shell commands run in a per-session Kubernetes sandbox created by , NVIDIA's agent runtime, rather than in the main gateway. Cluster inspection takes a separate path: a server-side tool, run_kubectl, holds the read-only credentials, and the sandbox receives neither kubectl nor those credentials.

Kubernetes , which grants identities specific permissions, sets the limits. In NVIDIA's sample check, the agent can read pods and configmaps but cannot read secrets or create or update pods.

NVIDIA recommends running the agents on a separate GPU-equipped control plane, so they use no training GPUs and the production cluster only supplies telemetry. In shared clusters, each tenant's SREs see only their own namespaces, jobs and nodes.

What NVIDIA says it learned

NVIDIA reports five lessons from production, all without data:

Overlapping agent ownership creates noise.
Start read-only, because accurate investigation earns trust before automation.
Cross-layer correlation finds causes that isolated tools miss.
Precise tool definitions reduce wrong tool selection.
SREs need to see tool calls, evidence and conclusions before acting.

What's missing

NVIDIA says it tracks root cause accuracy and SRE agreement but doesn't publish either number, so its effectiveness claims can't be checked here. The post also leaves autonomy limits somewhat unclear: the text stresses recommendations, while its diagram description mentions running commands to resolve issues.

NVIDIA has described the AI Factory Operations Agent, a multi-agent system it uses to help run its internal clusters, which are its large reference-design GPU clusters. It has also released the orchestration framework as experimental open source, with a warning that APIs, configuration and deployment requirements may change between releases. The domain agents NVIDIA runs internally are a separate matter: customers must request access through an account manager.

The write-up is an architecture overview, not an evaluation. It gives no root cause accuracy, time-to-diagnosis or incident counts, so its value lies in the design choices.

The problem

When training performance drops, the cause could sit in compute, network, hardware or scheduling. , who keep production systems running, often have to work through several dashboards to find it. NVIDIA's aim is to do that correlation behind one chat interface, where engineers can ask questions in natural language, switch between investigations and audit every action.

The job is harder than summarising logs. A symptom in one layer can have its cause in another, so the system has to gather evidence across tools with different data and vocabulary, then present a conclusion an engineer can check.

The domain agents

A central runtime receives each query and routes it to a specialist. Internally, NVIDIA describes six:

is the open standard for connecting models to tools, and each MCP server here exposes a focused set of approved operations. Web search, code execution and Grafana dashboards are available as shared tools. Teams can add their own MCP servers and APIs to fit their schedulers and monitoring systems, without changing existing agents.

An illustrative incident, not taken from the article: a training job's goodput falls. The Job Recovery Agent checks the scheduler and finds nothing wrong, the Hardware Agent finds rising ECC errors on one GPU in the job, and the Observability Agent charts the errors against the slowdown. The SRE sees the tool calls, the evidence and a recommended step, then decides what to do.

How it runs

The system is a Kubernetes control-plane application. , Kubernetes' package manager, deploys the UI, the OpenClaw gateway that connects agents to infrastructure, the MCP tool servers, the inference endpoint and the runtime as services. The chat UI sends each request to , NVIDIA's blueprint for the agent runtime, which picks the domain agent. Inference uses locally served open models, so the whole system can stay on-premises. It runs across DGX H100, B200 and GB300 production clusters.

The security design is the most concrete part of the post. Shell commands do not run in the main gateway pod. OpenShell, NVIDIA's agent sandbox runtime, creates a separate Kubernetes sandbox for each session, loaded with the workspace, selected skills and a command policy.

Cluster inspection is kept apart from that sandbox. A server-side tool called run_kubectl calls a pinned kubectl binary using registered read-only credentials, and OpenShell receives neither kubectl nor the credentials. Agent-generated shell code therefore has no direct route to the cluster API. Kubernetes service accounts and , which grants identities specific permissions, define what can be inspected.

NVIDIA's sample permission check shows the result: the agent can get pods and configmaps, but cannot get secrets or create or update pods.

Deployment and multi-tenancy

NVIDIA prefers a dedicated GPU-accelerated control plane, isolated from production. It handles orchestration and reasoning, while the production cluster only supplies telemetry, so the agents neither consume training GPUs nor disrupt workloads. NVIDIA's own deployments run inference on dedicated GPU capacity and put runtime services in the existing CPU Kubernetes admin space. Other sites can use local or cloud inference, and smaller ones can use an RTX PRO Workstation or DGX Spark.

For multi-tenant clusters, scoped agents, tools, sandboxes and permissions limit each tenant. Tenant SREs see only their own namespaces, jobs and nodes, while central operations see more. OpenShell isolates each tenant's agents and tools with kernel-level restrictions.

The five lessons

NVIDIA reports five lessons from production use, stated briefly and without supporting data:

Clear boundaries: overlapping agent ownership creates noise.
Read-only first: accurate investigation earns SRE trust before any automation.
Cross-layer correlation: connecting workload, hardware and network signals finds causes that isolated tools miss.
Precise tools: clear tool definitions improve routing and reduce wrong selections.
Transparent reasoning: SREs need to see tool calls, evidence and conclusions before acting.

What the article leaves open

NVIDIA says it measures accuracy and SRE agreement, but it publishes neither figure. The claim that the system finds causes isolated tools miss therefore cannot be checked from this post, and it comes from a vendor that also sells the hardware and models.

The boundary of autonomy is also a little unclear. The text stresses recommendations and read-only access, while the architecture diagram's description includes running commands for issue resolution. The post doesn't spell out which commands can touch what, beyond the sandbox and the read-only Kubernetes path. It also doesn't cover how agents handle wrong diagnoses, which Nemotron model is used, or what inference capacity is needed at a given cluster size.

Google Research has released , an open-source framework for building image classifiers around subjective concepts such as "healthy food". It helps people work out what they mean while the classifier is being built.

The problem

Normal training assumes the labeler already has a stable definition. That works for a stop sign. With "healthy food" or a moderation policy, people often discover the boundaries only when they meet real examples, yet standard workflows demand firm labels from the start.

How it works

Scoping. The system guides the user in breaking a concept into sub-concepts and exclusions. "Healthy Food" might include salads and lean proteins but exclude processed foods.

Iteration. The system then uses , which reason over both images and text, to find images that sit on the edge of the written definition. This borrows from , where a system chooses which examples a person should label, but the aim is to challenge the rules rather than only reduce the model's uncertainty.

Take a fruit smoothie: healthy because it contains raw fruit, or processed because it's high in sugar? The user decides, and updates the VLM's instructions to match. The user reviews a gallery of images and never has to edit the prompt.

The test

Subjective concepts have no fixed , the agreed correct labels used for scoring, so Google ran 18 live sessions of 1.5 hours each. Participants tried the systems in randomized order on two tasks: Healthy Food and Pay-to-Play, which means spotting gaming or entertainment clickbait that promises misleading rewards.

The baselines were:

: the user writes one prompt and gets no chance to refine it.
: an AI agent breaks the concept down alone, with no user input.
Manual deliberation: the user searches spreadsheets and rewrites prompts by hand.

Results

Google says participants labeled more consistently over time and that their classifiers "substantially outperformed" all baselines, especially on the hardest images. Experts said the system exposed blind spots in their definitions.

Manual prompting caused marked fatigue and a persistent worry that fixing one error would break something else. Participants felt significantly less stressed and drained with Agile Deliberation, and all preferred it to manual methods.

The caveats

The post gives no accuracy figures or effect sizes, so the size of the gains can't be judged from it; the accompanying paper may have them. It also doesn't explain what classifier performance was scored against after ruling out objective ground truth. The stress findings are self-reported.

The core habit still looks useful for anyone writing LLM guardrails or eval rubrics, in our view: go looking for the cases your written policy can't decide.

Google Research has introduced , a framework for building image classifiers around subjective concepts such as "healthy food". Instead of asking people to label data against a fixed definition, it helps them work out the definition as they go. Google has released the code on GitHub, along with a paper and a notebook demo. It tested the system in 18 live sessions with people rather than on a benchmark.

The problem: the definition doesn't exist yet

Standard training assumes the labeler has a clear, stable category in mind before starting. Google's post borrows the term "" for this idealized annotator, who already holds a complete set of rules.

That assumption suits a golden retriever. It fails for "inspiring artwork" or content moderation, where boundaries vary with culture and personal philosophy and shift as policies change. People usually begin with an intuition and discover the exceptions only when they meet real examples. A conventional workflow makes them commit to rigid labels while their thinking is still changing.

Stage 1: concept scoping

Few people can list every rule for a broad concept upfront, so the system first guides the user in turning it into a hierarchy of sub-concepts and explicit exclusions.

In Google's example, "Healthy Food" might include "Salads", "Grain Bowls" and "Lean Proteins", while excluding "Processed Foods" and "Raw, Unprepared Ingredients". The result is only a starting structure, and the next stage is designed to test it.

Stage 2: concept iteration

The second stage borrows from , a method that picks the most useful examples for a person to label. Classic active learning selects examples to reduce the machine's uncertainty. Agile Deliberation has a different aim: it uses , which reason over both images and text, to find images that are semantically borderline against the current written definition. These images are chosen to challenge the rules, not simply to feed the model.

Google's example is a fruit smoothie. Is it a healthy drink because it contains raw fruit, or an excluded processed food because it is high in sugar? The written rules don't settle the question, so the user has to.

The user's answer then feeds an loop, which adjusts the VLM's instructions so its classification boundary moves with the user's refined definition. The user reviews a curated gallery and gives natural feedback without having to edit the prompt.

As an illustration of the cycle (not taken from a study session): suppose a user decides that smoothies count only when they have no added sugar. The prompt would be updated to reflect that, and the next round of borderline images would test the updated definition.

The post doesn't explain how the images are mined and ranked, or how the prompt optimizer works. It refers readers to the paper for the active learning and for the scoping and clustering algorithms.

How it was tested

Google argues that offline evaluation can't work here. Subjective concepts have no static , the agreed correct labels a classifier is scored against, and a pre-labeled dataset can't capture a person's definition changing mid-task.

So the team built a prototype and ran 18 sessions of 1.5 hours each. Participants used the system and the alternatives in randomized order to limit learning effects. There were two test domains:

Healthy Food: views vary with health philosophy, dietary restrictions and cultural background.
Pay-to-Play: a moderation task that needs expert nuance, identifying clickbait images in online entertainment or gaming that promise misleading financial or digital rewards.

The three baselines were:

: the user writes one prompt, with no way to see edge cases or correct the model's reading of the boundary.
: an AI agent breaks the concept down by itself, generating criteria automatically with no way for the user to guide or correct its logic.
Manual deliberation: the user inspects data in spreadsheets, guesses search queries to find anomalies, and rewrites prompts by hand.

What Google reports

Performance. Participants who went through deliberation labeled more consistently over time, and their classifiers "substantially outperformed" all baselines. The largest gains came on the hardest subjective images, where Google says standard classifiers fall back on generic assumptions rather than the user's intent.

Blind spots. Expert participants described the system as a diagnostic tool that exposed scenarios their initial definitions hadn't anticipated.

Cognitive load. Under manual deliberation, users reported marked fatigue from trying to predict search terms that would surface edge cases. They also worried that a prompt tweak fixing one mistake might silently break something they had already fixed. With Agile Deliberation, participants reported feeling significantly less stressed, frustrated and drained. All of them preferred it to manual methods.

How far the evidence goes

The post's only study numbers are the session count and length, the two domains and the three baselines. It gives no accuracy or F1 scores, effect sizes or significance tests. Those may be in the paper, but "drastically" and "substantially" can't be assessed from the post alone.

Three further questions stand out:

Measurement: Having said there is no objective ground truth, the post doesn't explain what classifier performance was scored against. A plausible reference would be each participant's own labels, but the post doesn't confirm that. The answer matters, because a system that helps people refine their intent could also shape the labels used to judge it.
Sample: The post doesn't say whether 18 sessions means 18 people, or how many were experts.
Self-report: The fatigue and stress results are self-reported, and participants could see which tool they were using. The unanimous preference is strong evidence about experience, not about classifier quality.

The post also doesn't report what the workflow costs in VLM calls, how many rounds are needed before a definition settles, or how a classifier holds up as policies keep changing.

What practitioners can take from it

The idea is easy to carry beyond this tool, though this is our reading rather than Google's claim. Searching for examples that the current written policy can't decide treats a disagreement as a bug in the specification, not just as a bad label. Teams writing LLM guardrails or eval rubrics face the same problem of definitions that are incomplete when work begins, and the same worry that one prompt fix may undo another.

The research was done in partnership with Google Ads, which fits the moderation use case. Google also suggests applying the framework to digital curation and scientific image categorization, though the post reports tests only in the two domains above.

OpenAI has released Images 2.6, an update to its image generator, in ChatGPT, in its coding agent and in its API. All ChatGPT and Codex subscription tiers get it today. The headline additions are animated GIFs, faster generation and more control for developers.

Speed and two models

OpenAI says Images 2.6 generates images 50% faster than before, though the post doesn't state the baseline or test conditions.

The API offers two models:

is the fast option. At low quality it can produce an image in as little as six seconds. OpenAI also says it improves layout for posters and product imagery.
is for detailed work. At medium or high quality, OpenAI says it delivers higher quality and better consistency across edits.

How GIFs appear to work

The post doesn't describe the animation pipeline, but its two sample prompts are revealing. Both ask for a , a single image containing an animation's frames in a grid. They specify exactly six frames in a three-by-two layout, identical panel sizes, a fixed camera, no borders, and a sixth frame that leads back into the first so the loop is seamless.

That makes frame-to-frame consistency the real test: the same character, desk and outfit must survive every panel. OpenAI also says a rough pen sketch can serve as the starting point for an animated character.

New API controls

, a way to mark exactly where a change should happen, moves from ChatGPT to the API. export, a scalable vector format, is meant to make designs usable across websites, apps and print.

The most interesting addition for engineers is a , which sets how much planning the model does. It is separate from quality: quality governs visual detail and fidelity, while complexity governs how much goes into the scene. OpenAI recommends low complexity for most edits and medium or high for scenes with many elements or constraints.

What's missing

The evidence here is thin. The post gives no benchmarks, side-by-side evaluations or error rates, and the consistency and layout claims rest on OpenAI's word and curated examples. It also doesn't say whether SVG export produces clean, editable vectors. Pricing and safety evaluations sit in separate documents that the post links to.

OpenAI has released Images 2.6, the latest version of its image generator, in ChatGPT, in its coding agent and in its API. It is available globally today across all ChatGPT and Codex subscription tiers on desktop, mobile and web. The update has three parts: faster generation, animated GIFs and more developer control over how images are planned, edited and exported.

OpenAI also gave a usage figure: 4 billion images per week across those three products, or more than half a billion a day.

Two models, one speed claim

OpenAI says Images 2.6 generates images 50% faster than before. The post doesn't say what it was measured against, at which quality setting, or at what image size.

In the API, developers choose between two models:

favors speed. At low quality it can generate an image "in as little as six seconds", which is a best case rather than a typical latency. OpenAI also says Flare improves layout quality for posters, product imagery and other professional visuals.
favors fidelity. At medium or high quality, OpenAI says it provides higher-quality generation and greater consistency across edits.

Consistency is the practical issue behind that second claim. Image models synthesize pixels rather than editing a stored scene, so a follow-up request can quietly alter a face, a logo or a product. For brand work, that drift can make an otherwise good edit unusable. OpenAI says Images 2.6 keeps characters, products and brand details consistent as you edit, but the post offers no measurement of it.

What the GIF prompts reveal

OpenAI pitches GIFs for animated mascots, social media reactions and memes, looping product shots, and storyboards that explore character movement before video production. It also says a simple sketch can become an animated character.

The post doesn't explain how animation is produced. Its two sample prompts, credited to OpenArt, do offer a clue. Each begins by requesting "a single animation for a seamless looping GIF". A sprite sheet is one image holding all of an animation's frames in a grid. The prompts are tightly constrained:

exactly six sequential frames in a three-column, two-row grid, read left to right and then top to bottom
identical panel dimensions
a fixed camera, with the same character, desk, monitor, outfit and background throughout
clear incremental motion from frame to frame, such as tears forming or banknotes flying
a sixth frame that transitions back toward the first, so the loop closes
no captions, labels, borders or gutters (the gaps between panels)

The no-gutters and equal-panel rules matter if software slices the grid into frames, because uniform panels can be cut mechanically. That is my reading of the prompts, not something OpenAI states. It also isn't clear whether every GIF goes through this route, or who handles the assembly step. Still, the prompts suggest that animation here may be less like video generation and more like an extreme consistency test within one image. If that's right, frame count and resolution could be constrained by how much fits in a single picture, though the post doesn't give limits.

New API controls

, already in ChatGPT, comes to the API, letting teams pinpoint exactly where a change should happen. export, a vector format that scales without pixelation, is meant to make designs ready for websites, apps and print. The post doesn't say how the vectors are produced or how clean and editable the resulting files are, which determines whether a designer can actually work with them.

The most consequential change for API users may be the new , which adjusts how much planning the model does to meet a request. OpenAI separates it from quality: quality controls visual detail and fidelity, while complexity guides how much goes into the scene.

An illustrative example, not from the post: changing a shirt's color on an existing product photo calls for little planning, while a poster with a dozen labeled items in a strict layout needs much more. OpenAI's guidance fits that split. It recommends low complexity for most edits, paired with whatever quality the workflow requires, and medium or high complexity for scenes with many elements or constraints.

This gives developers two separate settings to tune, plus the choice of model. The post doesn't quantify how complexity affects latency, cost or accuracy, so teams will need to measure that on their own workloads.

How much to believe

Judged as evidence, this is a product announcement. The only performance numbers are the 50% speedup and the six-second best case. There are no benchmarks against earlier versions or competitors, no text-rendering or layout accuracy figures, and no consistency metric. The examples are chosen by OpenAI, and its sample prompts are long and heavily specified, so results from loose prompts may differ.

The post leaves several things to other documents or leaves them unanswered. Pricing is on a separate page, and safety evaluations are in a linked . Neither is summarized in the announcement. It also doesn't describe GIF limits such as maximum frames, resolution or duration. For engineers deciding whether to adopt it, the useful next step is a small test: run your own edit sequences through Flare and Sunburst across complexity levels, then measure latency, cost and detail drift.