Agents Meet Gatekeepers, Git and Factory Limits

The problem

Suppose a customer asks an agent to compare products on your site, find a support article, or complete a purchase. To the site owner, Cloudflare says, all three look like automated traffic. Without knowing which agent is visiting or what it is doing, the owner can't easily tell whether that traffic helps the business or creates risk.

The obvious defence is to block all of it. Cloudflare's argument is that this can lose real customers, because a person is often behind the request. It is also selling the alternative, so that argument isn't neutral, but the trade-off is real. An automated request can no longer be assumed to be an unwanted one.

What site owners can now see

The second change is a Personal Assistants widget in the Top statistics view. It lists which agents are visiting and how many requests each sends. Filtering on one agent updates the other widgets to show only its traffic:

• Paths: which pages it requests.
• Status codes: the numeric results the server returns, such as 200 for success or 404 for not found.
• Networks: where its traffic comes from.

Together, these can show whether an agent is logging in, reading product pages, reaching checkout, or hitting errors on pages it can't use.

Seeing which account an agent acts for

• An overview page shows which agents are active across users and how many accounts use them.
• Every login and sign-up event records whether an agent made the request, and which one.
• Fraud teams can filter accounts by AI assistant events or by number of AI assistants.
• Each account's page lists the agents acting for it.

The account view helps with investigations. An analyst looking at an unusual login can check whether it came from an agent the customer has used for weeks or one that appeared today.

Setting the terms

Earlier this year Cloudflare introduced AI bot policies, with separate choices for bots used for Search, Training and Agent behaviours. The Agents policy now extends to individual supported agents, and the setting has been renamed AI Crawler and Agent access policies. Existing Search and Training choices don't change.

In Security Settings, an owner sets an overall Agents policy, then can override it per supported agent. There are three options: Allow (do not block), Block on pages with ads, or Block.

Here is an illustrative example, with invented agent names. A publisher sets the overall policy to Block on pages with ads, then sets Agent A to Allow. Agent A can read everything. Agent B, which has no override, is blocked from ad-supported articles but can still reach pages without ads.

The ads option reflects a publisher's business concern. An agent that reads a page for its user may never show that user the ads paying for it. Cloudflare doesn't make this argument at length, but it does say the same agent could reasonably be allowed by one publisher and blocked by another.

What the post leaves open

This is a product announcement, and it contains no measurements: no agent traffic volumes, no accuracy figures, no list of supported agents. It also doesn't explain how Cloudflare recognises an agent.

It also doesn't say how Cloudflare determines which pages carry ads.

Cloudflare itself says identity is only part of the problem. A site may recognise the agent but not know whether the user is an existing customer, what they authorised the agent to do, whether they subscribe, or which protections should apply.

The company says closing that gap will need privacy-aware ways to carry user context through an agent, to help sites assess reputation, and to establish accountability when something goes wrong. It offers no design or timeline, only a promise to share more as it works with agent operators and others in the industry.

The announcement is short and contains no performance numbers or technical design. What follows separates what OpenAI says from the general background needed to make sense of it.

The problem OpenAI describes

That gives two distinct bottlenecks: machines storing and serving code, and people understanding what changed. Pierre's products touch both.

What Pierre makes

Performance matters here for a mundane reason. Interfaces can slow down when they must display a change spanning thousands of lines or a project with a huge file list. A review tool that stutters on big changes gets worse just as agents start producing more of them. That is general background, and the article doesn't explain how Pierre's libraries handle it.

Edit mode in Diffs lets a reviewer fix a line of code right where they spot the problem. Without something like it, a reviewer would typically leave a comment asking for the fix or switch to an editor and make the change there. OpenAI describes this as putting "an editor inside a diff."

What OpenAI says it will do

OpenAI makes two specific commitments:

• Pierre's open-source projects will remain open source, and OpenAI will continue contributing to them.
• Customers can keep using Code Storage after closing, while OpenAI works to bring its features and functionality into Codex.

The rest is ambition. OpenAI says it wants to build on existing Git workflows and adapt them to teams working with agents, covering the span "from the first branch an agent creates" through to deployment. It says it hopes to "set the standard for modern software development" and wants Codex to connect the code a company ships with the decisions its people make. No product, date or mechanism accompanies those lines.

Reading between the lines

Three things stand out, and these are our reading rather than OpenAI's claims.

First, this deal is partly about a team and code OpenAI already relies on. Trees and Diffs are in production in Codex, so the integration story is less speculative than in most acquisitions.

Second, the deal extends OpenAI's reach into the layer below the model. Between Pando for its own backend and Pierre's review tools, OpenAI is investing across the path code travels. That puts it closer to the territory of code-hosting platforms, though the announcement names no competitors.

Third, the commitments leave gaps. "Remain open-source" says nothing about how much investment each project will get, or whether future features will land in the open libraries or only in Codex. The Code Storage wording promises continued use but sets no end date or long-term guarantee, and moving its features into Codex raises an obvious question for customers who use it as neutral infrastructure inside their own apps. Anyone building on it should treat its future as an open question until OpenAI says more.

What's missing

The article doesn't say how much OpenAI is paying, how large Pierre's team is beyond calling it "small," when closing is expected, or how Pierre's work would connect to Pando. It also offers no evidence on the central premise, such as measurements of how agent workloads strain Git or of how review tools affect the quality of agent-written code. For those answers, the Pando write-up OpenAI mentions is the more likely place to look.

The problem: the definition does not exist yet

It breaks down for subjective categories such as healthy food, inspiring artwork or content that breaks a moderation policy. These boundaries vary with culture and personal philosophy, and policies change over time. The authors make a further point: people rarely hold a complete definition even privately. They start with an intuition and discover the exceptions when they meet real examples. A standard workflow demands firm labels while the person's own idea is still moving.

Writing a prompt doesn't remove the problem. With vision-language models (VLMs), AI models that process images and text together, you can create a classifier by describing the category in words. But the description still has to anticipate cases you haven't thought of yet.

Stage 1: concept scoping

The first stage turns a vague idea into a structure. The system guides the user to break the concept into a hierarchy of sub-concepts and explicit exclusions. For "Healthy Food," a user might list salads, grain bowls and lean proteins as positive sub-categories, and exclude processed foods and raw, unprepared ingredients. This gives the model a starting baseline and gives the user something concrete to test.

Stage 2: concept iteration

Take a fruit smoothie shown against the healthy-food definition above. The rules so far don't settle it. It could count as healthy because it contains raw fruit, or as an excluded processed food because of its sugar. The user has to decide, and in deciding may surface an assumption they hadn't noticed. As an illustration (this example is ours, not the article's), they might add a rule that blended whole fruit counts as healthy but drinks with added sugar do not.

As an analogy, this resembles case law more than legislation. Instead of trying to write a complete statute upfront, you rule on hard cases, and the rule sharpens with each ruling. The post leaves the detailed scoping and clustering algorithms to the accompanying paper.

How it was tested

The study covered two domains:

• Healthy Food: views differ with health philosophy, dietary restrictions and cultural background.
• Pay-to-Play: a real content moderation task requiring expert nuance, which involves spotting clickbait images in online entertainment or gaming that promise misleading financial or digital rewards.

Agile Deliberation was compared with three baselines:

Results

The findings fall into three groups, all described qualitatively in the post:

• Consistency and performance: participants labeled more consistently over time, and their classifiers outperformed all the baselines. The largest gains came on the hardest subjective images, where the authors say default classifiers fall back on generic assumptions rather than the user's intent.
• Blind spots: expert participants described the system as a diagnostic tool that exposed scenarios they hadn't anticipated in their initial definitions.
• Fatigue and anxiety: in the manual process, users tired from trying to predict search terms that might uncover edge cases. They also worried that fixing one mistake in a prompt would quietly break something they had already fixed. Participants using Agile Deliberation reported feeling less stressed, less frustrated and less mentally drained, and all preferred it to manual methods.

The regression worry will sound familiar to anyone who has patched a prompt: one long block of text controls every decision, so a local edit can have global effects.

What to be cautious about

The post contains no accuracy numbers, effect sizes or statistical tests, so "substantially outperformed" can't be judged from it. Eighteen sessions is a small sample, and the post doesn't say how many distinct participants took part or how many were experts. The fatigue and stress results are self-reports from people who knew which tool they were using.

The article uses "Astra", "GPT-6" and "GPT-6 Astra" and appears to mean the same model throughout, though it never says so outright.

Why hardware is a different problem

Software is forgiving in ways hardware isn't. A code change can be compiled and tested in minutes. A physical part can take weeks to come back from a factory, and a mistake may mean paying for another batch.

The work is also split across disciplines. OpenAI's framing is that engineers must combine mechanical designs, electronics, software and manufacturing requirements, each in specialized tools. A general example of the coupling: move a connector on a circuit board and the opening in the case may have to move too. The hard part is often less any one design than keeping all of them consistent.

The device test

The model first defined the functional requirements, meaning what the device must do. It then worked in four areas:

OpenAI gives no comparison with how long a conventional team would take, so "under a week" has no baseline. It also doesn't describe how the model drove the design tools, or how much the two staff steered it through prompts.

Where it fell short

Tolerances are the allowed variation in a manufactured dimension, since no machine cuts exactly to the number on a drawing. An illustrative example, not from the article: a 3.00 mm pin must enter a hole specified at 3.00 mm with a tolerance of ±0.05 mm. Some holes will come out at 2.95 mm and the pin won't fit, even though the digital model looks perfect. Assembly details raise a similar problem, because a part can fit on screen and still leave no room for a hand or tool to install it.

These failures are significant because this knowledge sits outside the geometry. It depends on a specific factory's capabilities. The article doesn't say how many rounds were needed, what the errors were, or whether the "no direct human edits" condition survived them. OpenAI says the gaps show what future models must improve.

The numbers

The article doesn't explain how the score is calculated, what a score means in practice, or which tasks drove the gain. The newer model also isn't publicly identified. OpenAI itself notes that design is only part of the engineering lifecycle, and it calls the improvement early evidence. Notably, the weakness exposed by the device test, factory awareness, isn't among the skills the article lists for this benchmark.

Trials with outside engineers

Chip Ganassi Racing, a motorsport team, is testing workflows with OpenAI that turn 3D scans of existing parts into editable CAD models and 2D manufacturing drawings. This helps when the original design files are unavailable. A rough analogy: a scan is like a photograph of a printed page, while a CAD model is like the word-processor file. Both show the same content, but only one is easy to change. The team's engineers review the results for accuracy and manufacturability.

As an analogy, the simulator plays a role similar to a test suite for a coding agent: it gives the model a score it can check and try to improve. That comparison is ours, not OpenAI's. It may also explain why simulation suits this kind of work, though simulations only approximate the real world.

What remains open

Neither collaboration has published results; OpenAI promises more from Mazda in the coming months. The report is also explicitly a recruiting call for engineers, firms and software vendors. No independent evaluation appears in it. The useful signal is narrower than the headline capability: a model can reportedly draft a whole small product across disciplines, while the knowledge that makes a design buildable still needs experienced review.

Why reading a page isn't enough

Take a batch of supplier invoices arriving every morning, where you need the supplier, invoice number, due date and total from each. OCR returns every string on the page, with no indication of which number is the total. Someone still has to write code that assigns each value to the right field before it reaches a database.

Two ways to ask

The Field Extractor has two request modes:

Here is an illustrative example of the idea; the article doesn't show its exact request format. Ask for supplier (string), total (number) and po_number (string) from an invoice with no purchase-order number, and you might get:

{"supplier": "Acme Ltd", "total": 412.50, "po_number": null}

Returning null matters because a confident invented value could flow straight into a downstream system unnoticed.

How it was built

How the benchmark scores

The test covers 546 documents in three scenarios, for 1,638 cases: extract everything with no schema, fill a partial schema naming only real fields, and fill a schema that also includes deliberately nonexistent fields. The third scenario tests whether a model will make up values.

The score multiplies three numbers:

An illustrative calculation: a model with an F1 of 0.80, similarity of 0.95 and 98% valid JSON scores 0.80 × 0.95 × 0.98 ≈ 0.74. Multiplication means weakness in any one area pulls down the whole score.

The results

The Field Extractor's 0.834 compares with 0.816 for Claude Opus 5, the closest, and 0.741 for GPT-5.6 Luna, the furthest. That is a gap of 1.8 to 9.3 points. The article's text doesn't give every rival's score or name the fourth configuration.

What to keep in mind

The benchmark is the vendor's own, and the 1.8-point lead comes with no confidence interval or independent replication. Only SROIE and FUNSD are described as held out, and the article doesn't say how the rest of the benchmark was separated from training data. The two showcased examples are chosen by the company.

The scoring formula also deserves scrutiny. Under standard Levenshtein normalization, "1,000.00" and "7,000.00" are 87.5% similar, yet that one-digit error could be costly in finance. The article doesn't explain how heavily such errors count overall.

Rivals' low reasoning setting may also affect their quality, though the article doesn't test higher settings. The cost comparison sets list prices against raw GPU time. It doesn't mention staffing, idle capacity or redundancy, and Dharma-AI itself says the figure is not a sales price and depends on hardware, volume and latency.

Dharma-AI argues that on a bounded task, a small model trained close to the job beats a frontier model asked to do it in general. This result supports that view, but it comes from one company's test. The practical next step is to run your own documents and error tolerances through the same comparison.