Reverse-Engineered Meters and Leaky Watermarks

To measure the rates, SemiAnalysis ran experiments that each maximize one token type:

• Input: War and Peace excerpts with a random tag on every call, so nothing gets cached.
• Cache write: the same prompt marked for caching.
• Cache read: a fixed tag, so every call after the first reads the cache.
• Output: long technical essays.

Meters move in whole units, so they counted the tokens between meter ticks ("steps"), dropped the partial first and last steps, and kept going until the range was within ±5%. Cache reads count as free if the meter hasn't moved after 500M tokens. Rates on the 5-hour and weekly meters were then converted into a monthly token count and priced at API rates, using SemiAnalysis's own September agentic usage mix as the workload.

The method was sensitive enough to catch a quiet A/B test: one of three identical accounts had about 20% lower limits. The provider confirmed it was an "extremely tiny" experiment.

What the numbers show

• Top tier: Fable 5.1 and GPT-6 Astra are similar. But Fable can only use 50% of a Claude plan's limit, so the $200 Claude plan still has half its limit left after $2,485 of Fable. The $200 OpenAI plan is used up after $2,897 of Astra.
• Mid tier: Opus 5.5 gives about 5x the API-equivalent value of GPT-6.1 Sol. The gap stays large when you count raw tokens, even though Sol is cheaper per token.
• OpenAI's cut: OpenAI halved token limits on its $200 plan. Sol's value fell more than 50% because OpenAI also cut Sol's cached-input price. Existing plans keep the old limits until October 29. The new $500 plan gives only 21% more Astra than the old $200 plan, and its main selling point is 300 TPS Ultrafast. All Pro tiers now give the same value per dollar.
• Price cuts: Opus 5.5 cut input and output prices by 20% and cache reads by 60%. Anthropic raised Opus limits by about 20% on Max and about 50% on Pro, which didn't fully make up for it.

The two labs are cutting subsidies in different ways. Anthropic gives less value on its premium models. At 20% utilization and 92% API margins, SemiAnalysis estimates subscription gross margins of 6% for Opus 5.5 and 80% for Fable 5.1. OpenAI instead cut every model down to Fable-level limits at once. Chinese labs' plans are still subsidized, slightly below OpenAI's ~12x value per dollar. Third-party wrappers such as Cursor and Devin give less than first-party plans.

Results:

• Claude Fable 5.1 (xhigh): 45.3% pass rate, $40.61 per task
• GPT-5 Astra (xhigh): 32.5%, $52.37
• Claude Opus 5 (xhigh): 30.5%, $46.31

Clark calls this a "warning shot" for automated science, comparable to what 2026 has shown for automated AI R&D.

SynthID Bio

Google DeepMind's SynthID Bio watermarks AI-designed biology. The method depends on the data type: it chooses specific amino acids in sequences and adjusts atomic coordinates in predicted 3D structures. DeepMind ran wet-lab tests on three targets: VEGF-A, the SARS-CoV-2 spike RBD and PD-L1. Watermarked designs matched unwatermarked ones on hit rate, binding affinity and natural sequence diversity. Clark says watermarking only helps alongside monitoring of physical manufacturing equipment and classifiers run by AI providers.

A market for AI science

A DeepMind paper argues that AI scientists will be limited by physical resources and empirical validation, not by ideas. It proposes an Automated Scientific Economy with four parts:

1. Proof of ideation: records who had an idea first.
2. Ex-ante evaluation: agents stake compute credits on forecasts of whether a proposal is sound.
3. Brokerage: ideas are licensed in fractions to agents or institutions that run labs.
4. Validation payout: royalties go automatically to the idea's author when it is validated in the physical world.

The issue also covers a CSAIP poll (n=2,498). It found that 61% of Americans, including 53% of Trump voters, consider the labs' voluntary self-governance commitments "not enough."

Same task, four runners

The post runs Terminal-Bench 2.1's regex-log task in each framework:

• Harbor: its oracle agent runs the reference solution through the verifier without calling a model. This checks the task itself before you test a model on it.
• Verifiers: runs the same task directories with a model, a Docker runtime and a minimal bash harness. It needs Harbor's registry.json conventions, so a bare Hub repo ID won't work.
• OpenEnv: runs OpenCode in Docker and returns the reward with the agent's trace. A reward of None means the verifier produced nothing, so check the error before blaming the model.
• NeMo Gym: collects trajectories and rewards for RL training. Its Structured Outputs verifier rewards schema adherence only, not factual correctness.

Hugging Face has opened tagging PRs on existing environments, including:

• Multi-SWE-RL-Verified: 2,232 of 4,703 rows that pass gold-patch validation across six languages.
• Scale-SWE-Verified: 17,202 of 20,181 Python tasks with a clean end-to-end reward.
• Workplace Assistant: 26 tools, five databases and 690 business tasks.
• Harbor-Mix: 100 hard tasks.

Planned next: per-config snippets, structural detection for frameworks with strict layouts, and automatic tagging on upload, which OpenEnv already does.

The authors argue that LLMs can, for the first time, close the gap between high-level norms and low-level permissions. Their example is "protect my data while organizing my travel for a conference." Turning that into rules about what to share with airlines, visa offices and organizers was never practical to write by hand. They propose a contextual policy engine inside a supervisor layer. It would generate policies in real time, tailored to each request and to tools discovered at runtime, and check each data flow before anything leaves the user's workspace.

The rest of the stack

The report lists open problems at five levels:

• Sandboxing: limiting capabilities dynamically rather than statically, giving agents identities, and revoking access when the context changes.
• Models: resolving underspecified prompts and reasoning about norms as they shift.
• User controls: replacing "Notice and Choice," which assumes users can foresee what the agent will do.
• Multi-agent guardrails: stopping agents from colluding.
• Governance: sharing norms across agents, resolving conflicts between them and verifying compliance.

On evaluation, the authors call for open-source multi-agent "Agent Gym" sandboxes that simulate long chains of interactions where effects cascade. The report doesn't address how reliable an LLM-generated policy engine would be, or how it would resist the same prompt injection it is meant to guard against.

Output quality

OpenAI ran its new frontier model, Astra, at max settings with and without the watermark. It reports no meaningful difference. Every gap is within about 3 points, and the sign varies: GPQA Diamond went from 94.44% to 93.94%, DeepSWE v1.1 from 72.80% to 71.68%, and Terminal-Bench 4.0 from 53.90% to 56.06%. The table shows no variance or number of runs, so it suggests rather than proves that the watermark costs nothing.

What a detection doesn't mean

OpenAI is explicit about the limits. A positive result does not measure how much a human contributed, establish ownership, identify the user, or say whether the text is accurate. A negative result does not prove a human wrote it, because the text may be short, edited, translated, from an unsupported model, or from another vendor. Images and audio keep public verification through C2PA Content Credentials and SynthID, which survive metadata stripping, plus openai.com/verify.