Evidence first, proofs still pending · Morning Gist
Thursday, October 85 stories · 11 min
Evidence first, proofs still pending
Code puts limits on agents, databases ration queries, a cheap model has costly fine print, and AI's biggest math claims still need checking.
Picked from 18 new articles across 23 sources · arrow keys for other days
Today in 30 seconds
30s left
Cloudflare's Managed Defense uses plain code to collect evidence and scope tenants, then has LLM agents interpret replayable snapshots. It gives no accuracy figures. story 1 ↓
PlanetScale's Traffic Control estimates each Postgres query's cost, then admits or blocks it against token-bucket budgets measured in server capacity. story 2 ↓
Claude Haiku 5.5 is 10x cheaper than Haiku 4.5. A new tokenizer and 5x pricing above 100K tokens reduce the real savings. story 3 ↓
NVIDIA's fine-tuned Nemotron 3 scored 535.4/600 at IOI and 30/42 at IMO 2026. The checkpoints, data and pipelines are all public. story 4 ↓
OpenAI published 722 math manuscripts from an unreleased model at about three compute-hours each. None are verified, including a faster-than-n-log-n multiplication claim. story 5 ↓
GPT-6 is rolling out in ChatGPT with "Intelligent UI", which gives faster answers with visual and interactive elements. The launch post is aimed at consumers.
1/5Cloudflare· Deanna Tran· Oct 7
Cloudflare's security agents do analysis only after code collects the evidence
Managed Defense keeps evidence collection, tenant scope and citation checks in deterministic code. Narrow specialist agents only interpret a fixed, replayable evidence snapshot.
Why it matters · It's a concrete pattern for making LLM agents auditable in production: code enforces boundaries, inputs can be replayed, and missing data is recorded. That applies well beyond security triage. The post gives no accuracy or latency numbers, though.
Failure modes of the single general-purpose agent Cloudflare first tried.1 / 4
Cloudflare has described the harness behind its Managed Defense alert-triage beta. The design is mostly about what the models are not allowed to do.
Why one agent failed
Cloudflare's first prototype put the whole investigation in front of a single general-purpose agent. Telemetry, detector descriptions, policies and threat intel all went into one prompt. The analysis was useful, but the agent made claims the evidence didn't support. The team saw three failure modes:
Context became authority. The agent treated a detection, which is only a hypothesis, as proof that an attack happened.
2/5PlanetScale· Simeon Griggs· Oct 7
PlanetScale adds leaky-bucket admission control to Postgres
Traffic Control estimates each query's cost from its plan and history. It then admits or blocks the query against token-bucket budgets measured in server capacity.
Why it matters · Postgres and MySQL will try to run every query you send them. Overload ends up enforced by whatever limit breaks first. This post explains the mechanism PlanetScale uses to reject expensive work before it starts, and what that asks of your application.
Traffic Control setting that caps how much capacity any single query can consume.1 / 2
PlanetScale's Simeon Griggs walks through the history of the leaky bucket, from Babylonian water clocks to Jonathan Turner's 1986 IEEE paper on policing packet rates. He uses it to explain Database Traffic Control, PlanetScale's admission-control layer for Postgres.
How a leaky bucket works
Each unit of work has a cost that is checked against the bucket's current level. If there isn't room, the work is rejected. If there is, the work is admitted and its cost is added to the level. The bucket drains based on elapsed time, not on when work finishes. A meter decides whether work runs right away. A queue instead holds work until it drains out.
The implementation is cheap. Nothing runs in the background to do the draining. A bucket stores only two values: its last level and when that level was recorded. The current level is computed when someone checks it:
The post points to Shopify's GraphQL Admin API as a working example. Each request is charged by query complexity, not by request count. Shopify reserves the estimated cost up front, then refunds or charges the difference after the query runs. Responses show the bucket's state, so clients can slow down before they are throttled.
The Postgres gap
Griggs argues that a common production failure is a database that can't deliberately turn work away. Some hardware or connection limit ends up acting as the admission control, and it fails in ways nobody planned for. Oracle (1999) and SQL Server (2008) have resource governors, but those throttle work rather than reject it. Postgres has settings like , but these only act once a query has already used resources.
3/5Simon Willison· Oct 7
Claude Haiku 5.5 matches GPT-6 Luna's price, but only up to 100K tokens
Haiku 5.5 is 10x cheaper than Haiku 4.5 and always reasons. A new tokenizer and a 5x long-context price jump erode part of that saving.
Why it matters · If you pick a cheap model for high-volume pipelines, Haiku 5.5's sticker price and its real cost can differ a lot. That depends on prompt length and the new tokenizer, so check against your own workloads before switching.
Haiku 5.5's pelican at max reasoning: 5 minutes 9 seconds and 3.38 cents, with a correct bicycle frame.1 / 3
Anthropic has released Claude Haiku 5.5, its fast, low-cost tier, about a year after Haiku 4.5. Simon Willison's write-up is mostly about pricing and his usual hands-on test. He gives no benchmark figures and only notes that Anthropic reports higher scores than OpenAI's GPT-6 Luna.
Pricing is the real story
Haiku 4.5 cost $1 per million input tokens and $5 per million output. That was 10x the price of GPT-6 Luna, which OpenAI released last month. Haiku 5.5 matches Luna exactly at $0.10/$0.50, but only up to 100,000 tokens. Past that point, the price rises 5x to $0.50/$2.50. Luna's price also goes up for long contexts, but it starts later (272,000 tokens) and rises less (to $0.20/$0.75).
There is also a hidden increase. Haiku 5.5 uses a new, less generous tokenizer. Willison's Claude Token Counter shows the same long prompt using about 1.25x as many tokens as on Haiku 4.5. His verdict: under 100K tokens, Haiku costs the same as Luna and benchmarks higher. Above that, Luna is the better deal.
Reasoning is mandatory
Haiku 5.5 has five reasoning levels: low, medium, high, xhigh and max. Reasoning can't be turned off, and the default is medium. Haiku 4.5 had no reasoning levels at all. Willison tested through his CLI. The latest llm-anthropic plugin now refreshes its own model list (), so he no longer has to release a new plugin version for each model.
4/5Hugging Face· Oct 7
NVIDIA Fine-Tunes Nemotron 3 to Gold-Level Scores at IOI and IMO 2026
Ultra-CC scored 535.4/600 at IOI, above the top human. A generate-verify-refine proof system scored 30/42 at IMO. NVIDIA released the checkpoints, data and pipelines.
Why it matters · This is a concrete, reproducible case that ordinary SFT and RL on an open base model, combined with a feedback-driven inference loop, can reach top competition performance. The weights, data and pipelines are public.
Headline IOI and IMO 2026 scores compared with gold-medal thresholds.1 / 3
NVIDIA used one base model family, Nemotron 3, to build systems that reached gold-medal level at two very different competitions. IOI requires code that passes hidden tests under time and submission limits. IMO requires rigorous natural-language proofs.
IOI 2026: Nemotron-3-Ultra-CC scored 535.4/600. The gold threshold was 361.12 and the top human scored 498.27. The run was live, under contestant rules for time, internet access and submissions. It was an unofficial benchmark and did not appear in the official ranking.
IMO 2026: The system scored 30/42, one point above the gold threshold of 29. It got full credit on four of six problems, and official IMO graders marked the proofs.
The recipe
NVIDIA describes four steps: start from a strong base, curate domain problems and reasoning traces, apply standard SFT and, where useful, RL, then wrap the model in an inference loop that generates, evaluates and refines answers.
Competitive programming. The team curated 22,000 problems and generated synthetic reasoning traces for them. Two models were trained on this data:
5/5Latent Space· Oct 7
OpenAI publishes 722 math manuscripts from an unreleased internal model
About three hours of ChatGPT Pro compute per result, drawn from ~4,000 problems. Claims include quasi-Riemann and faster-than-n-log-n multiplication. None is verified yet.
Why it matters · If even part of this holds up, cheap, long-horizon reasoning on verifiable problems has reached research grade. That bears directly on what coding agents can do next. The open question is whether the gains carry over to work that can't be checked automatically.
Image: Latent Space
OpenAI has put a large set of mathematical results from an unreleased internal frontier model into a public GitHub repo. The release contains papers, proof artifacts and selected reasoning summaries, but not the model. OpenAI says it asked the Institute for Advanced Study's independent Advisory Group on Mathematics and AI how to release the work.
The numbers
722 manuscripts, grouped into 372 families of related results
Taken from an evaluation run over roughly 4,000 research problems
About three hours of ChatGPT Pro thinking compute per result on average
Latent Space says the results cover about 90 of the top 500 open problems in math. Compute is the striking contrast. The lab's earlier Navier-Stokes result took 88 hours and 10,000 agents. These results took a few hours of a commercial product tier each. Teortaxes notes that three hours "is not much."
Claimed results (not independently verified)
Result 003, the Quasi-Riemann Hypothesis. Latent Space places it somewhere between Fields Medal level and "the biggest result in number theory in 200 years."
An integer multiplication algorithm faster than n log n.
A uniqueness result for the elastic inverse problem, which the paper says had been open in 3D since 1994.
Scope drifted. The agent queried the wrong account, time range or source. A prompt can't serve as a boundary.
Failures disappeared. When a lookup timed out, the output didn't separate "not checked" from "checked, nothing found."
How the harness works
The fix was to move evidence collection and scope enforcement into application code.
Recon. Before any model runs, deterministic code runs fixed reconnaissance workflows using versioned API calls. They collect customer identity, detection history, traffic baseline, enforcement outcome and network observations, and every item is stamped with its source, version and timestamp. Because the snapshot is fixed, runs can be replayed. Any difference between agents' findings then comes from interpretation, not from fetching different data.
Triage. Known high-volume noise is classified as passive on arrival. Everything else goes to Clef, Cloudflare's open-source decision model running on Workers AI. Clef checks the alert against its recon data: has this fired for the customer before, what did analysts decide last time, and does the traffic look like normal human behavior? Alerts it scores as likely false positives skip deeper analysis.
Specialists. For alerts that need deeper review, a coordinator runs four agents in parallel:
traffic analysis
customer context
global telemetry, which sees only privacy-preserving aggregates and never another customer's records
threat intel, limited to indicators already admitted to the alert or case
A synthesis agent merges their typed findings. It can't fetch new evidence or pick a classification outside an approved vocabulary. Deeper analysis uses models including GPT-5.6 Cyber and Anthropic's Mythos.
Validation. Specialists must cite items in a versioned evidence package. Code checks that each citation exists, belongs to this investigation and supports its claim. Invalid findings are corrected or recorded as limitations. Clef then judges whether the evidence is sufficient or contradictory and picks from a reduced list of classifications. If the evidence is insufficient, the system makes no recommendation. The advisory also separates three states: not checked, checked with no match, and checked with evidence of absence.
The pipeline runs on Workers, with Workflows checkpointing each stage so a failed stage reuses evidence that already passed validation. State lives in D1, artifacts in R2, and case chat in Durable Objects. Advisories can suggest specific WAF, rate-limiting or DDoS rules, but a human analyst makes the final decision. The post reports no accuracy, false-positive or time-saved figures.
Remember
Evidence collection and tenant scoping run in deterministic code before any model sees data, so runs can be replayed.
Specialist agents must cite items in a versioned evidence package, and code rejects any citation the evidence doesn't support.
Missing data is recorded explicitly, and with insufficient evidence the system makes no recommendation. Cloudflare gives no performance numbers.
Traffic Control's resource budgets match queries by tags. Tags can come from SQL comments, the database user, or the application. Each budget has three main settings:
Server share and burst limit. This is a token bucket. Server share is the refill rate, as a percentage of server capacity. Burst limit is the bucket's capacity, in whole-server seconds. Measuring in capacity rather than wall time means a query that uses 8 cores for one second costs more than one that uses a single core. It also means a 25% share means the same thing on a 2-core or a 64-core machine.
Per-query limit. A cap on how much a single query can consume, regardless of what's left in the bucket.
Maximum concurrent workers. A cap on a single query measured in workers instead of execution time.
Postgres can't know a query's cost in advance. Traffic Control estimates it from the query plan and from how queries of the same shape have behaved before, then charges or refunds the difference once the query finishes. A query has to pass every budget that matches it. Budgets can either log a warning or block. PlanetScale recommends starting in warning mode so you can see how much traffic would be affected and prepare the application for blocked queries.
The post is partly product marketing. It gives no performance or overhead numbers.
Remember
A leaky bucket only stores its last level and a timestamp, so checking it costs almost nothing.
Traffic Control charges queries an estimated cost up front, then settles the difference after they run.
Budgets are measured in server capacity, so a 25% share means the same thing on 2 or 64 cores.
On his pelican-on-a-bicycle SVG test, every level above low drew a correct bicycle frame. Low took 7 seconds and cost 0.0936 cents. Max took 5 minutes 9 seconds and cost 3.3826 cents. The max run's reasoning trace began "This is the classic pelican-on-bicycle SVG test," which shows the model knows the benchmark. Willison responded by also running a new prompt: an armadillo in fishnet tights jaywalking on Mars. A year ago, Haiku 4.5 drew a poor pelican for 0.7583 cents.
Subscription API credits
On the same day, Anthropic halved cache-read prices for Sonnet 5.5. It also added monthly API credits to subscriptions: $100 for Max 5x, $200 for Max 20x, and up to $500 pooled for Team. To claim them, go to Settings → Billing and choose an API organization. Unused credits don't roll over. If you turn off auto-reload, requests stop when the balance runs out, so you can't overspend. Willison calls the scheme generous because the credits equal the subscription price. He adds that OpenAI still lets Codex subscribers use their subscription for personal API use, which remains the better deal for heavy users.
Remember
Haiku 5.5 costs $0.10/$0.50 per million tokens up to 100K, then $0.50/$2.50. Luna's long-context price stays far lower.
A new tokenizer uses about 1.25x more tokens per prompt, a hidden price increase over Haiku 4.5.
Reasoning can't be turned off. It has five levels, and the test cost ranged from 0.09 cents (low) to 3.4 cents (max).
Nano-CC (30B total, 3B active parameters) received SFT and RL.
Ultra-CC (550B total, 55B active) received SFT only.
On IOI 2025, Nano went from 130 points to 280 after SFT and 291 after RL. Adding GenCorrect, an iterative generate-evaluate-refine loop, took it to 468, past the 438.3 gold line. Ultra-CC reached 502 with the same loop. SFT produced most of Nano's gain. A single SFT epoch on Ultra beat the fully post-trained Nano on IOI, ICPC and LiveCodeBench Pro, and that result shaped the 2026 entry.
Olympiad proofs. Starting from Nemotron 3 Ultra, the team trained two specialists:
SFT model: 414,890 quality-filtered examples across 15,818 proof problems. The data covered proof generation, refinement, verification and meta-verification, so the model learned to find gaps and judge whether a proof was complete, not just produce answers.
RL model: 9,597 problems chosen near the edge of what the model could solve.
The SFT checkpoint was strongest in the first search round. The RL checkpoint had the best overall single-checkpoint result. Because their strengths differed, the final system used both alongside the general model. For each problem, the models generated proofs, scored them, wrote critiques and refined the best attempts. A separate high-compute stage then picked the submission. The system had no formal prover, tools or internet access.
The claim
The authors argue that neither fine-tuning nor brute-force sampling alone produced the medals. Specialization produces better candidates and better critics for the search loop. At IMO, using two complementary checkpoints helped more than drawing more samples from one. The post does not give total compute figures.
NVIDIA released the IMO SFT and RL checkpoints, both training datasets, a new 200-problem benchmark (Nemotron-IMO-Bench), and Ultra-CC. Both inference pipelines, plus the IMO prompts and submitted proofs, are in the NeMo-Skills repository.
Remember
Ultra-CC (550B/55B active) scored 535.4/600 at IOI 2026, above the top human's 498.27, with one SFT epoch plus GenCorrect.
IMO system used complementary SFT and RL checkpoints in a natural-language generate-verify-refine loop and scored 30/42.
On IOI 2025, SFT drove most of the gains (130→280) and test-time refinement added the most after that (291→468).
Partial progress on Riemann, Hodge and BSD, according to commenters.
Levent Alpöge, who worked on Anthropic's competing Navier-Stokes effort, praised the quasi-Riemann and no-Siegel-zeros results. He called the release "the most significant moment in mathematical history." He also flagged reported scooping and conflict-of-interest problems involving users of other labs.
One analysis estimates that about 20% of the results are disproofs or counterexamples. It argues this undercuts the view that AI math wins are mostly brute-force search.
Skepticism
Will Depue expects some results not to survive scrutiny. He built citedbyagi.com to track which human papers the release cites. François Chollet asks the structural question: do gains in math and code generalize? Those are the domains that suit reinforcement learning with verifiable rewards (RLVR). Chollet's alternative is that domains without automatic checks stay limited by human data.
Also in the issue
Mistral Large 4 "Le Chonk" is a natively multimodal mixture-of-experts model with 1T total and 49B active parameters, trained on ~3,800 Grace Blackwells. Pricing is $1.36/$4.18 per million input/output tokens. Open weights are promised for the end of October, and Mistral says the RL run is still not saturating. It scores 38 on Artificial Analysis's Intelligence Index, level with GPT-6 Luna (max). It costs $1.13 per task, more than 4x comparable open models. Cline attributes its lead on cyber evals largely to fewer refusals.
In tooling, Mitchell Hashimoto published OSC 7501, a terminal spec that lets programs report their status. More than 250 agent orchestrators currently guess with heuristics whether tools like Claude Code are working or blocked. Separately, an audit of AutomationBench found 206 verifier bugs. Fixing them changed 27.9% of grades across 1,235 runs.
Remember
722 manuscripts from ~4,000 problems, about 3 hours of ChatGPT Pro compute each. The model itself stays unreleased.
Headline claims (quasi-Riemann, sub-n-log-n multiplication) are unverified. Critics expect some results to fail review.
About 20% are disproofs. The open question is whether gains in verifiable domains generalize.
