Netflix fixes identity and health across its fleet, Cloudflare makes trust and policy programmable, and Stripe offers to keep your Postgres in sync.
Picked from 60 new articles across 34 sources · arrow keys for other days
Today in 30 seconds
30s left
Netflix built an RDF knowledge graph in QuipuDB so telemetry, alerts, deploys and incident chat agree on what a "service" is. story 1 ↓
Netflix's automated GPU repair keeps nodes above 99.95% available and lifted job success from about 98% to above 99.5%. story 2 ↓
Cloudflare will grade verified bots on whether they keep obeying robots.txt and back off after 429s, and plans to publish the resulting trust bands. story 3 ↓
Stripe's public preview copies payment data into your Postgres within seconds, but it doesn't document ordering or consistency guarantees. story 4 ↓
Cloudflare Gateway can now pass traffic through a Worker as inline middleware and use values the Worker computes in firewall and data-loss rules. story 5 ↓
1/5Netflix Tech Blog· Netflix Technology Blog· Oct 16
Netflix builds a shared knowledge graph to link its observability tools
Netflix built a typed knowledge graph that connects its monitoring, alerting, deployment and incident tools under one shared vocabulary. The aim is to cut most of a typical four-hour investigation, which goes into joining data by hand.
Why it matters · Netflix argues that slow root-cause analysis is mostly a data-modelling problem: tools disagree on entity identity. That diagnosis applies to any large engineering organisation.
Know this first
Context from the editor, not the article · 23 sec
1 · MELT and its limits
Observability data usually comes as metrics, events, logs and traces (MELT). Each kind is stored in its own tool. During an incident, engineers join these sources by hand, often using naming conventions that only a few people know.
2 · RDF and triples
RDF is a W3C data model that stores facts as triples: subject, predicate, object, for example api-gateway ownedBy TeamBedrock. OWL defines the types and relationships, SHACL validates data against shape rules, and SPARQL queries the resulting graph. Together they work like a schema and query language for linked data.
Timeline of a typical four-hour investigation, mostly spent gathering context.1 / 4
2/5Netflix Tech Blog· Netflix Technology Blog· Oct 14
How Netflix finds failing GPU machines and repairs them automatically
Netflix built layered health checks, graded automatic repair and a checkpoint-and-retry policy into its Kubernetes platform for GPU machines. GPU nodes stay above 99.95% available, and job success rose from about 98% to over 99.5%.
Why it matters · It shows fleet-health practice that applies to any lockstep distributed workload: monitors that report failure when they can't run, repair matched to the fault, and blaming the right layer before retrying.
Know this first
Context from the editor, not the article · 35 sec
1 · Gang-scheduled distributed jobs
Large training jobs split work across hundreds of GPUs on many machines. Each participant, called a rank, swaps results with the others every step. So the job moves only as fast as its slowest rank and stops if any rank dies. One bad machine affects the whole job.
2 · Kubernetes node conditions and taints
Kubernetes records each machine's health as node conditions, such as NotReady. Controllers watch these conditions and can add a taint, a mark that stops new work from being scheduled on the node. They can also drain the node, moving its workloads off before it is removed.
3 · GPU Xid errors
3/5Cloudflare· Chris Tsoukaladelis· Oct 16
Cloudflare will score Verified bots on whether they keep obeying site rules
Cloudflare now watches how its Verified bots behave over time: whether they follow robots.txt and slow down after "too many requests" errors. The results feed a trust score that customers will eventually be able to filter on.
Why it matters · It shows a fixed, one-time credential being replaced by trust that keeps getting re-earned from observed behaviour. The same design question comes up for any allowlist, API key or service identity.
Know this first
Context from the editor, not the article · 37 sec
1 · What a Verified bot is
Cloudflare runs a large share of the web's traffic and keeps a list of known, well-behaved crawlers, such as search engines, called Verified bots. Sites using Cloudflare's bot controls can let these through while blocking unknown automation. Until now, getting on the list mostly meant proving your identity.
2 · How bots prove identity
In the older approach, a bot was identified by its user-agent string or IP range, and both are easy to fake. Web Bot Auth, a proposed standard, has the bot cryptographically sign its HTTP requests so the receiving side can check who actually sent them.
3 · How sites signal limits
4/5Stripe· Oct 15
Stripe now copies your payment data straight into Postgres
Stripe's Data Pipeline can now keep a copy of Stripe data, such as payments, customers and subscriptions, in your own Postgres database, updated within seconds. Stripe says this replaces the webhook-syncing code developers usually write themselves.
Why it matters · Keeping a local copy of third-party data in sync is a familiar consistency problem. Handing it to the vendor removes code you'd own, but you inherit guarantees Stripe hasn't documented.
Know this first
Context from the editor, not the article · 31 sec
1 · Webhooks and their failure modes
A webhook is an HTTP callback a service sends when something changes. Delivery is at-least-once and not ordered, so receivers must drop duplicates, handle events that arrive out of order, and retry failures. A missed event leaves the local copy silently wrong.
2 · Why teams mirror Stripe data
Apps check subscription status or payment state on hot paths. Calling Stripe's API on every request is slow and rate-limited, so teams keep a local copy in their own database and then must keep that copy matching Stripe.
3 · Stripe Data Pipeline
An existing Stripe product that exports account data in batches to data warehouses such as Snowflake and Redshift, or to cloud storage, for analytics and reporting. Until now it did not target the live databases that applications read from.
5/5Cloudflare· AJ Gerstenhaber· Oct 15
Cloudflare lets customer code run inside its web gateway's policy decisions
Cloudflare now lets customers run their own small programs (Workers) inside its corporate web proxy. The code can rewrite traffic, send alerts, or compute values that security rules match on, with no separate proxy servers to run.
Why it matters · It is a concrete case of the "build or buy" infrastructure trade-off, and of what co-locating compute with a proxy buys and costs: latency, failure modes, and coupling.
Know this first
Context from the editor, not the article · 34 sec
1 · Secure web gateways
A secure web gateway is a proxy that all employee web traffic passes through. It decrypts traffic, checks it against rules (user, destination, content category) and applies one fixed action: allow, block, isolate or log. SASE (Secure Access Service Edge) is the industry label for delivering this as a cloud service.
2 · Cloudflare Workers
Workers is Cloudflare's serverless platform. It runs customer JavaScript or TypeScript in lightweight isolates on the same servers that make up Cloudflare's network, in 330+ locations. It comes with storage services: KV (key-value), D1 (SQLite), R2 (object storage).
Netflix has described the Observability Ontology, a typed knowledge graph that links its monitoring, alerting, deployment and incident tools. On an ordinary day Netflix handles more than 2 billion requests and 38 million log events per second, so no single dashboard holds the whole picture.
The problem
The authors' example is a home screen that renders about 1.5 seconds slower than usual. The cause could be a client release, an A/B test, load shedding, or a bad GraphQL response from a service six hops downstream. A typical investigation takes about four hours, and little of that is hard analysis. Most of it goes into reconstructing relationships the tools already know but never recorded in a readable form.
The underlying issue is identity. Atlas (telemetry), Radar (alerting), incident.io, PagerDuty, 1 (deployments) and Slack each make sense on their own, but the same thing is an "application" in one, a "service" in another and a "cluster" in a third, with names that differ slightly. Joining them depends on the few engineers who know the mapping. Netflix concludes that end-to-end observability is a data-modelling problem.
How it works
The 2 defines entity types and the relationships allowed between them. Facts are stored as 3 triples, such as INC-5377 ops:affects api-gateway, and queried with 4, the standard query language for RDF graphs. That makes "incidents affecting Team Bedrock's apps in us-east-1" a single query instead of a hunt across three tools. The graph is stored in QuipuDB, Netflix's distributed 5 built on the Rust key-value store redb.
Two modelling choices matter most:
No vague "service" type. Every entity has to resolve to a single ops:Application type and pass a check against a software catalog of about 49,000 entries before it is created.
Incident chat is modelled too. The hypotheses people tried and the fixes that worked become triples instead of fading in Slack.
Dozens of 6 turn noisy input into typed triples. They run sequentially to avoid race conditions, and they call external systems on a best-effort basis so a slow API can't stall a harvest. Some enrichers use probabilistic tools such as LLM summarisation and fuzzy name matching. Their output must still fit fixed types and allowed values, which keeps the graph deterministic.
A correlation engine reads thirteen signal types, including deploys, config changes, log errors and metric anomalies. It scores chains of events within a ten-minute window and drops anything below 0.6 confidence, storing each surviving chain with its evidence.
Results and limits
Netflix publishes no measured time savings. It says the graph removes the hour or two spent gathering context, while engineers still judge the evidence and write the fix. An internal multi-agent tool, AutoRCA, ranks hypotheses on top of the graph, and the authors say those hypotheses are sometimes wrong. They also list lessons for others: model identity first, treat the ontology as code, and use open standards.
Netflix has described the Observability Ontology, its attempt to fix slow incident investigations by treating them as a data-modelling problem. Netflix runs more than 2 billion requests and over 38 million real-time log events per second on an ordinary day, and served about 65 million concurrent streams for the Paul–Tyson fight. At that scale, the authors (Prasanna Vijayanathan and Renzo Sanchez-Silva) say no single team or dashboard holds the whole picture.
Where four hours go
Their example is a routine regression: the home screen takes about 1.5 seconds longer to render, and an alert fires. Any of these could be the cause: a client release, an A/B test, load shedding in the infrastructure, or a GraphQL response gone bad because a service six hops downstream rolled out. Metrics, events, logs and traces (1) each answer "roughly a quarter of the question."
The investigation takes about four hours, and the authors stress that very little of it is hard computation. Most of it goes into reconstructing relationships that the tools already know but that nobody has written down in a form a person or machine can read.
The real problem: tools disagree on identity
Each system is internally coherent but isolated from the others:
Atlas (telemetry) knows applications and their time series.
Radar (alerting) knows thresholds and firing history.
incident.io knows severities and responders.
PagerDuty knows on-call rotations.
2 (deployment) knows what shipped and when.
Slack holds who solved something similar last quarter, in a form no query engine can reach.
The same thing is an "application" in one tool, a "service" in another and a "cluster" in a third. Names differ by a suffix, by casing, or by a rename only a couple of people remember. Joining across tools depends on a human who carries that mapping in their head, and that person becomes the bottleneck. More data or better dashboards don't fix this. Netflix also argues that adding AI agents doesn't help unless they get correct context, because "we need the right needle, not any needle."
The 3
The fix is an ontology: a formal specification of entity types, their properties and the relationships allowed between them. Facts are stored as triples. A worked example:
With these facts in place, "Which incidents affected applications owned by Team Bedrock in us-east-1?" becomes one 4 query instead of a lookup across three tools.
The stack is standard W3C: 5 for the data model, OWL for the ontology, 6 for validation, SPARQL for queries. The authors chose it for portability across graph stores. The graph lives in QuipuDB, Netflix's own distributed 7 (RDF triples plus a fourth field naming the graph each belongs to), built on redb, an embedded key-value store written in Rust. Each harvest also writes a full Turtle file (a plain-text RDF format) for archiving. Because the files are human-readable and easy to diff, they turned out to be more useful for debugging than expected. The ontology is split into twelve namespaces, one per operational domain.
Two modelling decisions stand out:
No generic Service type. "Service" meant four different things across Netflix's tools. Everything resolves to a single ops:Application type, and each one is checked against the software catalog of about 49,000 entries before it is created. Usernames, team names and dependency names scraped from network topology never become applications. Dependency edges are capped in both directions, because one popular service can have thousands of reverse dependencies and would otherwise dominate the graph.
Model human reasoning too. An incident Slack channel (the example is 200 messages, 50 people, 15 services) holds the hypotheses people tried, the mitigation that worked, and who understood why. All of that is converted into ordered triples before the channel goes quiet and the knowledge is lost.
Deterministic output from stochastic steps
Several pipeline steps are probabilistic: LLM summaries of discussions, fuzzy matching of service names, and a probabilistic engine that proposes causal chains. Netflix's claim is that the type system makes the overall output deterministic. A model can write any prose, but what it emits must fit fixed predicates. For example, category must be one of incident/question/discussion/decision, and sentiment one of positive/neutral/negative/urgent. The authors describe five layers that progressively lower "entropy," meaning the share of information still stuck in unstructured form.
The work is done by dozens of 8. Each one reads noisy input, processes it (possibly stochastically) and emits typed triples. Each enricher can build on what earlier ones wrote. Three rules apply to all of them: every entity has an rdf:type, every URI follows one naming scheme, and the catalog must approve every ops:Application.
Two operational choices matter here. The pipeline runs sequentially (load, add, save), which removes a whole class of race conditions at the cost of wall-clock time. Calls to external systems are best-effort, so a slow API degrades the graph but never blocks the harvest.
Correlation and the flywheel
A correlation engine reads thirteen signal types, including log errors, metric anomalies, slow or failed traces, alerts, deployments, config changes, experiments, autoscaling and commits, and searches for paths between them. Some ordered pairs get a high prior weight because they are known to be causal, such as a deployment before a log error or a config change before a metric anomaly. Chains are scored on how close together the events are in time and how severe they are, within a ten-minute window. Chains below 0.6 confidence are dropped. Each surviving chain is stored with the records that produced it, so a 0.87 chain counts as a strong hypothesis to check, not a verdict.
Each harvest runs a loop of Observe, Enrich, Infer and Adapt. In the Adapt step, established patterns become detection rules. Over many cycles the graph produces aggregate findings such as "api-gateway deploys cause 40% of SEV-2s." The authors present this as a way to prevent incidents, not just respond faster.
Results and caveats
The post gives no measured reduction in investigation time. It says the hour or two spent assembling context (finding the owning team, working out what shipped, recalling a similar past incident) is the part the graph absorbs. Engineers still weigh the evidence and write the fix. AutoRCA, an internal multi-agent tool, sits on top of the graph, and the ontology acts as the interface between its parts: each component only needs to learn the graph's vocabulary, not the others' formats. The authors admit the ranked hypotheses are sometimes wrong, which is why each one carries its evidence.
Lessons and what's next
The authors list the principles they would apply on a second attempt:
Model identity first.
Treat the ontology as code: versioned, reviewed, changed through pull requests.
Keep concepts (which change slowly) separate from instances (which change constantly).
Use open standards.
Constrain stochastic tools with a type system. This is the principle they would defend hardest.
Prefer ordered, best-effort enrichment.
Keep humans as the approvers.
Next, they plan automated remediation and a proof of concept that models incidents in Upper, the domain-modelling language behind Netflix's Unified Data Architecture.
The post leaves several things open. It gives no numbers for graph size, query latency or harvest duration, and no measured before-and-after investigation times.
Remember
Most investigation time went into joining tools that disagree on entity names, not into analysis.
Validating every entity against one authoritative catalog before creating it was the foundation of the design.
Netflix keeps output from probabilistic tools trustworthy by forcing it into a narrow typed schema.
NVIDIA drivers write numbered fault codes called Xid (and SXid for the NVSwitch interconnect) to the Linux kernel log. Some codes mean the GPU must be reset or replaced, and others are harmless. What each code means can differ between GPU generations.
Ways a GPU machine passes basic checks while faulty, and the design rule that closes each gap.1 / 4
Netflix's compute platform team has described how it keeps GPU machines healthy for training runs that last weeks and for inference that runs around the clock, all on its Kubernetes platform. After moving from basic checks and manual draining to layered checks and automatic repair, GPU node availability stays above 99.95%. Job success rose from about 98% to above 99.5%.
The problem
A large training job runs as a gang across hundreds of GPUs. Its ranks (participants) exchange data every step, so one slow or dead GPU slows or stops everyone. Cloud instance checks only confirm that a machine is up. A driver-responds check catches only dead GPUs.
The failures that cost the most get past both. A GPU can answer queries but fault under real load. A network adapter can have too few working links, so the job crawls instead of crashing. And the checks themselves can hang or silently stop running.
Detection
A node agent built on Kubernetes 1 runs every 30 seconds. It uses NVIDIA's 2 (Data Center GPU Manager) to check GPU errors, memory, interconnect and temperature. It also counts AWS 3 (EFA) network links. Results become Kubernetes node conditions, which controllers act on.
The checks follow four rules:
Fail closed: a check that times out marks the machine unhealthy.
Fallback: if the diagnostic tool breaks, the agent reads 4 fault codes straight from the kernel log.
Own fatal-error map: Netflix keeps its own list of fatal codes for each GPU generation.
No silent no-ops: a missing dependency fails the check. That rule came after machines with too few adapters went into service for days unnoticed.
Repair
A problem condition stops new work from landing but spares running jobs. If it persists past a tolerance window, it becomes a failure, and the machine is drained and replaced. Netflix customized 5, a Kubernetes node provisioner, so each condition has its own window. A cap limits how much of a cluster can be disrupted at once.
Repairs escalate from killing the job, to resetting the GPU, to rebooting, to replacing the machine. Faulty GPUs are blocklisted by hardware ID so they can't come back under a new instance.
New machines run diagnostics and wait two check cycles before taking work. Jobs checkpoint their state, and the scheduler retries only failures it has classed as the platform's fault. Netflix once traced 6 timeouts (NCCL is NVIDIA's GPU-to-GPU communication library) to data-loading stalls, not the network. Blaming hardware would have recycled healthy machines and fixed nothing.
Caveats and next steps
Netflix gives no fleet size, failure-rate breakdown or repair times. Next it plans:
topology-aware placement through Kubernetes 7
a CPU scheduler built on 8 that prioritizes GPU coordination and data loading
treating a GPU that runs 10–15°C hotter than its peers as a fault
Netflix's compute platform team has described how it keeps GPU machines healthy for training runs that last weeks and for inference that runs around the clock. Both run on its Kubernetes-based platform. Before the change, the team relied on basic checks and drained bad machines by hand. Now it uses layered checks and automatic repair. The two numbers it reports: GPU node availability is consistently above 99.95%, and job success rate rose from about 98% to above 99.5%. That second figure means platform-related job failures fell from about 2% of jobs to under 0.5%.
Why one bad GPU is everyone's problem
A large training job runs across hundreds of GPUs as a gang. Its ranks (the individual participants) exchange data every step, so the job moves at the pace of the slowest rank and stops if any rank dies. A slow GPU or a degraded network link on one machine therefore slows the entire job. GPUs are among Netflix's fastest-growing and most expensive infrastructure. A GPU sitting idle behind a slow neighbor is capacity the company paid for and isn't using.
What basic checks miss
Cloud providers check that a virtual machine is powered, reachable, booted and has its storage attached. They also announce planned maintenance. Netflix uses all of these signals, but they only confirm that the box is up. The usual next step, checking that the GPU driver responds, catches a dead GPU. The article lists failures that get past both:
A GPU answers status queries but has a memory or firmware fault. The fault shows up only when a job allocates memory or runs sustained compute.
A network adapter is present but has fewer working links than expected. Collective operations (group data exchanges between GPUs) still run, but at a fraction of the bandwidth. The job crawls instead of crashing.
The checks themselves break. The diagnostic tool hangs when the driver hangs, so a check that waits on it hangs too. Or a driver upgrade removes a binary the check needs, and the check quietly stops reporting.
The team argues that a check which can silently stop running is worse than no check, because it gives false confidence.
Four layers
Netflix stacks four layers, each meant to catch what the one below misses:
Node and hardware health finds bad devices and taints or recycles the machine.
Scheduler and platform validates machines before they take work, retries tasks and jobs, and drains machines safely.
Training loop uses checkpoints and elastic workers so a job survives losing a machine.
Observability covers metrics, service-level objectives and failure attribution.
Detection
Checks run in two tiers. Device-level checks use NVIDIA's 1 (Data Center GPU Manager). They read driver error codes, uncorrectable memory errors, interconnect integrity and thermal state. Network checks count the 2 (EFA) interfaces, AWS's high-speed network adapters for traffic between machines, and check their link state. A node agent built on the open-source Kubernetes 3 runs every 30 seconds. It writes the results as node conditions, the same mechanism behind NotReady.
Four design rules close the gaps listed above:
Fail closed. If the driver or the tool is unresponsive, the check times out and marks the machine unhealthy.
Fallback path. If the tool that reads 4 and SXid codes is broken, the agent reads the kernel log () directly.
Graded repair
A failed check first raises a problem condition. That stops new work from landing on the machine but leaves running jobs alone, because many problems clear on their own and evacuating a job is disruptive. If the condition lasts longer than a tolerance window, it becomes a failure condition. The machine is then evacuated, drained (overriding normal disruption limits), terminated and replaced.
Netflix customized 5, an open-source node provisioner for Kubernetes, and its node auto-repair feature. A policy sets how long each condition is tolerated: a clearly fatal error gets acted on within minutes, and a flaky signal gets a longer window. A cap limits how much of a cluster can be disrupted at once, so a bad rollout of the checks themselves can't tear down a cluster. The team plans to contribute this upstream.
Repairs escalate step by step, each matched to a kind of fault:
Killing the job clears a software wedge.
Resetting a GPU clears many transient device faults.
Rebooting clears firmware state that has drifted over a long uptime.
Replacing the machine is reserved for hardware that fails a test the team trusts.
For replacements, Netflix records the GPU's immutable hardware ID, which stays with the physical device even when the instance changes. It reports the ID to the cloud provider and adds it to its own blocklist, so the same faulty card can't quietly come back under a new instance.
Netflix also found a race condition. Capacity was being handed to the scheduler while slow checks were still settling, so a job could land just before a check flagged the machine. Capacity is now held until the checks settle.
Admission and survival
A new machine has to prove itself first. It passes system checks, confirms its GPUs and their topology are present, runs GPU diagnostics, and then waits for two health-check cycles before its startup taint is removed. The wait is needed because slower checks, such as the adapter count, take more than one cycle to report. When a machine is torn down, deeper diagnostics run and its GPU IDs feed the blocklist. When a job is allocated machines, a short collective run and pairwise network tests are done first, so a machine that would fail in the first minute fails before the job is admitted.
When a machine is lost mid-run anyway, jobs resume from their last checkpoint, and the scheduler retries failures the platform caused. Each failure is first classified as platform or application. Retrying everything would hide real bugs and burn GPU hours. One example: a class of timeouts in 6 (NVIDIA's library for exchanging data between GPUs) looked like network faults. The real cause was data-loading stalls at high worker counts. Treating them as hardware faults would have cycled healthy machines and fixed nothing.
Measurement and sharing
Three service-level objectives (SLOs) track availability of schedulable capacity, job-start success and job success. Each is split into hardware, driver, platform and user causes. The team optimizes for 7, the useful work actually completed, because a job held back by one slow GPU can look fine on an uptime chart. The article doesn't publish the SLO targets.
For small inference jobs, Netflix splits GPUs with 8 (Multi-Instance GPU), which gives each tenant its own memory. It can change how GPUs are split without rebooting. Allocation currently uses Kubernetes device plugins. The team plans to move to 9 (DRA), a newer Kubernetes API, once Karpenter supports it. DRA would let the platform take one faulty GPU out of service without losing the whole machine. It would also avoid the bandwidth penalty when a GPU is paired with a network adapter or CPU on a different PCIe root.
Gaps and next steps
The article gives no fleet size, failure rates by type, or time-to-repair numbers. The two headline results come without a breakdown of which layer contributed what.
Planned work includes:
more active checks under load, wider burn-in testing, and more in-place fixes for certain Xid errors
periodic diagnostics between jobs
topology-aware placement that respects IMEX domains, PCIe roots and NUMA nodes
a custom Linux CPU scheduler built on 10 that prioritizes collective coordination and data loading
treating a GPU that runs 10–15°C hotter than its peers as a fault, even when it isn't throttling
broader NCCL RAS monitoring to catch hangs and mismatched collectives sooner
Remember
A health check that can hang or silently stop running should report failure, never pass by default.
Match the repair to the fault: kill the job, reset the GPU, reboot, and replace only as a last resort.
Sort failures into platform or application before retrying, or retries hide bugs and recycle healthy machines.
Sites publish crawl rules in robots.txt, standardised as RFC 9309. The file lists which paths each bot may or may not fetch. Separately, a server can answer 429 Too Many Requests, sometimes with a Retry-After header, telling the client to slow down. Neither is enforced; obeying them is voluntary.
Figure placed in the post's section on judging bot traffic against a site's robots.txt rules.1 / 2
The news
Cloudflare has started grading its 1 on how they behave, not only on who they are. Verified bots are crawlers on Cloudflare's list of trusted automation. Until now, joining the list mostly meant proving identity. Bots sign their HTTP requests with 2, a proposed standard for cryptographically signed requests, and a registry called 3 catalogues them. The new system checks whether those identified bots keep following site owners' rules, and it summarises the result as a "4."
Author Chris Tsoukaladelis compares this to a driver's license. It proves who you are, not that you will drive well tomorrow. The post adds that misbehaving bots are often not malicious. A misconfiguration or a bug is a common cause, so a bot that passed review can go wrong later.
Risk and trust
Cloudflare separates risk, the chance that a single request is harmful right now, from trust, which builds up through consistent behaviour over time. The trust band measures the second. It ignores one-off violations and looks for repeated patterns. For precedent, Cloudflare removed Perplexity's crawler from the list in 2025. The crawler had fetched content from sites that opted out of scraping and then tried to get around blocks.
The two signals
Following instructions. Does the bot's traffic match the site's 5 over time? Robots.txt is the file, standardised in 6, that lists which paths each bot may crawl. Cloudflare says the file sets a minimum: anything it doesn't forbid is not automatically allowed.
Responding to feedback. A bot can stay within allowed paths and still overload a server. When a site returns 7 "Too Many Requests," Cloudflare checks whether the bot slows down, pauses, or keeps going at the same rate. Backing off consistently raises trust. Ignoring 429s repeatedly lowers it.
The band is a range, not a pass/fail. It comes with the specific behaviours behind it and is not a permanent verdict.
Where it fits
Cloudflare describes an accountability loop. Bot Preference Sync helps sites publish their preferences. The new system checks whether bots honour them. Bot Management, Cloudflare's bot-handling product, lets site owners act on the result.
Status and gaps
The system runs internally for now while Cloudflare builds up behaviour history. Next, Cloudflare plans to publish the scores on 8, its public traffic-statistics site, add more signals, and let customers set their own trust threshold for Verified bot traffic.
The post says nothing about how the score is calculated: weights, observation windows, what counts as "repeated," or how much slowdown after a 429 counts as compliance. Engineers will want those details before relying on the bands.
What changed
Cloudflare has built a system that keeps checking whether its 1 behave well after they are admitted. Verified bots are crawlers on Cloudflare's list of known, trusted automation. Until now the program focused on identity. Bots sign their requests with 2, a proposed standard for cryptographically signing HTTP requests, and a registry called 3 catalogues those identities in an open format. The new system adds a behavioural check. It records how each bot actually treats the sites it visits and condenses that into what Cloudflare calls a "4."
Author Chris Tsoukaladelis compares this to a driver's license. A license proves who you are and that you once passed a test. It says nothing about how you will drive tomorrow. Being verified works the same way: you can identify a bot without that telling you anything about its future behaviour.
The three rules
To qualify as a Cloudflare Verified Bot, an operator broadly has to:
Identify itself accurately and securely.
Do only what it says it will do.
Avoid abusive behaviour.
Web Bot Auth and BotBase cover the first rule. The new system covers the third, which the post calls "thou shalt do no harm." The post says misbehaviour usually isn't malicious. Often it comes from a misconfiguration or a bug, which is part of why one-time vetting falls short. A bot that behaved well at review time can start misbehaving after its next deploy.
Risk and trust
The post separates two ideas:
Risk is the chance that a single request or action is harmful right now. It can swing from moment to moment.
Trust builds up through repeated, consistent behaviour over time.
Most bot-management scoring is about risk: it judges each request on its own. The trust band is about history. That is why Cloudflare says it ignores isolated violations and one-off events and looks for sustained patterns. One bad fetch shouldn't sink a crawler. A crawler that keeps ignoring the rules should lose trust.
The post gives one precedent for enforcement. In 2025 Cloudflare removed Perplexity's crawler from the Verified list. The crawler had requested pages on sites that had opted out of scraping, and when blocked it tried to get around the controls.
Signal 1: does the bot follow instructions?
The first signal is whether the bot's traffic stays consistent with the site's 5 over time. Robots.txt is the file, standardised in 6, that tells all bots or specific bots which paths they may crawl. Cloudflare expects responsible bots to read and honour it.
The post also makes a point about how to read the file. Robots.txt sets a minimum list of things bots should not do. Anything it leaves unmentioned is not automatically allowed. A site doesn't need a robots.txt line saying "stop when I tell you to stop by other means." That leads to the second signal.
Signal 2: does the bot respond to feedback?
A bot can stay entirely within allowed paths and still swamp a site's origin servers. The clearest pushback a site has is an 7 "Too Many Requests" response, sometimes with guidance on how long to wait. Cloudflare checks whether the bot's request rate changes after it gets a 429. It might slow down, pause, or carry on at the same pace. Consistently slowing down or pausing raises the bot's trust. Repeatedly ignoring the signal lowers it.
Here is an illustrative example; the post does not give thresholds or numbers. Picture a crawler fetching 50 pages per second from one origin. The origin starts returning 429s with Retry-After: 60. A compliant crawler stops or cuts its rate sharply for about a minute. A non-compliant one keeps sending 50 requests per second and treats the 429s as failures to retry. If this happens once, the system ignores it. If it happens across many sites and weeks, the bot's band moves down.
What the output looks like
Cloudflare calls the result "explainable." The trust band is a range, not a single yes/no. It comes with the specific observed behaviours that produced it. Cloudflare also says the band is not a permanent verdict on the bot or its operator. Because it is built from ongoing observation, a bot that fixes its behaviour should see its band improve.
The post places the system inside what Cloudflare calls an accountability loop:
Bot Preference Sync helps site owners publish their preferences in a consistent way, aligning robots.txt with the policies they have configured.
The new trust system checks whether Verified bots keep honouring those preferences.
Bot Management, Cloudflare's product for classifying and acting on automated traffic, lets site owners decide what happens to that traffic.
Status and plans
The system currently runs internally only. Cloudflare is using this phase to build behavioural history and refine how observations turn into guidance. Planned next steps:
Publish the scores on Cloudflare 8, its public traffic-statistics site.
Add more behavioural signals beyond robots.txt compliance and 429 handling.
Let every Cloudflare customer choose the trust threshold that decides how Verified bot traffic is handled on their sites.
What the post leaves open
The post is a policy announcement rather than a technical write-up, and several details a systems engineer would want are missing:
How the score is calculated. The post doesn't say how violations are weighted, how long the observation window is, how recent behaviour is weighted against old behaviour, or what counts as "repeated."
How it attributes behaviour. Fair scoring requires linking observed traffic to a specific signed identity and separating it from impostors. The post doesn't describe that step.
What counts as adapting to a 429. Cloudflare doesn't say how much slowdown is enough, or how a 429 issued by one origin is matched to that bot's later requests to the same origin.
How it handles disputes and edge cases. Robots.txt files can be malformed, can change mid-crawl, and can be cached by crawlers. The post doesn't say how these cases affect a bot's score.
How published scores will be used. Once the bands appear on Radar, they become a public ranking that operators may try to game. The post doesn't address that.
What the post does make clear is the design choice behind the system. Admission to the Verified list is no longer one check at the door. Cloudflare will keep checking: "verify, but continuously re-verify."
Remember
Cloudflare treats identity and trust separately: a signed identity says who a bot is, not whether it behaves.
The two current signals are robots.txt compliance and whether the bot slows down after HTTP 429 responses.
The system scores sustained patterns, not single requests, but the post doesn't explain how the score is calculated.
1, Stripe's product for exporting account data, can now keep a live copy of Stripe data in a 2 database that your application reads from. Until now it mainly fed analytics warehouses and cloud storage. You connect a database, which can be hosted on Supabase, AWS, Neon, Databricks Lakebase or elsewhere, and choose the tables you want, such as payments, customers and subscriptions. No code is needed. The feature is in public preview and free to start.
The problem
Apps often decide things from Stripe state: whether a subscriber gets paid features, or whether an order can ship. Calling Stripe's API for every check is slow and rate-limited, so teams keep a local copy. They build it either by polling the API or by consuming 3, the HTTP callbacks Stripe sends when an object changes.
Stripe lists the costs of the webhook approach: monitoring the integration, handling failed and out-of-order events, and regularly checking that local rows still match Stripe. Ordering is the subtle part. Here is an illustrative case: if a subscription's active → past_due → active events arrive in the wrong order, a naive handler leaves a paying customer locked out. Getting this right takes version checks, duplicate-safe writes and a periodic 4 job.
How Stripe's version works
The post describes three stages:
5: copy the existing rows from the selected tables.
Stream: apply changes to Postgres "in seconds" after they happen in Stripe. No latency figures are given.
Audit: regularly compare the synced records against Stripe and fix any differences.
The audit is Stripe running the reconciliation job you would otherwise write. In practice the promise is 6: updates arrive quickly, and periodic repair catches whatever the stream drops or misorders.
What's missing
As a launch post, it leaves out what an engineer needs to evaluate it. It does not say whether updates are applied in order per object, whether related rows are written in a single transaction, how often audits run, how objects map to the schema, or whether your app may also write to the synced tables. Pricing after the preview isn't stated either.
Trade-offs
You get rid of a subsystem you'd otherwise own, and reconciliation is done by the side that holds the source of truth. In return, Stripe defines your schema, an outside system writes to your production database, and you lose visibility when something goes wrong. Webhooks are still the right tool for reacting to events, such as alerting on a failed payment, as opposed to reading current state.
Stripe says Snowflake, BigQuery, Redshift, Databricks and cloud storage will get real-time sync "soon."
The announcement
Stripe has added real-time sync to 1 to 2. Data Pipeline is Stripe's export product, and until now it sent account data to analytics warehouses and cloud storage. The new feature writes payment, customer, subscription and other Stripe objects into a Postgres database that your application reads from. Stripe lists Postgres hosted on Supabase, AWS, Neon, Databricks Lakebase "and other providers." It is in public preview and free to start. Setup happens in the Stripe Dashboard, and no code is needed.
This is a vendor launch post, short on internals. What follows separates what Stripe says from what it leaves out.
The problem it targets
Many apps make decisions from Stripe state. A subscription business unlocks paid features based on the current plan. A shop ships an order once payment is confirmed. Querying Stripe's API on every such check adds latency and runs into rate limits, so teams keep a local copy.
Stripe describes two common ways teams build that copy:
Polling: call the Stripe APIs again and again to pull current state.
Webhooks: receive Stripe's change events and apply them to your tables.
Both put the burden of correctness on you. The post names the usual pain points: monitoring the integration, handling failed or out-of-order events, and regularly checking that stored data still matches Stripe.
Here is why those points are hard, using standard webhook behaviour. The example is illustrative. A subscription goes from active to past_due and then back to active within a few seconds. If the two events arrive reversed, a naive handler writes past_due last and locks out a paying customer. To defend against that, you compare timestamps or version numbers before each write, or you re-fetch the object from the API whenever an event arrives. You also need to make each event safe to apply twice, because retries cause duplicate delivery. And since your endpoint can be down and events can expire, you need a periodic job that walks your tables against Stripe and repairs drift. That job is the "regularly check" chore the post mentions.
What Stripe now does instead
From the post, the pipeline works in three stages:
3. You connect a Postgres database and choose which Stripe tables to sync. Data Pipeline copies the existing rows across.
Incremental updates. As data changes in Stripe, updates arrive in Postgres "in seconds." No specific latency figure or percentile is given.
Audit and repair. Data Pipeline "regularly audits the synchronized records against the source data in Stripe and resolves discrepancies."
The third stage matters most. It is Stripe running the 4 job described above, so a lost or misordered update eventually gets fixed without your involvement. Stripe calls the result a "reliable, real-time copy." Put more carefully, the design promises 5: changes stream in quickly, and the audits correct whatever the stream gets wrong.
A customer quote from Josh Siegel, CPO and CTO at Company Sage, says the sync let them cut back their own API and webhook infrastructure.
What the post doesn't explain
For a senior engineer weighing adoption, the important details are missing. The article does not cover:
Ordering and atomicity. Are changes applied in order per object? Are related rows, such as an invoice and its line items, written in one transaction, or can a reader see half an update?
Audit cadence. How often reconciliation runs, which determines how long a wrong row can survive.
Latency. "Seconds" is stated with no distribution and no figures for the worst cases.
Schema. How Stripe objects map to tables, how schema changes are rolled out, and whether you can safely add indexes or columns.
Write ownership. Whether Stripe expects to be the only writer to those tables, and what happens if your app writes to them.
Connectivity and security. How Stripe reaches your database, for example through IP allowlists or private networking, and what permissions it needs.
Pricing after preview. The post says only "free to get started."
The documentation may answer these, but the announcement does not.
Trade-offs
The case for adopting it is that you remove a whole subsystem you would otherwise build and maintain: the webhook receiver, deduplication, ordering logic and drift repair. The reconciliation is run by the party that owns the source of truth.
The costs are coupling and visibility. Your schema is Stripe's to define, a sync problem becomes a support ticket rather than something you can debug, and an outside system writes directly to your production database. Webhooks also remain the better fit when you need to react to an event, such as sending an email the moment a payment fails, rather than just read current state. A mirrored table tells you what is true now. It does not notify you when something changes.
What's next
Postgres is the first destination with real-time sync. Stripe says Snowflake, Google BigQuery, Amazon Redshift, Databricks and cloud storage are "coming soon."
Remember
Stripe now does the backfill, streaming and drift repair that webhook-based syncing used to leave to developers.
Periodic audits are the real correctness guarantee: the copy converges on Stripe's state, but the post gives no bound on how stale it can get.
Ordering, transactional writes, schema rules and audit frequency are undocumented in the announcement. Check the docs before relying on it.
DLP (Data Loss Prevention) scans outbound traffic for sensitive data such as credentials, card numbers or customer records. It usually relies on regex patterns and keyword lists, so it often flags strings that look sensitive but don't matter, like revoked keys.
Diagram of the Forward action: Gateway hands a matched request to a Worker acting as middleware.1 / 2
Cloudflare has added hooks that let customers run their own code inside 1, its secure web gateway (a cloud proxy that filters employee web traffic). The code runs as a Cloudflare 2, Cloudflare's serverless function platform, on the same machines as the gateway. This is a vendor launch, and the performance claims are Cloudflare's own, with no measurements given.
The problem
Gateways work by match-then-act: a rule matches, and a fixed action fires (allow, block, isolate, log). When that isn't enough, teams build infrastructure beside the gateway. One example from the post: a pool of NGINX proxies kept only to turn a 3 (a signed identity token) into an HTTP header. Another: a log pipeline into a 4 (a security log platform) and on to Slack, which delivers alerts 5–15 minutes late with no delivery guarantee. The usual choice is between running your own proxy, which is flexible but costly to maintain, and living within the vendor's features.
Two entry points
Forward action (public beta). A matching rule sends an allowed HTTP request to a Worker that behaves like middleware. Its handle(request, context, next) method can rewrite the request, call next() to reach the origin server, rewrite the response, and return it. Gateway keeps handling TLS (encryption), certificates, connections and HTTP versions. A Worker can also just observe, for example by posting a Slack alert or saving a file to R2 storage, while the traffic continues.
Custom functions (due by year-end). A Worker's compute(ctx) receives identity, device posture, IP addresses and, for HTTP, the headers. It returns a string, number or boolean, which becomes a new field that rules can test. For example, a function pulls the user's group out of a JWT, and a rule says "JWT Role" contains "finance_contractor" → Allow. These also work in port-and-protocol network rules. The sample code skips verifying the JWT's signature, so it shouldn't be used for real authorization as written.
5 validation (beta now). DLP (Data Loss Prevention) scans outbound traffic for patterns such as API keys. When it matches, a Worker can check whether the key is live. The example calls GitHub with the token: a success scores a real match, and a 401 "vetoes" it. This cuts false positives without uploading reference data to the vendor.
Trade-offs
Cloudflare says calling a Worker is a local function call, so the only added latency is the Worker's own run time. Custom functions need a default value for failures, which forces an explicit choice between failing open and failing closed. The post doesn't say what happens when a Forward Worker fails. Usage is billed as ordinary Worker requests, and the free plan's limits will bind at gateway traffic volumes.
You stop running proxy servers, but you now own production code in your security path, along with its deployment, testing and dependencies. Email Security is the next product due to get these hooks.
Cloudflare has added two ways to run customer code inside the policy path of 1, its secure web gateway (a cloud proxy that inspects and filters employee web traffic). A new Forward action hands a matched HTTP request to a Cloudflare 2, a customer-written serverless function, which can act as inline middleware. Custom functions call a Worker during rule evaluation and use the value it returns as a condition in a rule. A third piece, custom validation functions for 3 (Data Loss Prevention, the scanning of outbound traffic for sensitive data), lets a Worker decide whether a detected secret actually matters. Forward and DLP validation are in public beta now. Gateway custom functions are promised before the end of the year.
This is a vendor launch post, so the claims about uniqueness and performance are Cloudflare's own. The interesting engineering question underneath it is a familiar one: where should extension logic live when a managed system can't express what you need?
The ceiling of match-then-act
Cloudflare's author, AJ Gerstenhaber, describes the gateway model as roughly 30 years old: traffic matches a condition, then a terminal action fires (Allow, Block, Isolate, Log). It covers common cases. When it doesn't, teams build infrastructure next to it. The post gives three examples:
Header injection. A team needs to put an HTTP header on requests based on claims in a 4 (JSON Web Token, a signed token from the identity provider that carries user attributes). Gateway can see the token but can't read it and write a header, so the team runs a pool of NGINX proxies just for that, and has to patch and monitor them indefinitely.
Alerting. A team wants a Slack message when someone signs up for a new SaaS tool. The gateway detects the sign-up but can only allow or block. So logs go through Logpush (Cloudflare's log export) into a (security event log platform) and on to Slack, with a 5–15 minute delay and no delivery guarantee.
The usual choice is between running your own proxy (flexible, expensive to maintain) and accepting the vendor's feature set. Cloudflare's argument is that its gateway and Workers already run on the same machines. Handing traffic to a Worker is therefore a local function call, with no "6" (sending traffic out to another network and back).
Forward action: Worker as middleware
Forward rules are evaluated for traffic that HTTP policies have already allowed. The first Forward rule that matches sends the request to the Worker bound to it. The Worker implements a handle(request, context, next) method. It gets the full headers and body, plus Gateway context such as the source IP. It can change the request, call next(request) to send it upstream, change the response, and return it. The pattern will be familiar from Express or Koa-style middleware.
Gateway keeps the hard parts of running a proxy: TCP connections, TLS termination and re-encryption, certificate management and HTTP version negotiation. The Worker only holds the transformation logic. A Forward Worker doesn't have to change anything. It can send a Slack message, write to R2 object storage, update a KV store or call a webhook while the traffic carries on to its destination. That replaces the slow Logpush-to-Slack chain with an alert that fires on the request itself.
Custom functions: computed 7
In Gateway, a "selector" is a field a rule can test, such as Source IP or Content Category. A custom function adds a new one. The Worker implements compute(ctx) and receives context that depends on the policy phase: source IP, destination, user identity, device posture, and HTTP headers where they exist. It returns a string, number or boolean.
Here is a worked example from the post. A function splits the bearer token on dots, base64-decodes the middle section (the JWT payload), and returns the first group, or "anonymous" if there is no header. An admin then writes the rule Custom Function "JWT Role" contains "finance_contractor" → Allow. Note that the sample code decodes the JWT without checking its signature. That is fine for showing how the feature works, but don't copy it as-is for authorization.
Network-layer policies don't see full HTTP requests, but they do carry identity, posture and connection metadata. So custom functions also work in Layer 4 rules (decisions based on port and protocol). Inside a function the Worker can do anything a Worker can, including calling external APIs or reading KV and D1. Gateway treats it as "data in, value out."
Cloudflare's guidance: use custom functions for quick, bounded, stateless calculations, similar to built-in checks like "port in range." Use Forward for heavier transformations, and design those carefully because they sit directly in users' live requests.
DLP validation: does this match matter?
Pattern-based DLP can tell you that a string looks like a GitHub token. It can't tell you whether the token is live. 8, the usual fix, means uploading reference datasets of your sensitive data to the vendor. With custom validation, a matched DLP entry dispatches a Worker before enforcement. The sample decodes the matched excerpt, calls GitHub's /user endpoint with the token, and returns a score. A successful response scores 1.0, meaning live and a real match. A 401 returns "veto", meaning revoked, so the match is thrown away. Anything else scores 0. Your own system stays the source of truth.
Performance, failure and cost
Cloudflare says the only added latency is your Worker's own execution time, because no network hop is involved. The post gives no measurements for this. It also doesn't say how slow external calls, like the GitHub lookup, affect user requests. For failures, meaning exceptions, timeouts or unexpected return values, each custom function needs a configured default value. That forces you to choose fail-open or fail-closed explicitly, which is the right design. The post doesn't describe how failures behave for Forward Workers, which sit directly in the request path.
Dispatched traffic is billed as ordinary Worker requests, and it works on the Free, Paid and Enterprise Workers plans. On the Free plan, CPU-time and daily request limits will bind quickly at gateway volumes.
What remains open
The trade-off moves rather than disappears. You no longer run NGINX boxes, but you now own production code inside your security path, with its own deployment, testing and rollback needs. The availability of your access control also comes to depend on that code and on whatever it calls. The post says nothing about versioning, staged rollout of Worker changes, or observability of per-request decisions. Email Security is named as the next product to get these hooks.
Remember
Forward makes a Worker inline proxy middleware; Gateway still handles TLS, certificates and connections.
Custom functions must declare default values, forcing an explicit fail-open or fail-closed choice.
Cloudflare claims dispatch to a Worker adds no network hop, but publishes no latency numbers.
Own fatal-error map. NVIDIA's error catalogs lag behind new GPU generations, so Netflix keeps its own list of which codes are fatal for each GPU family. One generation emits a burst of a normally fatal code on every healthy boot. The team learned to treat that burst as noise rather than drain fresh machines.
No silent no-ops. A missing dependency now fails the check. This rule came from an incident. An upgrade dropped a binary the adapter check needed, and for days machines with the wrong number of adapters went into service before anyone traced slow jobs back to them.
5
Justification pages. Customers want a warning page that asks users why they need a sensitive site, with behavior that depends on who they are. Policy engines are poor at serving dynamic responses.
