SQL in the hot path, features in seconds · Morning Gist
Monday, October 122 stories · 7 min
SQL in the hot path, features in seconds
Two infrastructure write-ups: Cloudflare puts customer queries inside its log pipeline, and Airbnb swaps a two-day batch job for streaming inference.
Picked from 47 new articles across 34 sources · arrow keys for other days
Today in 30 seconds
30s left
Cloudflare Logpush now runs customer SQL on logs in-process, using a Rust query engine called from Go, with no network hop. story 1 ↓
Arrow batches move across the Go-to-Rust boundary, so memory ownership has to be explicit. Failed transforms block delivery and never send raw data. story 1 ↓
Airbnb cut staleness in guest search profiles from about two days to seconds by triggering model inference straight from streaming events. story 2 ↓
Two new features in Airbnb's open-source Chronon make this possible without changing the serving path, and Airbnb measured what the extra freshness gained. story 2 ↓
1/2Cloudflare· Rob Taylor· Oct 12
Cloudflare runs customer SQL on logs inside its Go export service
Cloudflare's log-export service now runs customer SQL queries that filter, redact and reshape logs before delivery. It does this in-process by embedding the Rust engine DataFusion in its Go delivery service. A failed transform holds data back rather than sending it unredacted.
Why it matters · It shows a design many teams will face: putting user-defined compute into a high-throughput hot path, and making explicit choices about cross-language memory ownership and fail-closed behavior.
Know this first
Context from the editor, not the article · 37 sec
1 · What Logpush does
Logpush is Cloudflare's service for exporting logs, such as HTTP requests and firewall events, from its network to customer systems: security platforms, analytics tools and object storage like S3. It sends records in batches, close to real time, as newline-delimited JSON or other formats.
2 · Columnar query engines
Row formats like JSON store each record whole. Columnar formats such as Apache Arrow store each field as a contiguous typed array. A vectorized engine can then filter or compute over thousands of values in tight loops, which is much faster than handling one record at a time.
3 · Calling Rust from Go
2/2Airbnb Engineering· Pengyu Hou· Sep 17
Airbnb cuts search-profile staleness from two days to seconds using streaming
Airbnb swapped the nightly batch job that summarised each guest's activity for search ranking with an event-driven streaming pipeline. Updates now land in 10–30 seconds instead of up to two days, and bookings rose slightly.
Why it matters · It's a worked example of turning a batch-computed derived value into an event-driven streaming one without changing the serving path, and it measures what the extra freshness bought.
Know this first
Context from the editor, not the article · 37 sec
1 · Feature platforms
A feature platform computes per-entity values (say, a user's last 50 page views) from raw events and stores them for low-latency reads. The same definitions drive batch backfills for history and streaming jobs for live updates, so offline and online values match. Chronon is Airbnb's open-source one.
2 · Embeddings as cached state
Airbnb compresses each guest's history of bookings, views and searches into an embedding, a fixed-length vector of numbers. A search ranking service reads it with a key-value lookup. Computing it is expensive, so it is computed ahead of time and cached, much like a materialized view.
Go can call native code only through cgo, its bridge to C functions. A Rust library therefore exposes a C-compatible interface. The two runtimes manage memory differently (Go has a garbage collector, Rust uses ownership), so any data passed between them needs clear rules about who frees it.
Data path: Buftee feeds Bufstream (Go), which hands Arrow batches and SQL to DataFusion (Rust), then exports NDJSON.
What shipped
Cloudflare has made Transformers generally available in 1, its service for exporting logs from its network to customer systems. A Transformer is a SQL query that runs on each batch before export. Its WHERE clause drops records, and its SELECT output becomes the exported record, so it can redact, rename, nest, add or remove fields. Input and output are both 2 (newline-delimited JSON).
The goal is to replace the separate processing service that teams often run to strip health checks or sensitive fields before logs reach a 3 (security information and event management system). Filtering early also means teams stop paying to ship and store records they would discard anyway.
Why in-process
4, the Logpush delivery service, handles more than 110 million records per second on hundreds of Kubernetes pods. Each pod pulls time slices of data from 5, Cloudflare's in-house message broker. An early prototype ran transforms as WebAssembly in Cloudflare Workers. That put a network call on the hot path, with timeouts and retries that would cut throughput. Cloudflare instead embedded 6, a Rust SQL engine built on the columnar Apache Arrow format, directly in Bufstream. A prepared Transformer processed a synthetic 5,000-row batch in about 3 ms on one goroutine, roughly 1.6 million rows per second. High-volume jobs run transform batches in parallel.
Crossing from Go to Rust
Bufstream is written in Go. It decompresses incoming batches, builds a typed Arrow batch, and passes it to Rust as 7 (Arrow's serialized format) through 8, Go's bridge to C. Ownership rules are strict. Rust borrows the input pointer only for the length of the call. Rust owns the output, which Go copies and then tells Rust to free. The schema comes from each dataset's specification rather than being inferred per batch, so queries can be validated up front. The SQL is compiled once per time slice into a reusable DataFusion plan.
Failing closed
If a transform fails, Bufstream neither sends raw data nor advances the job's delivery offset, because unredacted output could break downstream assumptions. Retries back off progressively, and jobs that keep failing are disabled automatically.
Caveats
The author acknowledges that the Go/Rust boundary still costs serialization and copies, though per batch rather than per record. The only benchmark is synthetic. The post gives no production latency or CPU figures and doesn't list SQL limits.
What shipped
Cloudflare has made Transformers generally available in 1, its service that exports logs from Cloudflare's network to customer destinations. It has also opened Logpush to all self-serve plans. A Transformer is a SQL query that runs on each batch of logs before export. It can drop records, redact fields, and add, rename, remove or nest fields. The post, by Rob Taylor, is partly a launch announcement. Most of it, though, explains how Cloudflare embedded a Rust SQL engine inside a Go delivery service that pushes more than 110 million records per second.
The problem
Teams send Cloudflare logs to 2 (security information and event management systems), analytics platforms and object storage. Before the data lands, they often need to strip out health-check traffic, remove sensitive fields, or fit records to a shared schema. One example from the article: a team that uses several CDNs adds a static provider field so records from every vendor look alike. Today this usually means running a separate processing service that someone has to deploy, scale and watch. Teams also pay to export, ingest and store records they throw away on arrival, so filtering before delivery cuts downstream cost.
What a Transformer looks like
A Transformer targets one Logpush dataset, such as http_requests. The table in the FROM clause picks the dataset schema. Records that fail the WHERE clause are dropped, and whatever the SELECT returns becomes the exported record. Input and output are both 3 (newline-delimited JSON, one object per line), so this reshapes records but does not convert formats. In the article's example, one query renames and nests fields, masks part of the client IP, adds provider and error fields, drops unused fields, and filters out internal and health-check traffic. Customers write queries in a dashboard editor called Transformer Studio or through the API, preview them against a sample record, and attach them to a matching NDJSON job. Each edit creates a new version.
The constraints
Cloudflare's data pipeline takes in more than one billion events per second. 4, the Logpush delivery service, runs on hundreds of Kubernetes pods. Each pod claims a time slice of data from 5, Cloudflare's in-house distributed, log-based message broker. Job volume follows a Pareto-like distribution: most jobs are small, and a few process millions of records per second. Transformers had to keep reliable delivery and predictable resource use, and could not add new ways for the export hot path to fail.
The rejected design
The first prototype ran customer Transformers as WebAssembly inside Cloudflare Workers, the company's distributed serverless platform. That would have put a network call into every export. Transient timeouts and retries would cut throughput and add latency. Cloudflare chose to run transforms in-process instead.
Why DataFusion
6 is a Rust query engine with a SQL planner and a vectorized, columnar executor built on Apache Arrow. Cloudflare already uses it in Basin, another of its data products. In a benchmark, running a prepared Transformer on a synthetic 5,000-row http_requests batch from one Go goroutine took about 3 ms, roughly 1.6 million rows per second.
For scale (our arithmetic, not the article's): transforming every one of the 110 million records per second at that rate would take around 70 cores' worth of work. That is why Bufstream allows per-job transform parallelism, so large jobs can run several transform batches at once. Because the engine is embedded, transformed batches reuse Bufstream's existing scheduling, batching and destination controls. There is no separate transformation tier.
Crossing the Go/Rust boundary
Bufstream gets compressed batches of 7 records (a binary serialization format) from Buftee. For jobs with a Transformer, it decompresses the batch, parses the eligible logs, and builds a typed Arrow RecordBatch. It then serializes the batch with 8, Arrow's standard wire format, and hands it to Rust. The call goes through a C ABI (application binary interface) via 9, Go's mechanism for calling C.
Ownership is explicit:
Go lends Rust a pointer to the input buffer only for the length of the call. Rust never keeps it.
Rust allocates and owns the output buffer.
Go copies the output into Go-managed memory, then tells Rust to free its allocation.
Neither runtime ever frees or holds onto memory the other one manages, which rules out a class of cross-runtime bugs. Because the format is Arrow IPC, the Rust side reads typed batches directly with a StreamReader. It does not parse JSON rows and rebuild columns. Types, nullability and nested values look the same on both sides. The author notes the boundary still has a cost: batches are still serialized, decoded and copied. The cost is paid per batch, not per record.
The schema is not inferred from each batch. Bufstream generates it from the Logpush dataset specification, which gives DataFusion a stable set of fields and types and lets it validate queries up front.
Executing a query
Saved SQL is stored in 10, Cloudflare's object store. Bufstream fetches it through a multi-layer cache and compiles it into a DataFusion 11, the engine's tree of scan, filter, projection and expression steps. The plan is built once per time slice of a job's data and reused for every batch in that slice. For each batch, the Rust wrapper binds the batch as the plan's input table, runs it, and returns Arrow output.
Failing closed
If a transform fails, Bufstream does not advance the job's delivery offset and does not send the untransformed batch. Sending raw data could break the redaction, filtering or schema that downstream systems depend on. Bufstream retries with growing delays so a broken job doesn't starve healthy ones, and it reports errors on the job. Jobs that keep failing for an extended period are disabled automatically. A Transformer Health view shows successes, failures, data volume before and after, and transform duration.
After execution, Bufstream converts the Arrow output to NDJSON, batches and gzips it, and uploads it. It adjusts the number of upload goroutines to stay within each destination's concurrency limits.
What's left open
The only performance figure is the synthetic single-goroutine benchmark. The post gives no production numbers on added latency, CPU or memory per job, and no cost for the Cap'n Proto → Arrow → NDJSON conversions. It doesn't say which SQL features are supported, or what limits apply to expensive queries. Failing closed protects downstream consumers, but it means a broken query stalls delivery. The article doesn't define how long "an extended period" is before a job is disabled, or whether data held in Buftee can expire before the customer fixes the query.
Remember
Cloudflare dropped a Workers-based design because a network call in the export hot path brought timeouts and retries.
Explicit buffer ownership across the Go/Rust boundary means neither runtime frees or keeps the other's memory.
Transforms fail closed: nothing is sent and the offset stays put, so a broken query stalls delivery rather than leaking data.
A batch job reprocesses a snapshot on a schedule, so its output is only as fresh as the last finished run plus however long the run takes. A streaming job processes events as they arrive. Its output stays current, but the work becomes continuous, per-event and stateful.
The original two-stage design: a nightly batch job encodes guest history and the real-time ranker reads the stored embeddings.1 / 2
Airbnb has replaced the nightly batch job that summarised each guest's activity for search ranking with an event-driven streaming pipeline. Updates now land within about 10–30 seconds instead of up to two days. The work added two general features to 1, Airbnb's open-source 2 (a system that computes per-user data values from event streams and serves them at low latency). Both have been contributed upstream.
The old design
A Transformer-based sequence encoder, a neural network that reads a guest's bookings, reviews, views and searches in order, compresses that history into an 3, a fixed-length vector. Running it on every search would be too slow. Instead, a nightly batch job wrote embeddings to a low-latency key-value store, and the ranking service looked them up at query time. It is the classic precompute-and-cache pattern, and it has the classic flaw: the data was stale by a full day of schedule plus the job's own lag, close to two days in total.
Why "run it more often" wasn't enough
Running the batch more frequently would shrink only the schedule part of the staleness. The encoder also expected daily snapshots, not single events, and the team didn't want to fork its logic from the version used in training. And activity arrives on separate streams (views, searches) that each need merging with the guest's long-term history, so streaming results stay comparable to batch ones.
The two mechanisms
4: when a Chronon streaming job commits a new feature value, it publishes a lightweight notification, so downstream consumers no longer have to poll.
NRT (near-real-time) Model Transform: a streaming pipeline can call an already-deployed model, here the same encoder the batch job uses, and write its output back as a servable feature.
In the new flow, an event is merged into the guest's stored sequences, Push Mode signals the update, and the encoder runs. The new embedding goes to the same store as before, so the ranking service's read path doesn't change.
Results
Freshness went from roughly two days to well under a minute.
Offline ranking quality (5, a metric that rewards putting relevant results near the top) rose 1.67%, which the authors call large for a system tuned over a decade.
A/B tests showed about a third of a percent more uncancelled bookings.
The model didn't change, so the authors conclude that freshness alone is worth something to ranking quality.
What's missing
The post doesn't cover how batch and streaming writes to the same store avoid overwriting each other. It also says nothing about how bursts of events are coalesced, what inference now costs, delivery guarantees, failure handling, or where the 10–30 seconds actually goes.
What changed
Airbnb's search ranking uses a vector, or 1, that summarises each guest's history. Until now a nightly batch job produced it. Airbnb has replaced that job with a near-real-time pipeline built on 2, its open-source 3 (a system that computes per-user data values from event streams and serves them at low latency). The team added two general features to Chronon to make this work, 4 and Near-real-time (NRT) Model Transform, and has contributed both to the public repo. Staleness fell from up to about two days to typically 10–30 seconds. Offline ranking quality rose 1.67%, and uncancelled bookings rose about a third of a percent in an A/B test.
The original design
In an earlier post, Airbnb described a Transformer-based sequence encoder. A 5 is a neural-network architecture suited to ordered data, and this one reads a guest's bookings, reviews, views and searches in order and outputs an embedding. Running a multi-layer Transformer on every search request would be too slow. So the team split the work in two:
Offline: each night a batch job processed the previous day's activity and wrote fresh embeddings to a low-latency key-value store, but only for guests with new activity.
Online: when a guest searches, the ranking model reads that guest's embedding from the store and combines it with the live query to score candidate listings.
This is the familiar pattern of precomputing expensive derived state and serving it with a cheap lookup. The cost is that the derived state is only as fresh as the last completed run.
Why it was hard to fix
The authors name three problems.
Two sources of staleness. Activity from earlier in the day waited for that night's run, and the job's own processing lag added more on top. Together that came to nearly two days. Running the batch more often would shrink only the first. The pipeline had to react to individual events.
The encoder was built for snapshots. It expected a full day of events as a scheduled job, not a call per event. Putting it in a streaming path meant changing where it gets invoked, without forking the logic used in training. A fork would risk 6, where the model sees data at serving time computed differently from the data it learned on.
The two mechanisms
Push Mode. Chronon already ran streaming jobs that keep feature values current. Push Mode makes such a job publish a lightweight notification as soon as it commits a new value, so downstream consumers no longer have to poll. Polling a value that changes in bursts either wastes reads or adds latency, depending on the interval. A commit notification turns "something changed" into an event that consumers can act on straight away.
7. This lets a Chronon streaming pipeline call an already-deployed model as a step in the flow and write the output back as an ordinary feature value, ready to serve. Before, running a model meant either embedding its logic in application code or waiting for a batch job. Here the deployed model is the same sequence encoder the batch pipeline uses, which deals with the duplication concern in problem 2.
How an event flows through
An illustrative walk-through, based on the article's description:
A guest views a listing in Lisbon. The view event enters a Chronon streaming job.
The job merges the event into the guest's stored long-term and short-term activity sequences and commits the updated sequence feature.
Push Mode emits a "sequence updated for guest X" notification.
NRT Model Transform picks up the notification and runs the encoder over the merged sequence.
The new embedding goes to the same low-latency store the batch job writes to.
On the guest's next search, the ranking model performs exactly the same lookup as before and gets a vector that includes the Lisbon view.
The read path does not change at all. Only the write path got faster. Batch backfill still exists alongside it: the article's diagram shows an offline path (batch source to Chronon backfill) feeding the same model-transform stage.
Results
Freshness: typical delay is "well under a minute", often 10–30 seconds after the activity, down from roughly two days.
Offline quality: +1.67% 8 (Normalized Discounted Cumulative Gain, a ranking metric that rewards placing relevant results near the top) against the daily-batch baseline. The authors call this large for a system tuned over a decade, where fractions of a percent count.
Online: about +0.33% uncancelled bookings in A/B tests.
The model itself is unchanged. The authors argue this shows freshness alone is a real source of ranking quality: the same representation is worth more when it is current.
What the article doesn't say
For an infrastructure reader, the post leaves out most of the hard parts:
Write ordering and races. The batch job and the streaming path write to the same store. The post doesn't say whether a late batch write can overwrite a newer streaming embedding, or how versions or timestamps settle conflicts.
Bursts and cost. A guest who views a dozen listings in a minute could trigger a dozen encoder runs. The post doesn't mention debouncing, coalescing notifications, or the compute cost compared with the nightly job.
Delivery semantics. There's nothing on whether Push Mode notifications are at-least-once, what happens if the model call fails, or how backpressure is handled when the model service slows down.
Latency breakdown. We get 10–30 seconds end to end, but not how it splits between stream ingestion, the merge, the notification and inference, so we can't tell where the floor is.
Comparability. The authors say streaming embeddings must stay comparable to batch ones, but they don't say how they checked this beyond reusing the same encoder.
The pattern itself is broadly reusable: react to an event, merge it with stored state, compute the derived value straight away, and serve it from the existing store. Anyone running Chronon can now use both building blocks.
Remember
Running batch more often fixes only the schedule part of staleness. Processing lag needs event-driven triggers.
Writing to the same store as the batch job let the serving read path stay untouched.
With the model unchanged, faster updates alone gave +1.67% NDCG offline and about +0.33% bookings online.
Independent event streams. Page views, searches and other interactions arrive on separate streams. Reacting to one in isolation would miss the guest's longer-term profile. Every new event has to be merged with existing state and sent through the encoder the same way the batch did, so streaming embeddings stay comparable to batch ones.
