Each system today does work only when needed: lazy wire decoding, distributing only big queries, and Cloudflare storage that shows exactly what it gives up.
Picked from 20 new articles across 34 sources · arrow keys for other days
Today in 30 seconds
30s left
PlanetScale's Neki router keeps Postgres wire messages as raw bytes and decodes only what each query needs. Vitess converts every row up front. story 1 ↓
Datadog open-sourced Distributed DataFusion, which splits physical plans at network boundaries, streams over Arrow Flight and keeps small queries on one node. story 2 ↓
Hillel Wayne explains why every TLA+ property must hold per behavior, which leaves out reachability, percentile latency and security hyperproperties. story 3 ↓
Workers KV Instant opens up Cloudflare's Quicksilver: 1.62 ms p99 reads, ~250 ms global writes, 1 MB per namespace, one write/sec. story 4 ↓
Cloudflare K2 builds a Kafka-style log on R2 with stateless brokers and R2 atomics for ordering. Produce p99 is about one second. story 5 ↓
Firebase had a global outage, and Google's poor incident communication shows a hidden cost of depending on a managed backend.
1/5PlanetScale· Dirkjan Bussink· Oct 1
Neki's Postgres sharding router decodes only the bytes each query needs
PlanetScale's router keeps Postgres wire messages as raw bytes and adds a layer of decoding only when a query requires it. Unlike Vitess, it doesn't convert every row up front.
Why it matters · If you build a proxy or a query router, you pay per-row and per-column costs that nobody asked for. Neki shows how to decode lazily without breaking correctness, and which invariants you have to keep to make that safe.
Image: PlanetScale
Neki sits between PostgreSQL clients and sharded Postgres servers. The extra network hop can't be removed, so Dirkjan Bussink says the performance work comes down to skipping work: don't decode a result the router never inspects.
The Vitess baseline
Vitess decodes everything when a result crosses an RPC boundary. proto3ToRows calls MakeRowTrusted, which allocates a slice for every row and a Value for every non-NULL column. Sending the result back walks those values twice. The query engine gets one consistent type, but every result pays for the conversion even when the router reads nothing in it.
Neki's ExecuteResponse protobuf is mostly a single raw byte field holding Postgres response messages. NewQueryResponse walks the five-byte message headers, checks lengths, handles CommandComplete, and stores the region holding the DataRows as one slice. It goes further only when a query needs it.
Decoding in layers
DataRows has three representations, each built on demand:
Raw bytes. Point lookups and scatter queries with no ordering are forwarded to the client unchanged. For scatter queries, only the per-shard CommandComplete messages are merged (Collapse).
A row slice pointing into the buffer.LIMIT needs row boundaries. One pass counts DataRows so the slice can be allocated once, and a second pass builds wrappers.
2/5Datadog Engineering· Oct 1
Datadog's Distributed DataFusion runs one DataFusion query across many machines
An open-source Rust library splits DataFusion physical plans at network boundaries, streams data over Arrow Flight, and keeps small queries on one node.
Why it matters · It is a worked example of adding distribution to a single-node engine at the physical-plan layer. It also shows the trade-off between streaming and materializing intermediate data, and why the cost model that decides when to distribute matters as much as the distributed operators themselves.
Distributed ORDER BY plan: two workers' streams gathered by a NetworkCoalesce step.1 / 4
Datadog has standardized on Apache Arrow for in-memory data, Substrait for execution plans and Apache DataFusion as its query engine. Together they replace a set of separate engines built for each product. DataFusion only runs a query on one machine, so the team built Distributed DataFusion, an Apache-2.0 Rust library that spreads a single query across nodes.
The team says existing engines didn't fit. ClickHouse lacked the extensibility they needed. Spark and Ballista write intermediate data to storage between steps, which is too slow for interactive queries where a person is waiting on the result. Trino integrates poorly with Arrow and Substrait and is expensive to run at Datadog's scale. They wanted a framework for building an engine, with zero-copy streaming.
3/5Hillel Wayne· Sep 30
The properties TLA+ can't express: reachability, multi-step safety and hyperproperties
Hillel Wayne explains what TLA+ can actually state about a system. Every TLA+ property must hold for each behavior on its own, and that rules out a lot.
Why it matters · If you use TLA+ to check a concurrent design, you need to know which guarantees it can state at all. Percentile latency, security properties and "this state is reachable" are outside what it handles natively.
Image: Hillel Wayne
Formal verification is getting attention again after reports that an LLM used TLA+ to find race conditions. Hillel Wayne, a long-time TLA+ teacher, argues that the excitement skips a basic limit. You can only verify a property you can write down, and TLA+ cannot express many properties people care about.
What TLA+ can check
A TLA+ spec describes a set of behaviors, where each behavior is a sequence of states. Properties are built from boolean state expressions plus three temporal operators:
[]P (always)
P' (true in the next state)
<>P (eventually)
These combine into three main kinds of property:
Invariants such as [](at_most_one_green) say something about every single state.
Action properties such as [](x' >= x) constrain a single step.
Liveness properties say something good eventually happens. []<>P fits recovery, like a leader election that eventually reaches agreement. <>[]P fits termination with a correct result. P ~> Q (leads-to) fits guarantees like "every message put on the queue is eventually read."
Refinement mixes safety and liveness. Wayne says these kinds cover most of what people check in practice.
What it can't express
4/5Cloudflare· Rob Sutter· Oct 1
Cloudflare opens its internal Quicksilver store as Workers KV Instant
The new KV mode claims 1.62 ms p99 reads and about 250 ms global write replication, capped at 1 MB per namespace and one write per second.
Why it matters · This is a clear example of an explicit trade-off: you get push-replicated, read-optimized config storage by giving up capacity, write throughput and per-write cost. It shows what "instant global config" costs when nothing is cached lazily.
Image: Cloudflare
This is a product announcement, but the design trade-offs are the useful part. Cloudflare is putting its existing Workers KV API on top of Quicksilver v2, the internal key-value store it uses for its own configuration. Cloudflare says nearly every request it serves looks up at least one Quicksilver key. The new mode, KV Instant, is in private beta.
Push instead of cache
Classic Workers KV reads through caches, so a cold read is slow and a stale value stays until its TTL expires. KV Instant instead pushes every write to all of Cloudflare's 300+ edge locations, so each location holds the full dataset. Cloudflare's numbers:
p99 reads: 1.62 ms for Instant. Classic is 287 ms overall and 160 ms for cached reads. p95 Instant reads are in microseconds.
Write replication to all edge locations: median 107 ms, p95 181 ms, p99 256 ms. Classic reaches p99 at 4.38 s.
The two modes don't measure replication the same way. Instant's figure is time to reach every edge location. Classic's is time to replicate across its storage backends, and Cloudflare says classic has no sub-second measurement. So the "20x faster writes" claim compares slightly different things.
What the guarantee costs
Copying everything everywhere only works if the dataset is tiny, and the limits reflect that:
A namespace can hold at most 1 MB and 10,000 keys. Keys are limited to 300 bytes.
Writes are limited to , because all writes go through a single system of record. Classic allows one write per per second. Cloudflare presents the limit as a way to make update order easier to reason about.
5/5Cloudflare· Micah Wylde· Oct 1
Cloudflare K2 builds a Kafka-style partitioned log on R2 object storage
The new serverless event stream batches writes into R2 segments and relies on R2's atomic operations for ordering. The cost: about 1s p99 produce latency.
Why it matters · K2 is another data point for the log-on-object-storage design. Durability and consensus move to S3-like storage, and the broker becomes thin and stateless. It also shows the latency you pay for that, measured on a production system.
Image: Cloudflare
This is mostly a launch post. The real design write-up is promised for a later "technical deep dive." Still, it gives the architecture's shape and its main measured cost.
Why not just run Kafka
Cloudflare first built K2 as the durable ingestion buffer for Basin Pipelines. Pipelines uses a pull-based stream processor, so something has to hold events before they're read. Cloudflare also commits to never dropping an event once it's accepted. That rules out a lossy buffer.
The usual answer would be Kafka, but Wylde says Cloudflare's edge doesn't suit it. Each service gets small slices of machines, those machines are fairly ephemeral, and traffic between them often crosses the public Internet. Those are poor conditions for a stateful system that replicates its own data.
Log on object storage
Instead, K2 hands durability and consensus to R2, Cloudflare's object store. R2 is rated for eleven 9s of durability and has strongly consistent APIs. The author argues this makes the application layer "radically simpler, cheaper, and higher performance." It also separates compute from storage, so long retention is cheap.
Object stores can't append to a file, so K2 can't write a log the normal way:
An edge service holds incoming writes in memory for a short window.
It then writes the whole batch to R2 as one segment file, large enough to make each write and read worth its cost.
Ordering and strictly increasing offsets come from R2's atomic operations. No separate coordination service is involved. The post doesn't say which operations, presumably conditional writes.
A pointer slice. This lets the router take a subset or reorder rows without moving any messages.
ORDER BY created_at needs column offsets. The router reads each column's 4-byte length to skip earlier columns, then decodes and caches only created_at, because a row is compared several times during the merge. If created_at isn't in the SELECT list, SimpleProjection drops it by copying runs of adjacent column frames. {0,1} becomes a single append into a shared wire buffer, with a new header and a column count of 2. So the router decodes the column the client never sees and leaves the requested columns as bytes.
Bind parameters get the same treatment. Neki decodes only the sharding key, writes the original frames when it forwards, and rearranges frames with mappings like [1,0,1] when it rewrites a query. Values stay in the client's text or binary format. Only a change in client_encoding forces text parameters to be recoded.
Some queries still need full decoding. A cross-shard GROUP BY status, count(*) must decode values and serialize new ones.
Correctness and invariants
Bussink argues that passing bytes through is also about correctness. A NaN payload can change if it is parsed and re-serialized. Extension types can be forwarded without the router understanding them. Record typmods can't always be recovered from the wire bytes. The approach depends on several invariants:
Shared buffers must stay alive while any row points into them.
Three-index slices stop an append through one row from overwriting the next message.
Each consumer replaying a row gets its own wrapper and value cache.
Memory accounting has to cover rows that grow as values are cached.
The post reports no benchmark numbers.
Remember
Neki forwards raw Postgres DataRows untouched for point lookups and unordered scatter queries, with no per-row or per-column allocation.
Projection copies runs of adjacent column frames as byte ranges. Sorting decodes and caches only the sort key.
Passing bytes through also protects correctness: NaN payloads, extension types and record typmods can change if decoded and re-encoded.
Distribution happens only in the physical plan. SQL or Substrait parsing and logical planning are unchanged. The library extends DataFusion's existing partitioning model, where non-overlapping data streams run in parallel, from CPUs on one machine to multiple machines. A plan is cut into stages at network boundaries (other engines call these fragments). Each stage is split into tasks, and each task runs on one worker. A task plays the role a partition plays on a single node.
For a ORDER BY ... LIMIT 10 query on two workers with two CPUs each, four streams run in parallel. A NetworkCoalesce node pulls the remote streams over the network and merges them.
Aggregations take more work. A grouped average runs in three steps:
Partial aggregation: each partition emits a sum and count per group.
Hash repartition: rows are redistributed by grouping key.
Final aggregation: each partition combines the partial results for the keys it owns.
In the distributed version, a NetworkShuffle node exchanges the locally repartitioned data between machines, so each key belongs to exactly one partition across the whole cluster. Final aggregation then runs independently, and NetworkCoalesceExec plus CoalescePartitionsExec collect the result. The same building blocks support partitioned and broadcast joins, distributed unions and tree-reduce aggregations. Data moves over Arrow Flight streams.
Results and trade-offs
The benchmark ran on a 12-node c5n.2xlarge cluster (8 vCPU, 21 GiB, up to 25 Gbps per node) reading Parquet from S3. It used TPC-H and TPC-DS at 1–100 GB and compared against Ballista, Spark and Trino. The post shows a normalized chart but no per-query figures in the text. The benchmark setup is public and reproducible. In production, some heavy queries that struggled on one node now finish up to 10× faster at the same total cost.
The authors argue the main lesson is knowing when not to distribute. Network transfer, serialization and coordination all cost something. The planner keeps queries on a single worker when the expected scan size and compute are small. That decision depends on statistics, and custom data sources often don't provide them. Adaptive execution, which changes strategy based on the data volume actually observed, is next on the roadmap. The team is also exploring GPUs, using distribution to pool the limited memory of several devices.
Remember
Distribution happens only in the physical plan: stages split at network boundaries, and each task runs on one worker.
NetworkShuffle exchanges data between machines by grouping key, which makes distributed partial-then-final aggregation correct.
Some heavy production queries run up to 10× faster at the same cost, while small queries stay on one node.
Properties nobody can formalize. No formal method can help here; Wayne's example is the human notion of "a bird."
Multi-step safety. Invariants look at one state and action properties at one step. So "delete then undo restores the original state" and "the machine turns on within ten steps of pressing power" are out. TLA+ also has no floating-point arithmetic and no real time, only logical time.
Anything that isn't a property of each behavior on its own. Every property is implicitly checked against every behavior, one at a time. That rules out:
reachability, "there exists a behavior where P holds" (for example, proving a game is winnable);
hyperproperties, which compare sets of behaviors. Wayne's example is "power-saving mode never uses more energy than normal mode for the same actions." Disproving it needs two behaviors side by side. Many security properties and all statistical ones, such as "p95 latency is 5 ms," fall into this group;
state-space metaproperties, such as "there is only one path from X to Y."
The workarounds
Each gap has a partial fix:
auxiliary history variables for multi-step properties
self-composition, where each behavior of a combined spec stands for two system behaviors, for some hyperproperties
TLC's new REACHABLE keyword for basic reachability, and TLCGet for some state-space checks
fairness tricks for "always reachable"
Wayne calls these hacks, and each has a cost. History variables break refinement. Self-composition grows the state space exponentially. The hacks also combine poorly with other TLA+ features, and they make specs stop resembling the real system.
Other tools fill some gaps: CTL handles reachability and PRISM handles probabilistic properties. Both are weaker than TLA+ at what TLA+ does well. Wayne concludes that TLA+ is good for the easy wins, invariants and liveness, and that it is not a full answer to correctness.
Remember
TLA+ natively checks invariants, single-step action properties, liveness and refinement, and each must hold for every behavior.
Reachability, multi-step safety and hyperproperties like p95 latency or security comparisons can't be stated directly.
Workarounds such as history variables and self-composition exist, but they break refinement or grow the state space exponentially.
Metadata isn't supported (getWithMetadata returns null), and list returns the whole namespace in one unpaginated response.
You choose the mode when you create the namespace ("mode": "instant").
The pricing follows the same logic. Reads cost $0.20 per million keys versus $0.50 for classic, which is 60% cheaper. Each key in a multi-key get is billed separately. Writes, deletes and lists cost $0.10 per operation, against $5.00 per million in classic. Storage is $100 per MB-month, against $0.50 per GB-month.
Cloudflare's intended use is feature flags and settings that are read on every request and change rarely, such as switching a homepage at a set keynote time. Large or frequently updated data should stay in classic KV. The post does not say how consistency works during the roughly 250 ms propagation window, how the single writer handles failures, or what readers see while a write is still spreading.
Remember
KV Instant pushes every write to 300+ edge locations: p99 reads take 1.62 ms, and p99 global replication takes 256 ms.
Limits: 1 MB and 10,000 keys per namespace, one write per namespace per second, no pagination, no metadata.
Reads cost 60% less, but each write costs $0.10 and storage costs $100 per MB-month. Use it only for small, rarely changed config.
The cost is latency. Waiting for a batch to fill, then writing to object storage instead of local disk, adds up to roughly 1 second of produce latency at p99.
Consumption model
Consumers read through subscriptions. A consume call takes a 5-minute lease on a batch of events. The consumer can then:
ack the batch, so it won't be redelivered
nack it, to have it redelivered
extend the lease if it needs more time
Several consumers on one subscription split the work. Giving each consumer its own subscription is pub/sub fan-out, where every consumer sees every message. Ack and retry work on whole batches, not single messages. That's how Cloudflare separates K2 from Queues, which handles per-item retries, delays and dead-letter queues.
Limits and pricing
The beta requires Workers Paid and caps usage at 10 GB stored and 30 MB/s produce per stream. It's free during the beta. Expected pricing afterward is $0.04/GB produced, $0.04/GB consumed and $0.02/GB-month retained.
The roadmap includes multi-GB/s streams, message keys with per-key ordering, push-based consumers, an "Express" tier with lower latency, and support for unmodified Kafka clients. Per-key ordering isn't available yet, so today's ordering guarantees need checking before relying on them.
Remember
K2 offloads replication and consensus to R2, using atomic object operations for offsets instead of a coordinator.
Object stores can't append, so writes are batched in memory into segments, costing ~1s p99 produce latency.
Consumers take 5-minute batch leases with ack/nack/extend; there are no per-message retries or key ordering yet.
