Hot Tenants, Honest Watermarks, Retired Styled-Components

Simeon Griggs of PlanetScale starts from a common situation. A database is sharded on tenant_id, which spreads tenants evenly at first. Then a few tenants grow far larger than the others, and the product starts needing cross-tenant queries. At that point an even spread of tenants matters less than an even spread of data. Griggs's diagnosis is that you probably have the wrong shard key, at least for some tables.

The post is written for Neki, PlanetScale's sharded Postgres product built by the Vitess maintainers, so it is partly a vendor pitch. The reasoning applies to any range-sharded system.

The Slack precedent

Slack first sharded messages and channels by workspace ID and kept the workspace-to-shard mapping in a separate metadata cluster. That rested on the assumption that no single customer would outgrow the largest database. It stopped holding once enterprise workspaces passed 100,000 users. Some workspaces' messages tables grew too large for any one shard. Then cross-workspace messaging and Enterprise Grid, which groups workspaces under one organization, made the workspace key awkward.

Slack moved to Vitess and chose a shard key per table: user ID, channel ID or workspace ID. Messages were always queried by channel and never by workspace, and channel ID was already in the primary key. Sharding messages by channel turned "fetch a channel and its messages" into a single-shard query, even when the authors came from different workspaces. Cross-channel queries became cross-shard, but those were limited to admin and batch jobs. Users stayed together by workspace because "all users in this workspace" is a hot access path.

Three fixes, in order

This is the third post in Pinterest's series on its CDC ingestion framework, which runs Kafka → Flink → Iceberg CDC table → Spark upsert into a base table. It covers partition finalization: how a downstream batch job that runs once per hourly partition knows the hour won't change again.

In the old batch system the answer came for free. One job read the whole source DB, so when it finished, any partition with data was complete. In the streaming system, Flink commits small batches on every checkpoint. Event time can trail processing time, so an hour can keep receiving rows after the wall clock has moved past it. The answer has to be worked out from the stream itself, and in the same way across thousands of pipelines.

Facts per commit, one decision per table

• Extraction: a pluggable extractor reads each pipeline's configured event-time column and normalizes it to a timestamp.
• Facts: between checkpoints, each writer subtask tracks min and max event time plus a of percentiles (p95, p99). The t-digest can be merged, so the committer combines the subtask sketches in any order. The result goes into the Iceberg and stays at a few hundred bytes whether the checkpoint saw a thousand records or millions.

Netflix's internal services authenticate each other with mutual TLS using short-lived X.509 certificates from Metatron, a private PKI. A certificate is issued only after attestation, meaning proof that the requester really is the workload it claims to be. On EMR, though, a Spark job starts with only an AWS execution role. Its data access is governed by a Data Project, the unit that owns tables. Jobs run as the Data Project rather than as the person who launched them, so table ACLs and audit stay consistent.

The mapping comes first

Each Data Project identity maps 1:1 to its own IAM role, and the Data Project service is the authority for that mapping. Dhruv Pratap argues this mapping is the real hard part: "no amount of cryptography helps" without it. Netflix expects tens of thousands of Data Projects, more than one AWS account can hold as IAM roles. Roles are sharded deterministically across a small pool of dedicated accounts. As a side benefit, workload roles sit across an account boundary from the control planes that launch them.

PlanetScale's Ahmed Darwich explains how a database's CPU architecture affects Postgres performance and why you can't change it later on PlanetScale Postgres or Neki clusters.

A vCPU is not a fixed unit

On x86 instances with hyperthreading, each hardware thread is sold as a vCPU. Two threads share one core's execution units, and the core switches to the other thread when one stalls on memory. ARM server chips such as AWS Graviton and Google Axion don't use hyperthreading, so each vCPU is a whole core.

Darwich argues this gives ARM the advantage for high-concurrency OLTP, where many small queries spread across cores. The extra cores also leave room for parallel query workers and autovacuum. x86 usually runs at higher clock speeds, so it tends to win when one CPU-bound query doesn't use parallel workers, such as a dashboard aggregate scanning millions of rows.

Vector width matters too. x86 offers 256-bit AVX2, and some chips add AVX-512. Graviton4 and Graviton5 use 128-bit vectors. PlanetScale's TIN full-text index stores a 256-bit bitmap per page for each term and intersects two terms with a single AVX2 AND. pgvector has AVX-512 code paths for Hamming and Jaccard distance. On ARM it falls back to general-purpose code, so some index types run better on x86.

Why physical replicas can't cross architectures

A Postgres replica starts from a byte-for-byte copy of the primary's data files and then replays the WAL. That only works if both machines read those bytes the same way. The article's example is char signedness. C leaves it to the compiler, and GCC and Clang make plain char signed on x86-64 Linux and unsigned on ARM64 Linux. So the byte reads as −1 on one and 255 on the other. pg_trgm sorts trigrams inside its indexes using char comparisons, so a copied index can disagree with the new machine's ordering and return wrong results. Postgres 18 fixed this case, but the general problem is still there: identical bytes don't guarantee identical meaning.

By 2023, some GitHub pages rendered so many components that the runtime cost of its CSS-in-JS styling (styled-components) became a bottleneck. Styles were set up on the client, which slowed initial loads. Collecting styles during server-side rendering slowed SSR. Style updates grew with the number of components on a page. The Primer design system team chose CSS Modules: plain CSS files next to each component, with class names local by default and no runtime on either client or server. The styles compile into stylesheets that ship with the HTML.

Migrating without breaking GitHub

Each Primer component followed the same steps:

• Translate its styles into a CSS Module file.
• Put a feature flag in front of it to switch between old and new styles.
• Use existing visual regression tests to confirm the snapshots matched exactly.
• Roll it out to the team, then GitHub staff, then all users.

Primer finished in December 2024. Server-side rendering took 55% less time and component initialization 25% less.

The site itself was harder to move. Over the years, the prop had become GitHub's standard way to style things. It let callers pass inline style objects to Primer components. Teams liked it for its TypeScript and design-token integration and for keeping styles next to the component. But those dynamic objects were expensive at runtime and scaled badly as component counts grew. To keep old call sites working, the team built a shim package, . It wrapped the migrated components so they still accepted . Code that no longer used imported directly and got the speedup right away.