OpenShift tests show where etcd sharding helps hosted control planes—and where it does not

Red Hat's first published test results for etcd sharding in OpenShift hosted control planes show a substantial resilience gain under event-heavy workloads, alongside a memory tax and one unresolved regression. The feature is a technology preview in OpenShift 4.21 and splits resources by kind across multiple etcd instances.

The test compared the standard single-etcd topology with two shards: a main shard for cluster state and a dedicated shard for Events. Both arms ran on identical AWS infrastructure with HyperShift-managed control planes. Red Hat ran three samples per arm for each of four kube-burner workloads.

Event isolation produced the clearest win

During a synthetic storm of 200,000 Event objects, the main shard recorded a 96% reduction in slow applies: a mean of 11.5 versus 281 in the standard arm. The Events shard recorded none. Mean compaction on the main shard fell from 1.06 seconds to 597 milliseconds, while the two shards together completed 60% more WAL fsync operations and 90% more backend commits.

A second workload using 2,000 crashlooping pods produced a more realistic mix of event creates and updates. There, 99th-percentile backend commit latency on the main shard improved by about 24% and mean compaction time fell from 17.6 seconds to 5.64 seconds. Pod-ready latency and workload completion remained comparable between the two topologies.

Annotation churn also improved compaction by roughly 40% without introducing slow operations. These results support the design premise: short-lived diagnostic Events stop competing with durable cluster state for the same write-ahead log and compaction work.

The costs are not theoretical

Two etcd processes used materially more memory. Depending on the workload, combined container memory was about 1.8 to 2.3 times the single-instance result. Red Hat could not explain that increase from database-size metrics and cautioned that it should be treated as capacity-planning input.

The sharded topology also lost under the database-quota-pressure workload, which put about 3.5GB of large custom resources almost entirely on the main shard. Mean compaction took 2.50 minutes, 51.5% longer than the standard arm, and compaction growth was 85.4% higher. The article describes memory pressure as a working hypothesis, not a demonstrated cause.

What platform teams should do

The results justify testing the preview for hosted control planes exposed to event storms, build-farm churn or noisy controllers. They do not justify enabling it from headline percentages alone. The published numbers come from three runs per arm and display one representative run; Red Hat explicitly says they are not capacity guidance.

Evaluations should reproduce the open kube-burner workloads against the target hardware, measure combined etcd memory, and include large custom-resource tests. Event-shard quota exhaustion and more complex shard layouts remain on Red Hat's test list.