IBM maps the six controls that turn model servers into an inference system

IBM Research has published a useful map of the machinery between an efficient model server and a production inference service. Its Oct. 12 explainer describes six controls in llm-d: prefix-cache-aware routing, KV-cache offloading, flow control, batch scheduling, prefill/decode disaggregation and wide expert parallelism.

That distinction matters for platform teams. vLLM or SGLang can run a model efficiently on one server, but an agent platform also has to place requests across replicas, preserve reusable context and keep background work from crowding out interactive traffic. IBM’s argument is that distributed inference is a coordination problem, not simply a matter of adding accelerators.

Route work to context, not just capacity

Agent traffic repeatedly revisits long system prompts, repositories and tool histories. A conventional load balancer can send the next request to an idle replica that does not hold the relevant KV cache, forcing the model to process the shared prefix again. IBM says llm-d instead scores cache locality alongside queue and capacity information before placing work.

The accompanying llm-d GLM-5.2 study gives that mechanism a production-shaped workload: 219 Claude Code sessions containing 78,099 requests. In that corpus, 96% of main-agent turns reused at least 90% of their input as a verbatim prefix. In the project’s measured serving runs, prefix-aware routing captured 96% to 97% of the reuse available to the workload.

GPU memory is still finite. llm-d can move KV state into CPU memory or storage rather than discard it, then use the cache index during routing. IBM’s explainer also describes flow control as the guardrail that gives latency-sensitive work priority while pacing bursty agents and deferrable jobs.

Separate unlike work

Long prompts make prefill compute-heavy; token-by-token generation makes decode sensitive to memory access and communication. llm-d can place those phases on separate worker pools and scale them independently. Its batch gateway and asynchronous processor likewise keep job admission separate from request placement, allowing queued work to consume spare capacity without releasing an entire batch into the serving cluster at once.

For mixture-of-experts models, wide expert parallelism distributes experts across GPUs and servers instead of repeatedly synchronizing every model operation. The GLM-5.2 study reports 65.1 GiB of expert weights per GPU in its wide layout versus 89.5 GiB in the tensor-parallel comparison, leaving three to nine times as much KV-cache capacity depending on topology.

What platform teams should test

The practical lesson is to benchmark the traffic shape, not just the model. Teams serving agents should measure prefix reuse, pauses between turns, burst concurrency and the ratio of prompt processing to generated tokens. Those measurements determine whether the next useful control is cache-aware routing, a CPU cache tier, more prefill workers or stronger admission control.

IBM’s article is an explainer rather than a new llm-d release. Its value is architectural: it shows how the project’s individual features fit together, and where a Kubernetes inference platform can waste expensive capacity even when each model server is fast in isolation.