Inferact finds a throughput case for two-stage vLLM prefill
On two NVIDIA B300 GPUs, a tuned pipeline served more MiniMax requests at a two-second median first-token target, but tensor parallelism won at one second.
Inferact’s November engineering report makes a case for adding pipeline parallelism to the shortlist when tuning vLLM for long-context agent workloads. Serving MiniMax-M3.1-Flash on two NVIDIA B300 GPUs, the team reported that a tuned two-stage pipeline sustained 23% more requests per GPU than data-expert parallelism and 17% more than tensor parallelism at a two-second median time-to-first-token target. At a one-second target, tensor parallelism sustained the highest rate. The result is a trade-off between latency and capacity, not a default configuration change.
Why the pipeline held up
Prefill processes input tokens before generation can begin. In Inferact’s pipeline, each GPU ran half the model’s layers, and the first passed intermediate results to the second once per batch. The compared tensor configuration synchronized within each layer, while the data-expert configuration could leave a rank waiting when the other had more work. Asynchronous scheduling kept multiple batches in flight, and a CUDA copy-engine hand-off reduced time spent waiting between stages. In a separate comparison against the unchanged day-zero image, the hand-off lowered median first-token time by 8.5% to 16% at the four tested request rates, with little change to the saturated rate, Inferact reported.
What serving teams should compare
Use your own prompt mix and latency target to decide whether the extra capacity is worth it. Inferact tested a text-only workload on one model with two GPUs per instance and a shared 100,000-token prefix. Pipeline and data-expert parallelism finished within 1% of each other at saturation, and the team did not test token-aware routing for the data-expert comparison or larger GPU counts. If your service is judged on tail latency, measure that alongside throughput rather than treating a median target as a proxy.
Check the software build before attempting to reproduce the result. The upstream pull request documents changes merged into the benchmark branch, including workspace allocation for B200/B300 and support for fetching cached state from multiple prefill stages. That is evidence of what the benchmark used, not evidence that a standard vLLM release includes the same changes.
sources
comments · 0