live wire
▸IDENTITY · Keycloak 27 plans to remove Fine-Grained Admin Permissions V1 and Token Exchange V1Keycloak project▸AI · Inferact tests vLLM pipeline parallelism for prefill, reporting up to 23% more requests per GPU at a 2-second targetInferact Engineering (November 2026)▸SECURITY · Red Hat updates standard OpenJDK 21 RPMs for Hardened Images after earlier portable-package fixRed Hat Product Errata▸PLATFORM · Red Hat Multiarch Tuning 1.4.1 makes pod placement plugins optional and adds a downgrade caveatRed Hat Product Errata▸SECURITY · Red Hat updates OpenJDK 21 RPMs for Hardened Images to fix Java4ShellRed Hat Product Errata▸PLATFORM · Red Hat tests OpenShift Virtualization 4.22 with 10,000 active VMs during upgradesRed Hat Developer▸AI · vLLM adds day-zero support for Thinking Machines’ looped Inkling 2 MinivLLM Blog▸SECURITY · Red Hat rates OpenShift Dev Spaces 3.31.4 a Critical security updateRed Hat Product Errata▸SECURITY · Quarkus tells 3.27 users to switch from UBI 9 1.23 to 1.24 for the Java4Shell fixQuarkus Blog▸SECURITY · Red Hat issues Critical OpenJDK 25.0.5.1 update for WindowsRed Hat Product Errata▸PLATFORM · Red Hat resolves Lightspeed email outage; about 42 customers may miss daily summariesRed Hat Status▸JAVA · Red Hat details how to diagnose HikariCP connection pool stalls on OpenShiftRed Hat Developer▸SUPPLY CHAIN · Red Hat publishes Renovate presets for Lightwell dependency rebuildsRed Hat Developer▸PLATFORM · Red Hat resumes Lightspeed email delivery after outage, warns of delayed notificationsRed Hat Status▸IDENTITY · Keycloak 27 plans to remove Fine-Grained Admin Permissions V1 and Token Exchange V1Keycloak project▸AI · Inferact tests vLLM pipeline parallelism for prefill, reporting up to 23% more requests per GPU at a 2-second targetInferact Engineering (November 2026)▸SECURITY · Red Hat updates standard OpenJDK 21 RPMs for Hardened Images after earlier portable-package fixRed Hat Product Errata▸PLATFORM · Red Hat Multiarch Tuning 1.4.1 makes pod placement plugins optional and adds a downgrade caveatRed Hat Product Errata▸SECURITY · Red Hat updates OpenJDK 21 RPMs for Hardened Images to fix Java4ShellRed Hat Product Errata▸PLATFORM · Red Hat tests OpenShift Virtualization 4.22 with 10,000 active VMs during upgradesRed Hat Developer▸AI · vLLM adds day-zero support for Thinking Machines’ looped Inkling 2 MinivLLM Blog▸SECURITY · Red Hat rates OpenShift Dev Spaces 3.31.4 a Critical security updateRed Hat Product Errata▸SECURITY · Quarkus tells 3.27 users to switch from UBI 9 1.23 to 1.24 for the Java4Shell fixQuarkus Blog▸SECURITY · Red Hat issues Critical OpenJDK 25.0.5.1 update for WindowsRed Hat Product Errata▸PLATFORM · Red Hat resolves Lightspeed email outage; about 42 customers may miss daily summariesRed Hat Status▸JAVA · Red Hat details how to diagnose HikariCP connection pool stalls on OpenShiftRed Hat Developer▸SUPPLY CHAIN · Red Hat publishes Renovate presets for Lightwell dependency rebuildsRed Hat Developer▸PLATFORM · Red Hat resumes Lightspeed email delivery after outage, warns of delayed notificationsRed Hat Status
upstreambeat.ai
analysisAI

Inferact finds a throughput case for two-stage vLLM prefill

On two NVIDIA B300 GPUs, a tuned pipeline served more MiniMax requests at a two-second median first-token target, but tensor parallelism won at one second.

By The News Desk· Nov 21, 2026the quick take — two AI hosts go live when you do

Inferact’s November engineering report makes a case for adding pipeline parallelism to the shortlist when tuning vLLM for long-context agent workloads. Serving MiniMax-M3.1-Flash on two NVIDIA B300 GPUs, the team reported that a tuned two-stage pipeline sustained 23% more requests per GPU than data-expert parallelism and 17% more than tensor parallelism at a two-second median time-to-first-token target. At a one-second target, tensor parallelism sustained the highest rate. The result is a trade-off between latency and capacity, not a default configuration change.

Why the pipeline held up

Prefill processes input tokens before generation can begin. In Inferact’s pipeline, each GPU ran half the model’s layers, and the first passed intermediate results to the second once per batch. The compared tensor configuration synchronized within each layer, while the data-expert configuration could leave a rank waiting when the other had more work. Asynchronous scheduling kept multiple batches in flight, and a CUDA copy-engine hand-off reduced time spent waiting between stages. In a separate comparison against the unchanged day-zero image, the hand-off lowered median first-token time by 8.5% to 16% at the four tested request rates, with little change to the saturated rate, Inferact reported.

What serving teams should compare

Use your own prompt mix and latency target to decide whether the extra capacity is worth it. Inferact tested a text-only workload on one model with two GPUs per instance and a shared 100,000-token prefix. Pipeline and data-expert parallelism finished within 1% of each other at saturation, and the team did not test token-aware routing for the data-expert comparison or larger GPU counts. If your service is judged on tail latency, measure that alongside throughput rather than treating a median target as a proxy.

Check the software build before attempting to reproduce the result. The upstream pull request documents changes merged into the benchmark branch, including workspace allocation for B200/B300 and support for fetching cached state from multiple prefill stages. That is evidence of what the benchmark used, not evidence that a standard vLLM release includes the same changes.

Filed by The News Desk. Corrections: desk@upstreambeat.ai · Our standards →

comments · 0

    Comments are moderated before they appear. Your email is used once to confirm it is you — never shown, never sold. Corrections and questions get an answer from the desk when we have one.