Red Hat lays out a local AI Inference test loop with Podman and GuideLLM

Red Hat’s Emerging Technologies group has published a hands-on path for running, measuring and manually evaluating an open-weight model on a single workstation. The Oct. 15 guide combines Red Hat AI Inference, Podman, GuideLLM and Open WebUI into one local test loop.

This is an engineering recipe, not a product launch. Its useful contribution is the sequence: bring up an OpenAI-compatible inference endpoint, establish how it behaves under synthetic concurrent traffic, then put the same endpoint behind a chat interface for qualitative checks.

What the recipe assembles

The walkthrough targets RHEL 10.2 on x86-64 and provides separate container commands for NVIDIA and AMD GPUs. It pulls the Red Hat AI Inference 3.5.0 CUDA or ROCm image, mounts a persistent model cache and configures vLLM with a 16,384-token maximum model length. The authors use a single 48 GB NVIDIA L40S with a quantized Qwen3.6 35B mixture-of-experts model as their worked example, while also listing smaller and larger model options by available VRAM. Red Hat’s guide provides the complete requirements and commands.

Once the server is live, the workflow verifies its health endpoint and sends a chat-completions request over the local OpenAI-compatible API. That gives application teams a standard endpoint to exercise before they decide whether the model and hardware combination merits a larger deployment.

Benchmark first, then inspect behavior

For performance testing, the guide runs the GuideLLM 3.5.0 container against that endpoint with synthetic workloads. It supplies balanced, decode-heavy and prefill-heavy token profiles, then demonstrates a concurrency sweep representing 1, 5, 10, 25, 50 and 100 simultaneous streams. GuideLLM writes JSON and HTML results covering time to first token, inter-token latency and throughput, according to the published procedure.

The last stage starts Open WebUI on the same host and points it at the local inference API. Red Hat includes reasoning, summarization and programming prompts so evaluators can compare model behavior alongside the numerical report rather than treating throughput as a proxy for answer quality.

Why it matters

The workflow gives platform and AI teams a reproducible pre-cluster checkpoint. It can expose a poor model-to-memory fit, unacceptable latency under expected concurrency or weak task performance before teams commit shared infrastructure. It also keeps the initial experiment on hardware under the operator’s control instead of requiring an external model API.

Red Hat explicitly labels the Emerging Technologies material as early work that is not necessarily part of a supported product or future roadmap. Teams should therefore treat the commands as a reference workflow and validate support boundaries before carrying the setup into production.