vLLM Optimizer

Find better vLLM serving settings for your workload. Define a search space in YAML; vllm-opt runs the benchmarks and reports the best observed settings.

Run your first experiment · Documentation · PyPI

This is independent community alpha software, not affiliated with vLLM. Execution needs Python 3.11–3.12, Linux/WSL, an NVIDIA GPU, and a local model. Native Linux, L40/H100, tensor-parallel, and multi-GPU evidence remain hardware validation pending. See the compatibility matrix.

Install

Use a virtual environment. The runtime extra installs vLLM and GuideLLM; the core package requires an existing runtime to execute experiments. See installation choices for setup and existing CUDA environments.

First experiment

The quick start contains one complete example:

1. Install the runtime in a virtual environment.
2. Copy the YAML and set your local model path. Models are not downloaded.
3. Validate and run it; open the printed report.html path.

Read the walkthrough in 5–10 minutes. Installation and benchmark time depend on your machine and model.

What you can tune

• Fixed or tunable vLLM flags and environment variables.
• Grid, Random, or TPE search with a baseline comparison.
• GuideLLM or native vLLM Bench Serve workloads.
• Throughput objectives subject to latency/throughput targets, such as p99 TTFT ≤ "200 ms" or request throughput ≥ "50 req/s".

Search keeps its configured budget after meeting a target. Only configurations satisfying every limit can win. See search and scoring.

See a real report

Open the interactive RTX 3080 report · Experiment YAML · Baseline-control example

This deliberately small workload measured 194.34 → 1,082.27 output tokens/s. It demonstrates reporting and latency trade-offs; it is not a production benchmark or an expected improvement for other workloads. The published example used vLLM 0.28.0 and GuideLLM 0.7.3; dated runtime details and required WSL flags are in compatibility.

Next steps

Use vllm-opt --config experiment.yaml --verbose to stream logs. Full trial logs are saved regardless of verbosity; NO_COLOR disables terminal colors.

Install vllm-optimizer, import vllm_optimizer, and run vllm-opt. The former vtune aliases are deprecated; see migration.

Contribute

Contributing · Architecture · Roadmap · Release notes