vLLM Optimizer

Alpha status: native Linux, real vLLM 0.28, L40/H100, tensor-parallel, and multi-GPU evidence remain hardware validation pending.

vLLM Optimizer is a local-first benchmarking and optimization tool for vLLM serving configurations. Users define the parameters and workloads they care about; the vllm-opt CLI manages the server lifecycle, runs repeatable benchmarks, explores the search space, and reports which configurations performed best.

vLLM Optimizer is alpha software targeting Linux with NVIDIA GPUs and Python 3.11–3.12. Install vllm-optimizer, import vllm_optimizer, and run vllm-opt. The former vtune aliases remain available for one release cycle.

This independent community project is not affiliated with the vLLM project.

Documentation · Quick start · PyPI

The current code is verified with vLLM 0.28.0 and GuideLLM 0.7.3 on WSL2 with an RTX 3080. That host required VLLM_USE_V2_MODEL_RUNNER: "0" because UVA was unavailable and VLLM_USE_FLASHINFER_SAMPLER: "0" because the CUDA compiler toolkit was not installed. Native Linux systems may not require these settings.

Other combinations may work but are not yet verified.

Each new trial stores a typed execution assignment in its trial result and manifest. a5/a6 runs may lack it. Reports show only statistics supplied by the benchmark backend; a7 offline regeneration corrects derived a6 summaries in a new destination without changing the source run.

The published py3-none-any wheel installs on Linux and Windows. Configuration validation and stored-result inspection work on Windows, but starting an experiment is supported only on Linux because vLLM has no native Windows runtime.

Installation

Choose the installation that matches what you want to do:

The core package intentionally does not install GPU frameworks. The runtime extra adds vLLM and GuideLLM, which select large PyTorch/CUDA dependencies for the machine. See the installation guide for virtual environments, CUDA guidance, and verification commands.

Quick start

Create and activate a Python 3.11 or 3.12 virtual environment on Linux or WSL, then install the complete experiment runtime:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install "vllm-optimizer[runtime]"
vllm --help
guidellm --help

Create experiment.yaml:

experiment:
  name: first-run
server:
  model: /models/opt-125m
  gpu-memory-utilization: 0.8
tune:
  max-num-seqs:
    values: [8, 16]
benchmark:
  engine: guidellm  # Default. Use vllm for `vllm bench serve`.
  max_failure_percentage: 2  # Accept up to 2% errored or incomplete requests.
  # accept_any_request_failures: true  # Ignore the percentage (one success still required).
  runs:
    - name: throughput
      profile:
        kind: throughput
        max_concurrency: 16
      constraints:
        - kind: max_requests
          count: 10
      data:
        - kind: synthetic_text
          prompt_tokens: 32
          output_tokens: 16
optimization:
  maximize: output_tokens_per_second
  sampler: tpe
  trials: 2
timeouts:
  benchmark: 20m

Run it:

vllm-opt --config experiment.yaml

The short form is vllm-opt -c experiment.yaml. The command validates the file, runs the experiment, persists results, and generates its exports and report. The vllm-opt CLI binds vLLM to 127.0.0.1 by default. Set server.host explicitly only when the benchmark server must be reachable from another host.

Already have a native vllm serve YAML file? Reference it instead of copying its values into experiment.yaml:

server:
  config: ./vllm-config.yaml
  gpu-memory-utilization: 0.9  # Optional CLI override of the native file.

The path is resolved relative to experiment.yaml. The native file must define model; any other server settings, tuned values, and runtime-assigned values are passed on the command line and take precedence over the native file.

LMCache runtime verification was completed on an RTX 3080 with vLLM 0.28.0, LMCache 0.5.5, and Qwen2.5-0.5B-Instruct. Both the native YAML form above, with a nested kv-transfer-config, and the legacy inline JSON form launched successfully, completed two shared-prefix inference requests, stored 160 tokens, and reported a 160-token LMCache hit. The Ubuntu WSL test environment used VLLM_USE_V2_MODEL_RUNNER=0 to work around vLLM's WSL UVA limitation; this setting is not required by the configuration mapping feature.

Fixed vLLM flags go directly under server; tunable flags use top-level tune. Fixed and tunable environment variables use env and tune_env. See the configuration guide for categorical, boolean, integer-range, float-range, list, and environment examples. The complete YAML and benchmark guide show every supported control with copyable examples.

To use vLLM's native benchmark, set benchmark.engine: vllm. Its args map directly to vllm bench serve flags; the vllm-opt CLI supplies the model, server address, and JSON output path:

benchmark:
  engine: vllm
  runs:
    - name: throughput
      args:
        dataset-name: random
        random-input-len: 32
        random-output-len: 16
        num-prompts: 100
        request-rate: inf
        max-concurrency: 16

Interactive terminal output uses color and remains concise by default. Set the standard NO_COLOR environment variable to disable color. To stream server and benchmark logs:

vllm-opt --config experiment.yaml --verbose

The persistent equivalent uses GuideLLM's logging level names:

logging:
  level: DEBUG

Supported levels are DEBUG, INFO, WARNING, ERROR, and CRITICAL. Full per-trial log files are always saved and benchmark logs are flushed while the command runs. Request-limited runs show a live request counter; duration-only runs show an elapsed/limit timer. --verbose also mirrors subprocess output to the terminal and overrides the configured level with DEBUG for that invocation.

Retry one or more selected trials into a new immutable linked run:

vllm-opt retry --run runs/EXPERIMENT/RUN_ID \
  --trial trial-0001 --trial trial-0004

The source run is never modified.

Display every stored vLLM and GuideLLM command for a trial without executing anything:

vllm-opt reproduce --run runs/EXPERIMENT/RUN_ID --trial trial-0001

Each completed run also contains a self-contained report.html decision dashboard with the best observed configuration, per-benchmark elapsed time, average/median/P99 latency, baseline comparison, score history, throughput/latency tradeoff, metric definitions, and observed parameter effects.

Random and TPE runs never execute the same resolved configuration twice. If optimization.trials exceeds the unique search space, the vllm-opt CLI warns and runs every unique configuration once.

Multiple independent trials can run on explicitly assigned, non-overlapping GPU sets and ports. A sequential or tensor-parallel server receives port 8000 unless server.port overrides it; local-parallel trials use their configured port range. Sequential execution remains the default. See parallel trials for the YAML and measurement caveats.

Product documents

• First MVP specification
• Future implementation roadmap
• Architecture overview and early sketch
• Editable Draw.io architecture diagram
• Contributor guide
• Release notes

The MVP specification defines the first releasable version and its acceptance criteria. The roadmap describes capabilities that should be designed for now but implemented after the core experiment loop is reliable.

vLLM Optimizer is available under the MIT License.