vllm_bench

Run user-defined vLLM benchmarks against existing service URLs or with services managed through Docker Compose or Modal. Benchmark scripts and the runner execute on your machine; startup scripts execute in containers. The same Python library supplies arguments and service addresses to both. The provider owns service infrastructure; the runner and reports are shared.

Install

Requires Python 3.11 or newer on the host and in service images. Local runs require Docker Compose v2; Modal runs require a Modal account and the optional SDK. Once published on PyPI:

python -m pip install vllm-benchmark
# Or, with Modal support:
python -m pip install 'vllm-benchmark[modal]'

For development, clone this repository and run python -m pip install -e '.[dev]'. No vLLM installation or sibling repository is required on the host unless your own benchmark scripts need it. Project service images supply their own runtime dependencies, such as PyTorch, vLLM, or FAISS.

The local provider mounts the installed vllm_bench package read-only into every service container. Modal copies it into the remote image. Providers override startup to run a wrapper that launches your configured script, samples GPUs, and handles shutdown. Images need no CLI installation or special entrypoint; no packages are installed at service startup. Service scripts can use from vllm_bench import bench_util without an initialization call.

Try the minimal example with an ordinary Python image and no model download:

cd examples/minimal
vllm_bench run_scenarios --benchmark example --scenario baseline

Commands and selection

A project supplies .vllm_bench/configs/base.yaml plus environment configs: local_cpu.yaml, local_cpu_arm.yaml, and modal.yaml. The default environment is local_cpu; choose another with --config. Environment selection is separate from benchmark and scenario selection.

Both up and run_scenarios require --benchmark and --scenario. The plural aliases --benchmarks and --scenarios work too. Each accepts one or more names or '*' on its own. Quote the wildcard so your shell does not expand it into filenames.

# Every scenario for one benchmark:
vllm_bench run_scenarios --config local_cpu --benchmark example --scenarios '*'
# A named scenario across all benchmarks:
vllm_bench run_scenarios --benchmark '*' --scenario baseline
# Every benchmark and its own scenarios:
vllm_bench run_scenarios --benchmark '*' --scenario '*'
# Explicit subsets:
vllm_bench run_scenarios --benchmark example other --scenario baseline distributed
# Start one environment and follow its logs:
vllm_bench up --config local_cpu_arm --benchmark example --scenario baseline
vllm_bench down
vllm_bench report results/local-comparison/run001

A named scenario must exist in every selected benchmark. Unknown names fail before starting containers, stopping an active environment, or saving results. A scenario wildcard uses each benchmark's own scenarios; they need not match. run_scenarios creates fresh services for every selected benchmark/scenario/repeat and then generates reports. The former run command is now named run_scenarios.

up must resolve to exactly one benchmark/scenario pair. It only attaches to an active environment when provider, image, benchmark, scenario, and arguments match. It streams logs; Ctrl+C stops services and saves output. Use up -d to leave it running, then down to stop it. down uses saved state, and report uses its result directory, so neither needs benchmark/scenario selectors or an environment config.

run_scenarios stops an existing environment only after validating the complete selection. Failures retain partial results and continue; Ctrl+C stops the experiment. Cleanup failures preserve recovery state for down and stop further scheduling. Reports can be regenerated offline.

--current-dir /path/to/project works before or after any command and sets the root for config lookup, scripts, snapshots, runtime state, and results. Relative report paths also resolve from this root. --config works before or after up or run_scenarios; it accepts an environment name or YAML path.

Run against existing services

run_only executes exactly one benchmark/scenario once, regardless of runs. It does not start, stop, reconfigure, or restart services, and leaves any active up environment and its state file untouched. It uses the same benchmark executor and report format as run_scenarios. Managed runs use the startup/readiness routine shared with up and the provider shutdown used by down after each trial, including failed trials.

vllm_bench run_only --config local_cpu \
  --benchmark example --scenario baseline \
  --service-url leader=https://example.com:8000

Repeat --service-url NAME=URL for each required service. Names match the config's services keys. Supply every service selected by the scenario and benchmark service dependencies; unknown, duplicate, and missing names fail before execution. URLs must be HTTP(S) base URLs, including any reverse-proxy prefix. Each supplied service group describes one external endpoint; replicas controls managed services only. The client uses the supplied URLs for health checks, metrics, and benchmark requests. internal=True returns the same URLs in this mode, so services that call each other must also be able to reach those addresses.

Only benchmark scripts need to exist locally. An external-only config can omit base_image and service startup_script; up and run_scenarios still require those fields. Set global_args and service_args to match the models already served. Selecting a scenario records its arguments; it does not change the remote server's coordinator or model. The old run command has been renamed to run_scenarios.

Results are saved under results/external-<name>/runNNN/, with one trial, endpoint URLs, the client context, logs, metrics, and reports. External runs do not upload a project snapshot, transfer inputs, or collect server logs/GPU samples. When the client and servers share a filesystem, get_service_path() returns the local path unchanged. For cloud deployments, make inputs available to the servers first; --dataset-path /server/path passes that value as the benchmark's dataset_path argument. Your benchmark adapter decides how to use it.

After health checks pass, up prints a copyable run_only command before the endpoint and log details, including the selected config, project, benchmark, scenario, and service URLs. You can start an environment with up -d, then run that command as often as needed before stopping it with down. Supply dataset paths visible to that existing environment when it cannot access the client's paths.

Configuration

base.yaml defines shared services and scenarios. Benchmarks reference scenario names. For example:

name: comparison
services:
  leader:
    role: leader
    startup_script: scripts/start_leader.py
    health_port: 8000
    metrics_path: /metrics
  worker:
    role: worker
    startup_script: scripts/start_worker.py
    health_port: 8000
    metrics_path: /metrics
scenarios:
  baseline:
    args:
      coordinator: single_worker
    services: [leader]
  distributed:
    args:
      coordinator: round_robin
    services: [leader, worker]
benchmarks:
  example:
    script: scripts/benchmark.py
    script_args:
      dataset_percentage: 0.1
    scenarios: [baseline, distributed]
runs: 1
startup_timeout: 1800
benchmark_timeout: 3600
sample_interval: 1

local_cpu.yaml selects the local image and model:

provider: local
base_image: my-project:cpu
global_args:
  model: Qwen/Qwen2.5-0.5B-Instruct

local_cpu_arm.yaml uses the same structure with an ARM64 CPU image. An environment config automatically merges over its sibling base.yaml. Mappings merge recursively; child lists and scalar values replace base values. extends: other_config can inherit another environment before applying overrides. Saved runs contain the resolved environment, shared scenario definitions, and selected benchmarks with their selected scenario references. There are no separate benchmark YAML files or benchmark_scenarios field.

Scenarios are defined once in the top-level scenarios mapping. Each benchmark selects a nonempty list of unique scenario names. Every scenario declares a nonempty list of known service names. A benchmark may also declare services: [icp] to add dependencies to every selected scenario. The CLI combines both service lists, removes overlaps, and expands each service to its replica count. Scenario and benchmark arguments are arbitrary project data. Names use lowercase letters, digits, underscores, or hyphens, starting with a letter. Script paths are relative to the project root.

Optional settings:

• Service replicas: number of instances; defaults to one.
• Service gpus: all or gpus: ["0"]: local NVIDIA GPU allocation. Omit for CPU.
• Service environment: extra string environment values.
• Service service_args: arguments that override global_args for that service. Startup scripts use bench_util.get_service_args() to read the merged values; clients can use bench_util.get_service_args("cascade") for a named service. Unspecified arguments inherit global values; get_global_args() stays unchanged.
• Service health_path: defaults to /health; omit metrics_path if unavailable.
• skip_volume_mount: true: use project code already inside the image.
• image_workdir: project directory for that mode; defaults to /workspace/project.

Startup scripts bind to 0.0.0.0 on their assigned service port and run in the foreground. Returning stops the service. Images must support the selected hardware; a CUDA-only vLLM build cannot automatically run on CPU.

Modal

Install python -m pip install 'vllm-benchmark[modal]' and authenticate with modal setup. Define .vllm_bench/configs/modal.yaml:

provider: modal
base_image: your-registry/benchmark:cuda
global_args:
  model: Qwen/Qwen2.5-0.5B-Instruct
modal:
  app: vllm-bench
  gpu: T4:1
  cpu: 8
  memory: 16384
  timeout: 14400
  cache_volume: vllm-bench-model-cache

vllm_bench run_scenarios --config modal --benchmark example --scenario '*'
vllm_bench up --config modal --benchmark example --scenario baseline
vllm_bench down

The registry image contains project dependencies; Modal copies the installed CLI library into it without installing packages at startup. Each trial uses one sandbox in the named Modal App. Services share its CPU/GPU allocation and use separate ports. Service gpus settings apply to local Compose; modal.gpu sets the remote allocation. Memory is in MiB, and timeout is the sandbox lifetime in seconds. The model-cache volume mounts at /mnt/vllm-bench-cache; the provider points HF_HOME, VLLM_CACHE_ROOT, and XDG_CACHE_HOME there. This keeps the mount separate from cache files already baked into the service image. Pin the registry image digest for reproducible runs: Modal can cache mutable tags after the registry image changes.

Project snapshots and client-generated dataset files are uploaded. Benchmark scripts, metrics collection, and reports execute locally using HTTPS tunnels. Use bench_util for addresses, assigned ports, and file paths. Shutdown downloads service output and logs. If the sandbox exits early, available combined logs are saved as modal.log; service files may no longer be available.

Script library

from vllm_bench import bench_util

model = bench_util.get_global_arg("model")
options = bench_util.get_service_args()
coordinator = bench_util.get_scenario_arg("coordinator")
fraction = bench_util.get_benchmark_arg("dataset_percentage", 1.0)
repeat = bench_util.current_run
benchmark = bench_util.current_benchmark
output = bench_util.get_output_dir()

Other helpers:

• get_project_dir() returns the project root in the calling environment.
• get_service() returns the calling startup's name, role, replica, and URL; it returns None in client benchmark scripts.
• get_leader_url() and get_worker_urls() choose the correct addresses for the calling environment. Disabled workers yield an empty list.
• get_service_url("icp") returns a named service. For a replicated group, use its individual name such as worker-1.
• get_service_url("icp", internal=True) returns the container address when constructing a benchmark payload that the leader will execute.
• get_service_path(local_path) translates project/trial paths in a client benchmark to paths readable inside the containers.
• get_gpu_count(), is_gpu_available(), and is_cpu() inspect the calling environment using nvidia-smi without importing PyTorch.

Benchmark scripts write response.json in get_output_dir(). Arbitrary JSON is preserved. When present, a numeric latency_sec and a results list provide additional summary fields; process runtime is measured independently.

Files and infrastructure

The CLI library is mounted read-only at /opt/vllm-bench-lib/vllm_bench, independently of the project mount and even with skip_volume_mount: true. Keep the host installation available while containers run. Compose uses python3 -m vllm_bench.service as its startup override and prepends the library location to the configured service PYTHONPATH.

The project is mounted read-only at /workspace/project. Each trial directory is mounted writable at /run/vllm-bench/trial; get_output_dir() selects the client trial directory or a replica's service subdirectory. Docker logs and GPU samples are saved under services/<group>-<replica>/.

The generated Compose file, resolved per-service contexts, and provider IDs are saved under the trial's .provider/. The active state is recorded under the project's .vllm_bench/state.json. Host ports bind to 127.0.0.1 and container addresses use Compose DNS. A named model-cache volume persists after down.

By default, each run also saves a ZIP of nonignored project files. Results, environments, native binaries, caches, .env files, and generated wheels are excluded. The live local project mount reflects subsequent edits; keep files unchanged during a run when comparing its saved snapshot.

See RESULT_FORMAT.md for the saved contract and reports.

Development checks

python -m pip install -e '.[dev]'
ruff check .
python -m unittest discover -s tests
python tests/compose_smoke.py --image python:3.12-slim

The opt-in Compose check launches HTTP stand-ins with Prometheus metrics. It checks lifecycle, endpoints, mounting, scenarios, collection, and reports without loading any model or allocating cloud resources.

Publishing and license

See PUBLISHING.md for GitHub setup, package checks, and PyPI Trusted Publishing. The project uses the MIT license.