Pre-release

This release is a pre-release and may not be stable for production use.

vLLM Semantic Router CLI

vllm-sr configures and runs the local vLLM Semantic Router stack. It can also validate or migrate config files, deploy the router Helm chart, inspect virtual models and Recipes, and send test requests.

Full documentation: https://vllm-sr.ai/docs/installation/

Install

pip install vllm-sr
vllm-sr --version

For CLI development:

cd src/vllm-sr
python -m venv .venv
. .venv/bin/activate
pip install -e .

Local serve requires Docker or Podman on Linux, macOS, or WSL2. A native Windows Python environment can run config and catalog commands, but it cannot run the local container stack.

Start a local stack

# Start Router, Envoy, Dashboard, and observability.
vllm-sr serve

# Use Podman.
vllm-sr serve --runtime podman

# Check the stack and open the Dashboard.
vllm-sr status
vllm-sr dashboard

The Dashboard is available at http://localhost:8700. The routed OpenAI-compatible listener uses the first port in config.yaml (8899 in the reference config).

For local serve, listeners[].address controls the host port publication. Use 127.0.0.1 or ::1 for host-only access. Envoy listens on the container bridge interface so both the published port and Dashboard can reach it; this keeps the host loopback restriction, including after Dashboard config saves. Standalone config envoy generation retains the configured listener address.

vllm-sr serve starts the routing stack. It does not start the physical LLM backends referenced by providers.models; those endpoints must already be running and reachable.

Useful lifecycle commands:

vllm-sr logs router
vllm-sr logs envoy
vllm-sr logs dashboard
vllm-sr stop

Add --minimal to run Router and Envoy without Dashboard or observability. Add --readonly to keep Dashboard available without config editing.

Test routing

route preview reports which signals, decision, algorithm, and plugins matched without calling the selected model backend:

vllm-sr route preview --prompt "Explain inflation in plain English."
vllm-sr route preview --prompt "Explain inflation in plain English." --json
vllm-sr route preview \
  --model vllm-sr/mom-v1-blend \
  --prompt "Summarize this architecture plan." \
  --json

Use --messages for an OpenAI-style messages array and --endpoint when the Router management API is not at http://localhost:8080:

vllm-sr route preview \
  --messages '[{"role":"user","content":"Explain inflation."}]' \
  --endpoint http://localhost:8080

request chat sends a real one-shot completion through the routed listener. It uses vllm-sr/auto unless --model is set:

vllm-sr request chat "Hello"
vllm-sr request chat --model my-virtual-model --json "Hello"
vllm-sr request chat --base-url https://gateway.example.com "Hello"

--base-url must point to an OpenAI-compatible routed endpoint, such as an ingress or port-forwarded gateway. It is not the Router management API used by route preview and storage vector-stores.

Choose a configuration

The CLI reads canonical v0.3 YAML with version/listeners/providers/routing/global. Author a file directly, start from a maintained Recipe, or fork a bundled virtual model.

vllm-sr config validate --config config.yaml
vllm-sr serve --config config.yaml

Route policy lives in routing.decisions[]. For example, this decision fragment defines a final static fallback; merge it into a complete config that declares local-model under providers.models and routing.modelCards:

routing:
  decisions:
    - name: local-fallback
      description: Handle requests that did not match an earlier decision.
      priority: 0
      rules:
        operator: AND
        conditions: []
      modelRefs:
        - model: local-model
      algorithm:
        type: static

vllm-sr init was removed in v0.3. For older files or supported external provider configs, use the explicit conversion commands:

vllm-sr config migrate --config old-config.yaml
vllm-sr config import \
  --from openclaw \
  --source openclaw.json \
  --target config.yaml

The current field reference is generated in the configuration guide. Focused examples live under config/fragments/. Use those sources instead of copying plugin or algorithm schemas from this package README.

Keep credentials out of YAML. Reference environment variables and authorize Recipe-specific variables explicitly:

export PROVIDER_API_KEY=...
vllm-sr serve --config recipe.yaml --recipe-env PROVIDER_API_KEY

Connect Models and build Mixture-of-Models

Run vllm-sr serve, then use Dashboard to connect provider Models, choose a built-in or custom Recipe, and assign Models to its decisions. Dashboard keeps provider connection details separate from reusable routing policy and exposes the published names through /v1/models.

For source-controlled deployments, validate and serve one complete user-owned configuration:

vllm-sr config validate --config my-models.yaml
vllm-sr serve --config my-models.yaml

To evaluate concurrently running baseline and candidate deployments from one Dashboard, point EVALUATION_DEPLOYMENTS_DIR at the strict, read-only evaluation-deployments.v1 registry described in the Evaluation Plane guide, then use the same vllm-sr serve command. The CLI mounts that directory into Dashboard only; Router and Envoy do not inherit it. Leaving the variable unset preserves the current single-runtime behavior.

Deploy to Kubernetes

The Kubernetes target installs or upgrades the Helm release:

vllm-sr serve \
  --target k8s \
  --profile dev \
  --namespace semantic-router \
  --config config.yaml

vllm-sr status --target k8s --namespace semantic-router
vllm-sr logs router --target k8s --namespace semantic-router -f
vllm-sr stop --target k8s --namespace semantic-router

Kubernetes requires a complete, non-empty config. The CLI does not merge local Docker defaults or sample routes into it. Credential references are stored in a release-scoped Secret, and literal credentials or credential-bearing URLs are rejected.

--platform amd and --platform nvidia are local-container shortcuts. On Kubernetes, select GPU images, resources, and device plugins through Helm values, a deployment profile, or the operator.

See Kubernetes installation for gateway, profile, and production guidance.

Inspect vector stores

storage vector-stores reads vector stores from the Router management API. It does not create, modify, or delete stores.

vllm-sr storage vector-stores
vllm-sr storage vector-stores --endpoint http://router.example.com:8080

The Router must be running with a vector-store backend enabled. --endpoint points to the management API, not the routed inference listener.

Local ports and state

Default ports in the reference local stack are:

Service	Port	Purpose
Dashboard	`8700`	Configuration, Playground, and embedded observability
Routed inference listener	`8899`	OpenAI-compatible model requests
Router management API	`8080`	Eval, config, replay, and vector-store APIs
Router metrics	`9190`	Prometheus metrics
Jaeger	`16686`	Trace UI
Prometheus	`9090`	Metrics storage and queries

Listener and management ports can be changed in YAML. Local Dashboard data is stored under .vllm-sr/dashboard-data/ and survives stop unless that workspace directory is removed.

To run independent stacks from multiple worktrees, use a distinct name and port offset on every lifecycle command:

export VLLM_SR_STACK_NAME=lane-b
export VLLM_SR_PORT_OFFSET=200
vllm-sr serve
vllm-sr status
vllm-sr stop

Troubleshooting

route preview and storage vector-stores use the Router management API, normally port 8080.
request chat uses the routed inference listener from config.yaml, normally port 8899.
A healthy Router and Envoy do not prove that an external model backend can generate. Use Dashboard Verify or chat to test the backend path.
If a lifecycle command reports that the stack is busy, let the active serve or stop finish and retry.
Set NO_COLOR=1 for plain CLI output. JSON modes keep stdout free of status messages so it can be consumed by scripts.

Run vllm-sr COMMAND --help for command-specific options. For installation, security, configuration, and operations, use the website documentation.

License

Apache 2.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

vllm_sr-0.3.0.dev20260915012650.tar.gz (1.0 MB view details)

Uploaded Sep 15, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

vllm_sr-0.3.0.dev20260915012650-py3-none-any.whl (868.3 kB view details)

Uploaded Sep 15, 2026 Python 3

File details

Details for the file vllm_sr-0.3.0.dev20260915012650.tar.gz.

File metadata

Download URL: vllm_sr-0.3.0.dev20260915012650.tar.gz
Upload date: Sep 15, 2026
Size: 1.0 MB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/7.0.0 CPython/3.12.3

File hashes

Hashes for vllm_sr-0.3.0.dev20260915012650.tar.gz
Algorithm	Hash digest
SHA256	`069e937bc6bc5d831da61e0f166eb6e582399dd66d8832deee16957b06c62f08`
MD5	`13857fedb04e8020fd3a6a2685995118`
BLAKE2b-256	`4e4b70045a77deacfe207b926f289712eb7bb5e46121a804747f7efff3bdeb1c`

See more details on using hashes here.

File details

Details for the file vllm_sr-0.3.0.dev20260915012650-py3-none-any.whl.

File metadata

Download URL: vllm_sr-0.3.0.dev20260915012650-py3-none-any.whl
Upload date: Sep 15, 2026
Size: 868.3 kB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/7.0.0 CPython/3.12.3

File hashes

Hashes for vllm_sr-0.3.0.dev20260915012650-py3-none-any.whl
Algorithm	Hash digest
SHA256	`8eca8f523eff6f27afef595134ce0f67373f6c9bb65a864b3a1cbc75cbb17c36`
MD5	`68c4fef8b1fb05440cba39d3d2ed16ea`
BLAKE2b-256	`4a507e25456ccf1219967bbb65a9350a891a66bb73aa725885daa2f5140e752e`