This release is a pre-release and may not be stable for production use.
vLLM Semantic Router
Intelligent Router for Mixture-of-Models (MoM).
GitHub: https://github.com/vllm-project/semantic-router
Quick Start
Installation
# Install from PyPI
pip install vllm-sr
# Or install from source (development)
cd src/vllm-sr
pip install -e .
Usage
# Start the router (includes dashboard and first-run setup)
HF_TOKEN=hf_xxx vllm-sr serve
# Open the dashboard
# http://localhost:8700
# Optional: open the dashboard in your browser
vllm-sr dashboard
# View logs
vllm-sr logs router
vllm-sr logs envoy
vllm-sr logs dashboard
# Check status
vllm-sr status
# Stop
vllm-sr stop
If you start in an empty directory, vllm-sr serve bootstraps a minimal workspace and opens the dashboard in setup mode. Configure your first model there, then activate routing.
Advanced YAML-first setup
# Generate an advanced sample config if you prefer editing YAML directly
vllm-sr init
# Validate the sample before serving
vllm-sr validate config.yaml
vllm-sr init is optional. It generates a lean advanced sample plus .vllm-sr/router-defaults.yaml for users who want to hand-author routing config. router-defaults.yaml contains advanced runtime defaults; it is not required for first-run dashboard setup.
Features
- Router: Intelligent request routing based on intent classification
- Envoy Proxy: High-performance proxy with ext_proc integration
- Dashboard: Web UI for monitoring and testing (http://localhost:8700)
- Metrics: Prometheus metrics endpoint (http://localhost:9190/metrics)
Endpoints
After running vllm-sr serve, the following endpoints are available:
| Endpoint | Port | Description |
|---|---|---|
| Dashboard | 8700 | Web UI for monitoring and Playground |
| API | 8888* | Chat completions API (configurable in config.yaml) |
| Metrics | 9190 | Prometheus metrics |
| gRPC | 50051 | Router gRPC (internal) |
| Jaeger UI | 16686 | Distributed tracing UI |
| Grafana (embedded) | 8700 | Dashboards at /embedded/grafana |
| Prometheus UI | 9090 | Metrics storage and querying |
*Default port, configurable via listeners in config.yaml
Observability
vllm-sr serve automatically starts the observability stack:
- Jaeger: Distributed tracing embedded at http://localhost:8700/embedded/jaeger (also available directly at http://localhost:16686)
- Grafana: Pre-configured dashboards embedded at http://localhost:8700/embedded/grafana
- Prometheus: Metrics collection at http://localhost:9090
Note: Grafana is optimized for embedded access through the dashboard. For the best experience, use http://localhost:8700/embedded/grafana where anonymous authentication is pre-configured.
Tracing is enabled by default. Traces are visible in Jaeger under the vllm-sr service name.
Configuration
Plugin Configuration
The CLI supports configuring plugins in your routing decisions. Plugins are per-decision behaviors that customize request handling (security, caching, customization, debugging).
Supported Plugin Types:
semantic-cache- Cache similar requests for performancejailbreak- Detect and block adversarial promptspii- Detect and enforce PII policiessystem_prompt- Inject custom system promptsheader_mutation- Add/modify HTTP headershallucination- Detect hallucinations in responsesrouter_replay- Record routing decisions for debugging
Plugin Examples:
- semantic-cache - Cache similar requests:
plugins:
- type: "semantic-cache"
configuration:
enabled: true
similarity_threshold: 0.92 # 0.0-1.0, higher = more strict
ttl_seconds: 3600 # Optional: cache TTL in seconds
- jailbreak - Block adversarial prompts:
plugins:
- type: "jailbreak"
configuration:
enabled: true
threshold: 0.8 # Optional: detection sensitivity 0.0-1.0
- pii - Enforce PII policies:
plugins:
- type: "pii"
configuration:
enabled: true
threshold: 0.7 # Optional: detection sensitivity 0.0-1.0
pii_types_allowed: ["EMAIL_ADDRESS"] # Optional: list of allowed PII types
- system_prompt - Inject custom instructions:
plugins:
- type: "system_prompt"
configuration:
enabled: true
system_prompt: "You are a helpful assistant."
mode: "replace" # "replace" (default) or "insert" (prepend)
- header_mutation - Modify HTTP headers:
plugins:
- type: "header_mutation"
configuration:
add:
- name: "X-Custom-Header"
value: "custom-value"
update:
- name: "User-Agent"
value: "SemanticRouter/1.0"
delete:
- "X-Old-Header"
- hallucination - Detect hallucinations:
plugins:
- type: "hallucination"
configuration:
enabled: true
use_nli: false # Optional: use NLI for detailed analysis
hallucination_action: "header" # "header", "body", or "none"
- router_replay - Record decisions for debugging:
plugins:
- type: "router_replay"
configuration:
enabled: true
max_records: 200 # Optional: max records in memory (default: 200)
capture_request_body: false # Optional: capture request payloads (default: false)
capture_response_body: false # Optional: capture response payloads (default: false)
max_body_bytes: 4096 # Optional: max bytes to capture (default: 4096)
Validation Rules:
- Plugin Type: Must be one of:
semantic-cache,jailbreak,pii,system_prompt,header_mutation,hallucination,router_replay - enabled: Must be a boolean (required for most plugins)
- threshold/similarity_threshold: Must be a float between 0.0 and 1.0
- max_records/max_body_bytes: Must be a positive integer
- ttl_seconds: Must be a non-negative integer
- pii_types_allowed: Must be a list of strings (if provided)
- system_prompt: Must be a string (if provided)
- mode: Must be "replace" or "insert" (if provided)
CLI Commands:
# Generate an advanced YAML sample if you want to edit config directly
vllm-sr init
# Validate configuration (including plugins)
vllm-sr validate config.yaml
# Generate router config with plugins
vllm-sr config router --config config.yaml
File Descriptor Limits
The CLI automatically sets file descriptor limits to 65,536 for Envoy proxy. To customize:
export VLLM_SR_NOFILE_LIMIT=100000 # Optional (min: 8192)
vllm-sr serve
License
Apache 2.0
1 maintainer
Metadata
1 maintainer
Release files for vllm-sr 0.1.0b2.dev20260310064703
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| vllm_sr-0.1.0b2.dev20260310064703.tar.gz | 79.5 kB | Without provenance | Details for vllm_sr-0.1.0b2.dev20260310064703.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| vllm_sr-0.1.0b2.dev20260310064703-py3-none-any.whl | Python 3 | none | any | Details for vllm_sr-0.1.0b2.dev20260310064703-py3-none-any.whl |
Total release size: 160.6 kB
Release files / vllm_sr-0.1.0b2.dev20260310064703.tar.gz
| Download URL | vllm_sr-0.1.0b2.dev20260310064703.tar.gz |
|---|---|
| Size | 79.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2e71e4916369b67b406cc0b8468caf141a7c5713c3cc01570070a3d28b93ac50
|
|
BLAKE2b-256 checksum How to use checksums |
831ce52874e45be4542f034b16b77c5f17979efe611b069bc401920e0722fe57
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Release files / vllm_sr-0.1.0b2.dev20260310064703-py3-none-any.whl
| Download URL | vllm_sr-0.1.0b2.dev20260310064703-py3-none-any.whl |
|---|---|
| Size | 81.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
57b01081dbcd5cf1a3a8e5fe24e8f45a06d92a2b87a6da8034ec62fe53bab20f
|
|
BLAKE2b-256 checksum How to use checksums |
431ee31d620970204ef47ebbc147d010e26331042155a6ff92dba43c5c9b643f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Security for vllm-sr 0.1.0b2.dev20260310064703
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
