This release is a pre-release and may not be stable for production use.
vLLM Semantic Router
Intelligent Router for Mixture-of-Models (MoM).
GitHub: https://github.com/vllm-project/semantic-router
Quick Start
Installation
# Install from PyPI
pip install vllm-sr
# Or install from source (development)
cd src/vllm-sr
pip install -e .
Usage
# Initialize vLLM Semantic Router Configuration
vllm-sr init
# Start the router (includes dashboard)
# Provide your HF_TOKEN to run the evaluation tests; this is required for downloading the necessary datasets
HF_TOKEN=hf_xxx vllm-sr serve
# Open dashboard in browser
vllm-sr dashboard
# View logs
vllm-sr logs router
vllm-sr logs envoy
vllm-sr logs dashboard
# Check status
vllm-sr status
# Stop
vllm-sr stop
Features
- Router: Intelligent request routing based on intent classification
- Envoy Proxy: High-performance proxy with ext_proc integration
- Dashboard: Web UI for monitoring and testing (http://localhost:8700)
- Metrics: Prometheus metrics endpoint (http://localhost:9190/metrics)
Endpoints
After running vllm-sr serve, the following endpoints are available:
| Endpoint | Port | Description |
|---|---|---|
| Dashboard | 8700 | Web UI for monitoring and Playground |
| API | 8888* | Chat completions API (configurable in config.yaml) |
| Metrics | 9190 | Prometheus metrics |
| gRPC | 50051 | Router gRPC (internal) |
| Jaeger UI | 16686 | Distributed tracing UI |
| Grafana (embedded) | 8700 | Dashboards at /embedded/grafana |
| Prometheus UI | 9090 | Metrics storage and querying |
*Default port, configurable via listeners in config.yaml
Observability
vllm-sr serve automatically starts the observability stack:
- Jaeger: Distributed tracing embedded at http://localhost:8700/embedded/jaeger (also available directly at http://localhost:16686)
- Grafana: Pre-configured dashboards embedded at http://localhost:8700/embedded/grafana
- Prometheus: Metrics collection at http://localhost:9090
Note: Grafana is optimized for embedded access through the dashboard. For the best experience, use http://localhost:8700/embedded/grafana where anonymous authentication is pre-configured.
Tracing is enabled by default. Traces are visible in Jaeger under the vllm-sr service name.
Configuration
Plugin Configuration
The CLI supports configuring plugins in your routing decisions. Plugins are per-decision behaviors that customize request handling (security, caching, customization, debugging).
Supported Plugin Types:
semantic-cache- Cache similar requests for performancejailbreak- Detect and block adversarial promptspii- Detect and enforce PII policiessystem_prompt- Inject custom system promptsheader_mutation- Add/modify HTTP headershallucination- Detect hallucinations in responsesrouter_replay- Record routing decisions for debugging
Plugin Examples:
- semantic-cache - Cache similar requests:
plugins:
- type: "semantic-cache"
configuration:
enabled: true
similarity_threshold: 0.92 # 0.0-1.0, higher = more strict
ttl_seconds: 3600 # Optional: cache TTL in seconds
- jailbreak - Block adversarial prompts:
plugins:
- type: "jailbreak"
configuration:
enabled: true
threshold: 0.8 # Optional: detection sensitivity 0.0-1.0
- pii - Enforce PII policies:
plugins:
- type: "pii"
configuration:
enabled: true
threshold: 0.7 # Optional: detection sensitivity 0.0-1.0
pii_types_allowed: ["EMAIL_ADDRESS"] # Optional: list of allowed PII types
- system_prompt - Inject custom instructions:
plugins:
- type: "system_prompt"
configuration:
enabled: true
system_prompt: "You are a helpful assistant."
mode: "replace" # "replace" (default) or "insert" (prepend)
- header_mutation - Modify HTTP headers:
plugins:
- type: "header_mutation"
configuration:
add:
- name: "X-Custom-Header"
value: "custom-value"
update:
- name: "User-Agent"
value: "SemanticRouter/1.0"
delete:
- "X-Old-Header"
- hallucination - Detect hallucinations:
plugins:
- type: "hallucination"
configuration:
enabled: true
use_nli: false # Optional: use NLI for detailed analysis
hallucination_action: "header" # "header", "body", or "none"
- router_replay - Record decisions for debugging:
plugins:
- type: "router_replay"
configuration:
enabled: true
max_records: 200 # Optional: max records in memory (default: 200)
capture_request_body: false # Optional: capture request payloads (default: false)
capture_response_body: false # Optional: capture response payloads (default: false)
max_body_bytes: 4096 # Optional: max bytes to capture (default: 4096)
Validation Rules:
- Plugin Type: Must be one of:
semantic-cache,jailbreak,pii,system_prompt,header_mutation,hallucination,router_replay - enabled: Must be a boolean (required for most plugins)
- threshold/similarity_threshold: Must be a float between 0.0 and 1.0
- max_records/max_body_bytes: Must be a positive integer
- ttl_seconds: Must be a non-negative integer
- pii_types_allowed: Must be a list of strings (if provided)
- system_prompt: Must be a string (if provided)
- mode: Must be "replace" or "insert" (if provided)
CLI Commands:
# Initialize config with plugin examples
vllm-sr init
# Validate configuration (including plugins)
vllm-sr validate config.yaml
# Generate router config with plugins
vllm-sr config router --config config.yaml
File Descriptor Limits
The CLI automatically sets file descriptor limits to 65,536 for Envoy proxy. To customize:
export VLLM_SR_NOFILE_LIMIT=100000 # Optional (min: 8192)
vllm-sr serve
License
Apache 2.0
1 maintainer
Metadata
1 maintainer
Release files for vllm-sr 0.1.0b2.dev20260303140253
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| vllm_sr-0.1.0b2.dev20260303140253.tar.gz | 76.4 kB | Without provenance | Details for vllm_sr-0.1.0b2.dev20260303140253.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| vllm_sr-0.1.0b2.dev20260303140253-py3-none-any.whl | Python 3 | none | any | Details for vllm_sr-0.1.0b2.dev20260303140253-py3-none-any.whl |
Total release size: 155.0 kB
Release files / vllm_sr-0.1.0b2.dev20260303140253.tar.gz
| Download URL | vllm_sr-0.1.0b2.dev20260303140253.tar.gz |
|---|---|
| Size | 76.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5c719dfd4e6864772782188d7b9a43ab5e4babe09a59527df789da3a6288933a
|
|
BLAKE2b-256 checksum How to use checksums |
a149205ffabff3dec97e2db7f0e1a77441426afe7f3be5cc83eb4eb66b9d4e10
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Release files / vllm_sr-0.1.0b2.dev20260303140253-py3-none-any.whl
| Download URL | vllm_sr-0.1.0b2.dev20260303140253-py3-none-any.whl |
|---|---|
| Size | 78.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b70bf4a2c3e792cc1d7a99ef3f624334072ab293d968103d32a46e7402808c26
|
|
BLAKE2b-256 checksum How to use checksums |
dde5274d15cfcf258e466450becdf786cb6158ac94f65fb2796d9d5e6548d0f6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Security for vllm-sr 0.1.0b2.dev20260303140253
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
