sgl-router
Slim, KV-aware router for SGLang workers.
A router serves one model and spreads its requests across that model's SGLang engines. It picks an engine per request from live engine load, global KV-cache prefix matches, or session affinity, supports prefill/decode (P/D) disaggregation, bucketed deployments, and discovers engines from a static URL list or Kubernetes EndpointSlices.
- Architecture
- Quick start
- Key interfaces
- Routing policies
- Deployment examples
- Add a new policy
- Other features
Related docs: ARCHITECTURE.md (routing design and bucket configuration), REFERENCE.md (detailed feature behavior), monitoring/README.md (metrics and Grafana), BENCHMARKS.md, sgl-kv-indexer/README.md (external KV indexer), EXAMPLES.md (GLM-5.3-Flash and DeepSeek-V4.1-Flash, end to end).
Architecture
A deployment is a router in front of buckets of SGLang engines. A bucket is a logical group defined in the router configuration. It holds either one plain engine group, where each engine does prefill and decode, or separate prefill and decode (P/D) groups. Buckets can be length tiers (for example short and long context); without a bucket file there is one plain bucket and one P/D bucket containing every discovered engine.
Engines publish load reports and KV events. The router uses them, plus its own in-flight counts and session bindings, to pick engines. An optional external KV indexer can replace the router's local radix tree as the source of prefix matches. In P/D, the router sends the request to both the chosen prefill and decode engines, and prefill transfers KV directly to decode.
For each request, the router renders and tokenizes it, tries the buckets that fit
its length (ordered by SLO preference, capacity, then rank), lets the group's
policy pick an engine that passes admission, and dispatches it, retrying on
another engine if it fails before any response. Output handling (OpenAI
responses, tool and reasoning parsing) also happens in the router.
Scalability. Routers are stateless: nothing is persisted or shared between
replicas. Each one rebuilds its routing state (engine load, KV prefix tree,
in-flight counts) from the engines' own reports, so you scale a large deployment
by running more router replicas behind a load balancer. A new replica can copy
the KV tree from a warm sibling at startup
(peer bootstrap). Session bindings stay in each
replica's memory, so with session_aware and several replicas, route a session's
requests to the same replica (for example, a load balancer sticky on the session
header).
Quick start
Pull the container image (multi-arch, amd64 and arm64; see Releases and versions):
docker pull lmsysorg/sgl-router:latest # latest release
docker pull lmsysorg/sgl-router:nightly # built from main every night
docker run --rm -p 30000:30000 lmsysorg/sgl-router:latest --help
Or build from source:
cd sgl-router
cargo build --release
Run in front of two engines with cache-aware routing:
./target/release/sgl-router \
--host 0.0.0.0 --port 30000 \
--model-id qwen3 \
--worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000 \
--chat-routing reorg --policy cache_aware
Send a request:
curl http://localhost:30000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model": "qwen3", "messages": [{"role": "user", "content": "Hello"}]}'
--model-id is required. Any model name is served as --model-id, and the
engine receives it as sent. --model-alias (repeatable) forwards another model
name as --model-id, for example a gateway that sends the engine's
--model-path. Aliases are forwarded as --model-id, preserving an optional
:adapter LoRA suffix; names containing colons, such as s3://bucket/model,
are matched in full. /v1/models lists only --model-id. --tokenizer-path takes a local tokenizer.json or a
HuggingFace repo id; when omitted, the tokenizer for --model-id is downloaded
from HuggingFace (honoring HF_TOKEN / HF_HOME). sgl-router --help lists
every flag.
Key interfaces
HTTP endpoints
| Endpoint | Purpose |
|---|---|
POST /v1/chat/completions, POST /v1/completions |
OpenAI API, served over /generate |
POST/PUT /generate |
SGLang native generate, buffered or SSE |
POST /v1/messages, /v1/messages/count_tokens |
Anthropic Messages |
POST /v1/embeddings, /v1/classify, /v1/rerank |
Embeddings, classify and rerank |
POST /v1/tokenize, /v1/detokenize |
Tokenize with the router's tokenizer |
GET /v1/models |
The served model |
POST /flush_cache |
Fan out a cache flush to every worker, with the caller's timeout |
GET /server_info |
Answered by one healthy worker (a decode worker under P/D, which reports decode's state); /get_server_info is an alias |
GET /model_info, /health_generate |
Answered by one healthy worker (a prefill worker under P/D; /health_generate also runs on one of its decode peers and returns the decode's answer when both pass); /get_model_info is an alias |
GET /healthz, /readyz |
Liveness and readiness; /health, /liveness, and /readiness are aliases |
GET /metrics |
Prometheus metrics, see monitoring/README.md |
Endpoints for features the router does not provide yet (the Responses and
Conversations APIs, /v1/loads) answer 501 with error code
unsupported_endpoint.
Both edges speak cleartext HTTP/2 where the peer does; see
operations. With --grpc-port, the router also serves the
engine's native gRPC API.
Key flags
| Area | Flags |
|---|---|
| Model | --model-id, --model-alias, --tokenizer-path, --no-tokenizer |
| Engines | --worker-urls, or --service-discovery with --selector (plain) or --prefill-selector + --decode-selector (P/D) |
| Routing | --chat-routing reorg, --policy, --bucket-config, --affinity-*, --session-id-header |
| Admission | --max-inflight-requests, --max-waiting-requests, --max-pending-prefill-tokens, --max-running-usage, --max-kv-usage |
| Reliability | --retry-max-attempts, --request-timeout-secs, --stale-request-timeout-secs |
| KV signals | --cache-prefix-provider, --kv-indexer-endpoint, --kv-peer-selector |
Omitting --chat-routing keeps the legacy routing path (default
--policy round_robin). This README and ARCHITECTURE.md
describe --chat-routing reorg.
Routing policies
Each engine group runs one policy, chosen by --policy (default power_of_two
under reorg) or per group in a bucket file:
power_of_two: samples two engines and takes the less loaded. Prefill and plain engines compare estimated prefill queue time, waiting uncached tokens, waiting, running, then in-flight requests; decode engines compare waiting and running requests, then KV usage. Decode groups always default to it.cache_aware: prefers the engine holding the longest cached prefix of the prompt (at least--cache-affinity-min-matched-tokens, default 1024, and optionally--cache-affinity-min-match-ratioof the prompt). Plain and prefill groups only. Needs a tokenizer.session_aware: keeps requests with the same session header (--session-id-header, defaultx-session-id) on the engine they are bound to. Any stage, including decode via a bucket file.
Both affinity policies fall back to power_of_two among the remaining engines
when there is no preferred engine or it fails admission. With
--affinity-mode prefer (default) the preferred engine wins whenever it is
admitted. With --affinity-mode balanced the router also samples an
alternative and switches only when the preferred engine's load exceeds both
alternative * --affinity-load-factor (default 2) and
alternative + --affinity-load-gap, measured by --affinity-balanced-by
(prefill-tokens by default, or running-requests).
Admission flags set per-engine caps; an engine at a cap is skipped:
| Flag | Rejects an engine at |
|---|---|
--max-inflight-requests N |
N requests this router has in flight to it (--max-in-flight is an alias) |
--max-waiting-requests N |
N engine-reported waiting requests |
--max-pending-prefill-tokens N |
N engine-reported waiting uncached tokens |
--max-running-usage F |
share F of its reported running-request capacity |
--max-kv-usage F |
share F of its reported KV capacity |
Engine-reported caps lag by one load report, so a burst can pass them;
--max-inflight-requests counts dispatches as they happen.
For the full selection rules see ARCHITECTURE.md: load-aware routing, cache- and session-aware routing, and configuration examples.
Deployment examples
For complete model deployments (engine flags, router flags, prefer vs balanced affinity), see EXAMPLES.md.
Static workers, plain engines:
sgl-router --model-id qwen3 --chat-routing reorg --policy cache_aware \
--worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000
Kubernetes discovery, plain engines (omit the namespace to watch all namespaces, which needs cluster-wide RBAC):
sgl-router --model-id qwen3 --chat-routing reorg --policy cache_aware \
--service-discovery --service-discovery-namespace prod \
--selector app=engines-qwen3
Kubernetes discovery, P/D engines with session affinity:
sgl-router --model-id qwen3 --chat-routing reorg --policy session_aware \
--service-discovery --service-discovery-namespace prod \
--prefill-selector app=engines-qwen3,role=prefill \
--decode-selector app=engines-qwen3,role=decode \
--max-kv-usage 0.9
Length-tiered buckets from a file (schema and a full example in ARCHITECTURE.md):
sgl-router --model-id qwen3 --chat-routing reorg --policy cache_aware \
--service-discovery --service-discovery-namespace prod \
--prefill-selector app=engines-qwen3,role=prefill \
--decode-selector app=engines-qwen3,role=decode \
--bucket-config buckets.json
Add a new policy
Reorg policies live in src/policies_reorg/. Contributions
are welcome; read ARCHITECTURE.md first for how buckets,
admission and fallback fit together.
-
Implement the
Policytrait insrc/policies_reorg/<name>.rs. The trait is insrc/policies_reorg/mod.rs:pub trait Policy: Send + Sync + Debug { fn pick<'a>(&'a self, engines: &'a [Arc<Worker>], request: &'a PickRequest<'a>) -> BoxFuture<'a, Result<Pick, PickError>>; fn supports(&self, _stage: Stage) -> bool { true } fn needs_request_tokens(&self) -> bool { false } fn fallback(&self) -> Option<&dyn Policy> { None } }
pickmust return one of the suppliedenginesand check admission on it (self.admission.check(&engine, &metrics)), returningPickError::AdmissionRejectedon rejection.- The policy holds its own shared-state handles (load table, KV source,
session store), passed in at construction, and reads them inside
pick. - Override
supportsto restrict stages (plain, prefill, decode),needs_request_tokensif it readsrequest.token_idsorrequest.prefix, andfallbackif it delegates to another policy. power_of_two.rsis a complete ~80-line example;session_aware.rsshows affinity with a fallback.
-
Declare the module in
src/policies_reorg/mod.rs. -
Add a
PolicyKindvariant insrc/config/types.rswith its--policyname.PolicyKindis shared with the legacy path, so add an arm returning an error tobuild_kindinsrc/policies/factory.rs(the compiler lists every exhaustive match to update). -
Wire it into
src/policies_reorg/factory.rs: add it to the allowlist invalidate()and construct it in thematch kindofGroups::build. -
Test it in
tests/component/policies_reorg_<name>.rs, registered intests/component/main.rs, followingpolicies_reorg_power_of_two.rs. For end-to-end routing, add a case undertests/proxy/chat_routing/reorg/. -
Document it: a subsection in ARCHITECTURE.md next to section 1.4, and a bullet in Routing policies above.
Other features
Each item links to its full description in REFERENCE.md.
Native /generate
Same body and response as the engine's /generate, buffered or SSE. With a
tokenizer loaded, text is forwarded as input_ids so routing sees the
engine's exact tokens (--disable-input-ids-forwarding turns this off). See
native /generate.
OpenAI over /generate
For /v1/chat/completions and /v1/completions, the router runs SGLang's
OpenAI layer itself: it sends the request to /generate and builds the OpenAI
response. Requests it cannot reproduce exactly (for example media or strict
tools) go to the engine's own route unchanged. See
OpenAI over /generate for the exact
conditions, and chat rendering plus the
per-model sections (DeepSeek V4 and V4.1,
GLM-5.3, Qwen3.8,
Kimi-K3, MiniMax-M3,
StepFun) for which chats forward input_ids.
Anthropic Messages
/v1/messages is converted to a chat completion and back.
/v1/messages/count_tokens is answered by the router from the chat template.
See Anthropic Messages.
Embeddings, classify and rerank
Engine request and response bodies, routed on load. Not available on P/D fleets. See embeddings, classify and rerank.
Native gRPC
--grpc-port serves the engine's sglang.runtime.v1.SglangService. Each RPC is
routed like its HTTP route and forwarded to the same RPC on the engine's own
native gRPC port, so engines must run with --grpc-port too. See
native gRPC.
Bucket file
--bucket-config defines buckets whose groups set worker_services
(namespace/name) or worker_ids, plus policy, admission and affinity;
unset fields inherit the CLI flags. See
ARCHITECTURE.md and
bucket file membership.
KV cache signals
- External KV indexer:
--cache-prefix-provider indexer --kv-indexer-endpoint <url>; see sgl-kv-indexer and external KV indexer. - KV-event gap replay: add
"replay_endpoint"to the engine's--kv-events-configso the router can re-fetch dropped KV events; see gap replay. - Pending prefixes:
--cache-pending-prefix-ttl-mskeeps bursts that share a cold prefix on one worker until its KV events arrive; see pending prefixes. - Peer bootstrap (Kubernetes):
--kv-peer-selector kubernetes.io/service-name=<router-service>makes a new replica copy the KV tree from a sibling before/readyzpasses; see peer bootstrap. - LoRA cache identity: with workers that publish
BlockStored.lora_name, prefix matches are kept per adapter; see LoRA cache identity.
Operations
- No tokenizer:
--no-tokenizerfor load-only policies (power_of_two,session_aware); see optional tokenizer. - Retries:
--retry-max-attempts Nresends a request that failed before any response to another worker; see retries. - Circuit breaker: a worker leaves routing after
--cb-thresholdconsecutive failures;--disable-circuit-breakerturns this off; see circuit breaker. - DP-rank routing:
--dp-awarealso picks the DP rank inside multi-rank engines, and--dp-rank-policy round_robincycles ranks that no routing key or cached prefix decides; in PD mode,--decode-dispatch-limit Npicks the decode rank shortly before the KV arrives; see DP-rank routing. - PD version groups:
--pd-version-group-label <key>pairs prefill and decode only within the same engine version; see PD version groups. - Request body size: inference routes answer
413past 32 MiB;SGLANG_ROUTER_MAX_BODY_BYTES=<bytes>raises the cap for large multimodal requests, at the cost of more router memory per in-flight request; see request body size. - Engine API key:
--worker-api-keyfor engines launched with--api-key; see engines with--api-key. - Sampling contract:
--override-sampling-params '<json>'pins sampling parameters fleet-wide;--sampling-param-conflict reject|allowdecides what happens to requests that differ. See fleet-wide sampling contract for parameter domains and null handling. - Chat rendering: use the same model files and chat-template settings on
router and workers, or set
--disable-input-ids-forwarding.--tokenizer-backend fastand--tokenizer-l1-cache-mb Nspeed up tokenization. - HTTP/2: no configuration. h2c is used inbound when the client speaks it,
and outbound to workers that report
enable_http2; see HTTP/2. - Upgrading: the
cache_aware_zmqpolicy and the legacy cache and balance threshold flags were removed; see upgrading.
FIPS builds
For deployments that require FIPS 140-validated cryptography, build with the
opt-in fips cargo feature. The router then uses the aws-lc-rs FIPS module as
its TLS provider (outbound connections to workers and the Kubernetes API) and
exits at startup unless FIPS mode is active. Default builds are unchanged.
cargo build --release --features fips
# or the container image
docker build -f docker/sgl-router.Dockerfile --build-arg CARGO_FEATURES=fips .
- The FIPS module builds from source and needs cmake, Go and clang, on a glibc
(not musl) target. The Dockerfile installs them when
fipsis requested;RUNTIME_BASE_IMAGEselects the runtime base, e.g. a FIPS-mode OS image. - FIPS builds do not download tokenizers from Hugging Face Hub (its HTTP
client cannot use the FIPS module): pass a local
--tokenizer-path, or--no-tokenizerfor load-only policies. - On macOS, the FIPS module links as a shared library; set
DYLD_LIBRARY_PATHto its build output directory to run a native build.
Releases and versions
sgl-router is versioned on its own semver track, independent of SGLang
releases. Each release ships as:
- a container image,
lmsysorg/sgl-routertaggedv<version>andlatest(the newest release); - a PyPI package,
pip install sgl-router, whose wheel (linux x86_64 and aarch64) puts thesgl-routerbinary onPATH.
lmsysorg/sgl-router:nightly is built from main every night, and the SGLang
images build the router from their own source. The crate itself is not
published to crates.io.
sgl-router --version prints the version a binary was built from.
Releases are made manually:
- Bump
versioninCargo.tomland runcargo checkonce soCargo.lockrecords it (CI builds with--locked): the minor version for new routing behavior, flags, or protocol changes; the patch version for fixes only. Open a PR with the bump and the release notes. - Once the bump is on main, a maintainer tags that commit
router-v<version>and pushes the tag. That tag startsrelease-sgl-router.yml, which refuses a tag that differs from theCargo.tomlversion, builds the image and the wheels for amd64 and arm64, pusheslmsysorg/sgl-router:v<version>and:latest, and uploads the wheels to PyPI.
Do not tag a router release v<version>: SGLang's own PyPI and Docker release
workflows run on every v* tag.
License
Apache-2.0.
Metadata
Release files for sgl-router 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distributions (wheels)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| sgl_router-0.1.0-py3-none-manylinux_2_28_x86_64.whl | Python 3 | none | Linux glibc 2.28+ x86-64 | Details for sgl_router-0.1.0-py3-none-manylinux_2_28_x86_64.whl |
| sgl_router-0.1.0-py3-none-manylinux_2_28_aarch64.whl | Python 3 | none | Linux glibc 2.28+ ARM64 | Details for sgl_router-0.1.0-py3-none-manylinux_2_28_aarch64.whl |
Total release size: 32.2 MB
Release files / sgl_router-0.1.0-py3-none-manylinux_2_28_x86_64.whl
| Download URL | sgl_router-0.1.0-py3-none-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 16.2 MB |
| Tags | Linux glibc 2.28+ x86-64 Python 3 |
|
SHA-256 checksum How to use checksums |
f22d3c2de80afb11080bfd0b3be523088e2f0a1f754b6f9fc6e6267d5c5e6107
|
|
BLAKE2b-256 checksum How to use checksums |
acf56a609d50cbc365685e4c05eb6dbec67b1ca1f5a60da28d57bece7fe3f823
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.
Transparency logRelease files / sgl_router-0.1.0-py3-none-manylinux_2_28_aarch64.whl
| Download URL | sgl_router-0.1.0-py3-none-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 16.1 MB |
| Tags | Linux glibc 2.28+ ARM64 Python 3 |
|
SHA-256 checksum How to use checksums |
4ef8b573fd28bc94c93e47acccac48ba8c3d3ab726ada0c1bd1e3759befc7ba2
|
|
BLAKE2b-256 checksum How to use checksums |
1d2a25a9c733b5ecc9b66a6dba070c775b5848430fcb68284afdd5ba9452ea0d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.
Transparency logSecurity for sgl-router 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for sgl_router-0.1.0-py3-none-manylinux_2_28_x86_64.whl
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.