Skip to main content

sgl-router

Slim, KV-aware router for SGLang workers.

A router serves one model and spreads its requests across that model's SGLang engines. It picks an engine per request from live engine load, global KV-cache prefix matches, or session affinity, supports prefill/decode (P/D) disaggregation, bucketed deployments, and discovers engines from a static URL list or Kubernetes EndpointSlices.

Related docs: ARCHITECTURE.md (routing design and bucket configuration), REFERENCE.md (detailed feature behavior), monitoring/README.md (metrics and Grafana), BENCHMARKS.md, sgl-kv-indexer/README.md (external KV indexer), EXAMPLES.md (GLM-5.3-Flash and DeepSeek-V4.1-Flash, end to end).

Architecture

A deployment is a router in front of buckets of SGLang engines. A bucket is a logical group defined in the router configuration. It holds either one plain engine group, where each engine does prefill and decode, or separate prefill and decode (P/D) groups. Buckets can be length tiers (for example short and long context); without a bucket file there is one plain bucket and one P/D bucket containing every discovered engine.

Router deployment with short-context and long-context P/D engine buckets

Engines publish load reports and KV events. The router uses them, plus its own in-flight counts and session bindings, to pick engines. An optional external KV indexer can replace the router's local radix tree as the source of prefix matches. In P/D, the router sends the request to both the chosen prefill and decode engines, and prefill transfers KV directly to decode.

For each request, the router renders and tokenizes it, tries the buckets that fit its length (ordered by SLO preference, capacity, then rank), lets the group's policy pick an engine that passes admission, and dispatches it, retrying on another engine if it fails before any response. Output handling (OpenAI responses, tool and reasoning parsing) also happens in the router.

Scalability. Routers are stateless: nothing is persisted or shared between replicas. Each one rebuilds its routing state (engine load, KV prefix tree, in-flight counts) from the engines' own reports, so you scale a large deployment by running more router replicas behind a load balancer. A new replica can copy the KV tree from a warm sibling at startup (peer bootstrap). Session bindings stay in each replica's memory, so with session_aware and several replicas, route a session's requests to the same replica (for example, a load balancer sticky on the session header).

Quick start

Pull the container image (multi-arch, amd64 and arm64; see Releases and versions):

docker pull lmsysorg/sgl-router:latest            # latest release
docker pull lmsysorg/sgl-router:nightly           # built from main every night
docker run --rm -p 30000:30000 lmsysorg/sgl-router:latest --help

Or build from source:

cd sgl-router
cargo build --release

Run in front of two engines with cache-aware routing:

./target/release/sgl-router \
  --host 0.0.0.0 --port 30000 \
  --model-id qwen3 \
  --worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000 \
  --chat-routing reorg --policy cache_aware

Send a request:

curl http://localhost:30000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "qwen3", "messages": [{"role": "user", "content": "Hello"}]}'

--model-id is required. Any model name is served as --model-id, and the engine receives it as sent. --model-alias (repeatable) forwards another model name as --model-id, for example a gateway that sends the engine's --model-path. Aliases are forwarded as --model-id, preserving an optional :adapter LoRA suffix; names containing colons, such as s3://bucket/model, are matched in full. /v1/models lists only --model-id. --tokenizer-path takes a local tokenizer.json or a HuggingFace repo id; when omitted, the tokenizer for --model-id is downloaded from HuggingFace (honoring HF_TOKEN / HF_HOME). sgl-router --help lists every flag.

Key interfaces

HTTP endpoints

Endpoint Purpose
POST /v1/chat/completions, POST /v1/completions OpenAI API, served over /generate
POST/PUT /generate SGLang native generate, buffered or SSE
POST /v1/messages, /v1/messages/count_tokens Anthropic Messages
POST /v1/embeddings, /v1/classify, /v1/rerank Embeddings, classify and rerank
POST /v1/tokenize, /v1/detokenize Tokenize with the router's tokenizer
GET /v1/models The served model
POST /flush_cache Fan out a cache flush to every worker, with the caller's timeout
GET /server_info Answered by one healthy worker (a decode worker under P/D, which reports decode's state); /get_server_info is an alias
GET /model_info, /health_generate Answered by one healthy worker (a prefill worker under P/D; /health_generate also runs on one of its decode peers and returns the decode's answer when both pass); /get_model_info is an alias
GET /healthz, /readyz Liveness and readiness; /health, /liveness, and /readiness are aliases
GET /metrics Prometheus metrics, see monitoring/README.md

Endpoints for features the router does not provide yet (the Responses and Conversations APIs, /v1/loads) answer 501 with error code unsupported_endpoint.

Both edges speak cleartext HTTP/2 where the peer does; see operations. With --grpc-port, the router also serves the engine's native gRPC API.

Key flags

Area Flags
Model --model-id, --model-alias, --tokenizer-path, --no-tokenizer
Engines --worker-urls, or --service-discovery with --selector (plain) or --prefill-selector + --decode-selector (P/D)
Routing --chat-routing reorg, --policy, --bucket-config, --affinity-*, --session-id-header
Admission --max-inflight-requests, --max-waiting-requests, --max-pending-prefill-tokens, --max-running-usage, --max-kv-usage
Reliability --retry-max-attempts, --request-timeout-secs, --stale-request-timeout-secs
KV signals --cache-prefix-provider, --kv-indexer-endpoint, --kv-peer-selector

Omitting --chat-routing keeps the legacy routing path (default --policy round_robin). This README and ARCHITECTURE.md describe --chat-routing reorg.

Routing policies

Each engine group runs one policy, chosen by --policy (default power_of_two under reorg) or per group in a bucket file:

  • power_of_two: samples two engines and takes the less loaded. Prefill and plain engines compare estimated prefill queue time, waiting uncached tokens, waiting, running, then in-flight requests; decode engines compare waiting and running requests, then KV usage. Decode groups always default to it.
  • cache_aware: prefers the engine holding the longest cached prefix of the prompt (at least --cache-affinity-min-matched-tokens, default 1024, and optionally --cache-affinity-min-match-ratio of the prompt). Plain and prefill groups only. Needs a tokenizer.
  • session_aware: keeps requests with the same session header (--session-id-header, default x-session-id) on the engine they are bound to. Any stage, including decode via a bucket file.

Both affinity policies fall back to power_of_two among the remaining engines when there is no preferred engine or it fails admission. With --affinity-mode prefer (default) the preferred engine wins whenever it is admitted. With --affinity-mode balanced the router also samples an alternative and switches only when the preferred engine's load exceeds both alternative * --affinity-load-factor (default 2) and alternative + --affinity-load-gap, measured by --affinity-balanced-by (prefill-tokens by default, or running-requests).

Admission flags set per-engine caps; an engine at a cap is skipped:

Flag Rejects an engine at
--max-inflight-requests N N requests this router has in flight to it (--max-in-flight is an alias)
--max-waiting-requests N N engine-reported waiting requests
--max-pending-prefill-tokens N N engine-reported waiting uncached tokens
--max-running-usage F share F of its reported running-request capacity
--max-kv-usage F share F of its reported KV capacity

Engine-reported caps lag by one load report, so a burst can pass them; --max-inflight-requests counts dispatches as they happen.

For the full selection rules see ARCHITECTURE.md: load-aware routing, cache- and session-aware routing, and configuration examples.

Deployment examples

For complete model deployments (engine flags, router flags, prefer vs balanced affinity), see EXAMPLES.md.

Static workers, plain engines:

sgl-router --model-id qwen3 --chat-routing reorg --policy cache_aware \
  --worker-urls http://10.0.0.1:30000 http://10.0.0.2:30000

Kubernetes discovery, plain engines (omit the namespace to watch all namespaces, which needs cluster-wide RBAC):

sgl-router --model-id qwen3 --chat-routing reorg --policy cache_aware \
  --service-discovery --service-discovery-namespace prod \
  --selector app=engines-qwen3

Kubernetes discovery, P/D engines with session affinity:

sgl-router --model-id qwen3 --chat-routing reorg --policy session_aware \
  --service-discovery --service-discovery-namespace prod \
  --prefill-selector app=engines-qwen3,role=prefill \
  --decode-selector app=engines-qwen3,role=decode \
  --max-kv-usage 0.9

Length-tiered buckets from a file (schema and a full example in ARCHITECTURE.md):

sgl-router --model-id qwen3 --chat-routing reorg --policy cache_aware \
  --service-discovery --service-discovery-namespace prod \
  --prefill-selector app=engines-qwen3,role=prefill \
  --decode-selector app=engines-qwen3,role=decode \
  --bucket-config buckets.json

Add a new policy

Reorg policies live in src/policies_reorg/. Contributions are welcome; read ARCHITECTURE.md first for how buckets, admission and fallback fit together.

  1. Implement the Policy trait in src/policies_reorg/<name>.rs. The trait is in src/policies_reorg/mod.rs:

    pub trait Policy: Send + Sync + Debug {
        fn pick<'a>(&'a self, engines: &'a [Arc<Worker>], request: &'a PickRequest<'a>)
            -> BoxFuture<'a, Result<Pick, PickError>>;
        fn supports(&self, _stage: Stage) -> bool { true }
        fn needs_request_tokens(&self) -> bool { false }
        fn fallback(&self) -> Option<&dyn Policy> { None }
    }
    
    • pick must return one of the supplied engines and check admission on it (self.admission.check(&engine, &metrics)), returning PickError::AdmissionRejected on rejection.
    • The policy holds its own shared-state handles (load table, KV source, session store), passed in at construction, and reads them inside pick.
    • Override supports to restrict stages (plain, prefill, decode), needs_request_tokens if it reads request.token_ids or request.prefix, and fallback if it delegates to another policy.
    • power_of_two.rs is a complete ~80-line example; session_aware.rs shows affinity with a fallback.
  2. Declare the module in src/policies_reorg/mod.rs.

  3. Add a PolicyKind variant in src/config/types.rs with its --policy name. PolicyKind is shared with the legacy path, so add an arm returning an error to build_kind in src/policies/factory.rs (the compiler lists every exhaustive match to update).

  4. Wire it into src/policies_reorg/factory.rs: add it to the allowlist in validate() and construct it in the match kind of Groups::build.

  5. Test it in tests/component/policies_reorg_<name>.rs, registered in tests/component/main.rs, following policies_reorg_power_of_two.rs. For end-to-end routing, add a case under tests/proxy/chat_routing/reorg/.

  6. Document it: a subsection in ARCHITECTURE.md next to section 1.4, and a bullet in Routing policies above.

Other features

Each item links to its full description in REFERENCE.md.

Native /generate

Same body and response as the engine's /generate, buffered or SSE. With a tokenizer loaded, text is forwarded as input_ids so routing sees the engine's exact tokens (--disable-input-ids-forwarding turns this off). See native /generate.

OpenAI over /generate

For /v1/chat/completions and /v1/completions, the router runs SGLang's OpenAI layer itself: it sends the request to /generate and builds the OpenAI response. Requests it cannot reproduce exactly (for example media or strict tools) go to the engine's own route unchanged. See OpenAI over /generate for the exact conditions, and chat rendering plus the per-model sections (DeepSeek V4 and V4.1, GLM-5.3, Qwen3.8, Kimi-K3, MiniMax-M3, StepFun) for which chats forward input_ids.

Anthropic Messages

/v1/messages is converted to a chat completion and back. /v1/messages/count_tokens is answered by the router from the chat template. See Anthropic Messages.

Embeddings, classify and rerank

Engine request and response bodies, routed on load. Not available on P/D fleets. See embeddings, classify and rerank.

Native gRPC

--grpc-port serves the engine's sglang.runtime.v1.SglangService. Each RPC is routed like its HTTP route and forwarded to the same RPC on the engine's own native gRPC port, so engines must run with --grpc-port too. See native gRPC.

Bucket file

--bucket-config defines buckets whose groups set worker_services (namespace/name) or worker_ids, plus policy, admission and affinity; unset fields inherit the CLI flags. See ARCHITECTURE.md and bucket file membership.

KV cache signals

  • External KV indexer: --cache-prefix-provider indexer --kv-indexer-endpoint <url>; see sgl-kv-indexer and external KV indexer.
  • KV-event gap replay: add "replay_endpoint" to the engine's --kv-events-config so the router can re-fetch dropped KV events; see gap replay.
  • Pending prefixes: --cache-pending-prefix-ttl-ms keeps bursts that share a cold prefix on one worker until its KV events arrive; see pending prefixes.
  • Peer bootstrap (Kubernetes): --kv-peer-selector kubernetes.io/service-name=<router-service> makes a new replica copy the KV tree from a sibling before /readyz passes; see peer bootstrap.
  • LoRA cache identity: with workers that publish BlockStored.lora_name, prefix matches are kept per adapter; see LoRA cache identity.

Operations

  • No tokenizer: --no-tokenizer for load-only policies (power_of_two, session_aware); see optional tokenizer.
  • Retries: --retry-max-attempts N resends a request that failed before any response to another worker; see retries.
  • Circuit breaker: a worker leaves routing after --cb-threshold consecutive failures; --disable-circuit-breaker turns this off; see circuit breaker.
  • DP-rank routing: --dp-aware also picks the DP rank inside multi-rank engines, and --dp-rank-policy round_robin cycles ranks that no routing key or cached prefix decides; in PD mode, --decode-dispatch-limit N picks the decode rank shortly before the KV arrives; see DP-rank routing.
  • PD version groups: --pd-version-group-label <key> pairs prefill and decode only within the same engine version; see PD version groups.
  • Request body size: inference routes answer 413 past 32 MiB; SGLANG_ROUTER_MAX_BODY_BYTES=<bytes> raises the cap for large multimodal requests, at the cost of more router memory per in-flight request; see request body size.
  • Engine API key: --worker-api-key for engines launched with --api-key; see engines with --api-key.
  • Sampling contract: --override-sampling-params '<json>' pins sampling parameters fleet-wide; --sampling-param-conflict reject|allow decides what happens to requests that differ. See fleet-wide sampling contract for parameter domains and null handling.
  • Chat rendering: use the same model files and chat-template settings on router and workers, or set --disable-input-ids-forwarding. --tokenizer-backend fast and --tokenizer-l1-cache-mb N speed up tokenization.
  • HTTP/2: no configuration. h2c is used inbound when the client speaks it, and outbound to workers that report enable_http2; see HTTP/2.
  • Upgrading: the cache_aware_zmq policy and the legacy cache and balance threshold flags were removed; see upgrading.

FIPS builds

For deployments that require FIPS 140-validated cryptography, build with the opt-in fips cargo feature. The router then uses the aws-lc-rs FIPS module as its TLS provider (outbound connections to workers and the Kubernetes API) and exits at startup unless FIPS mode is active. Default builds are unchanged.

cargo build --release --features fips
# or the container image
docker build -f docker/sgl-router.Dockerfile --build-arg CARGO_FEATURES=fips .
  • The FIPS module builds from source and needs cmake, Go and clang, on a glibc (not musl) target. The Dockerfile installs them when fips is requested; RUNTIME_BASE_IMAGE selects the runtime base, e.g. a FIPS-mode OS image.
  • FIPS builds do not download tokenizers from Hugging Face Hub (its HTTP client cannot use the FIPS module): pass a local --tokenizer-path, or --no-tokenizer for load-only policies.
  • On macOS, the FIPS module links as a shared library; set DYLD_LIBRARY_PATH to its build output directory to run a native build.

Releases and versions

sgl-router is versioned on its own semver track, independent of SGLang releases. Each release ships as:

  • a container image, lmsysorg/sgl-router tagged v<version> and latest (the newest release);
  • a PyPI package, pip install sgl-router, whose wheel (linux x86_64 and aarch64) puts the sgl-router binary on PATH.

lmsysorg/sgl-router:nightly is built from main every night, and the SGLang images build the router from their own source. The crate itself is not published to crates.io. sgl-router --version prints the version a binary was built from.

Releases are made manually:

  1. Bump version in Cargo.toml and run cargo check once so Cargo.lock records it (CI builds with --locked): the minor version for new routing behavior, flags, or protocol changes; the patch version for fixes only. Open a PR with the bump and the release notes.
  2. Once the bump is on main, a maintainer tags that commit router-v<version> and pushes the tag. That tag starts release-sgl-router.yml, which refuses a tag that differs from the Cargo.toml version, builds the image and the wheels for amd64 and arm64, pushes lmsysorg/sgl-router:v<version> and :latest, and uploads the wheels to PyPI.

Do not tag a router release v<version>: SGLang's own PyPI and Docker release workflows run on every v* tag.

License

Apache-2.0.

Metadata

Release files for sgl-router 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for sgl-router 0.1.0
File Interpreter ABI Platform Actions
sgl_router-0.1.0-py3-none-manylinux_2_28_x86_64.whl Python 3 none Linux glibc 2.28+ x86-64 Details for sgl_router-0.1.0-py3-none-manylinux_2_28_x86_64.whl
sgl_router-0.1.0-py3-none-manylinux_2_28_aarch64.whl Python 3 none Linux glibc 2.28+ ARM64 Details for sgl_router-0.1.0-py3-none-manylinux_2_28_aarch64.whl

Total release size: 32.2 MB

Release files / sgl_router-0.1.0-py3-none-manylinux_2_28_x86_64.whl

Download URL sgl_router-0.1.0-py3-none-manylinux_2_28_x86_64.whl
Size 16.2 MB
Tags Linux glibc 2.28+ x86-64 Python 3
SHA-256 checksum
How to use checksums
f22d3c2de80afb11080bfd0b3be523088e2f0a1f754b6f9fc6e6267d5c5e6107
BLAKE2b-256 checksum
How to use checksums
acf56a609d50cbc365685e4c05eb6dbec67b1ca1f5a60da28d57bece7fe3f823
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Release files / sgl_router-0.1.0-py3-none-manylinux_2_28_aarch64.whl

Download URL sgl_router-0.1.0-py3-none-manylinux_2_28_aarch64.whl
Size 16.1 MB
Tags Linux glibc 2.28+ ARM64 Python 3
SHA-256 checksum
How to use checksums
4ef8b573fd28bc94c93e47acccac48ba8c3d3ab726ada0c1bd1e3759befc7ba2
BLAKE2b-256 checksum
How to use checksums
1d2a25a9c733b5ecc9b66a6dba070c775b5848430fcb68284afdd5ba9452ea0d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Security for sgl-router 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for sgl_router-0.1.0-py3-none-manylinux_2_28_x86_64.whl

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page