smg-grpc-servicer

gRPC servicer implementations for LLM inference engines. Supports vLLM, MLX, TokenSpeed, and SGLang.

Installation

For vLLM:

pip install smg-grpc-servicer[vllm]

For MLX:

pip install smg-grpc-servicer[mlx]

For TokenSpeed, install the TokenSpeed runtime first, then install the servicer bridge:

pip install smg-grpc-servicer

For SGLang:

pip install smg-grpc-servicer[sglang]

Usage

vLLM

vllm serve meta-llama/Llama-2-7b-hf --grpc

Worker-side multimodal processing (media refs)

By default the smg router fetches and preprocesses images itself and sends pixel tensors. A vLLM gRPC worker can instead accept media references (URLs) and run vLLM's own multimodal processor (or, on the Rust request path, smg's own pipeline with --mm-processor smg; see below):

vllm serve Qwen/Qwen3-VL-8B-Instruct --grpc --mm-processor inprocess \
    --allowed-media-domains example.com

The --mm-* flags come from a vLLM launcher that knows them (it hands an MmSettings to VllmEngineServicer), or from this package's general plugin, which defines them on a gRPC launcher's parser that lacks them: vllm serve --grpc and the stock python -m vllm.entrypoints.grpc_server (there the values also reach a servicer built without the namespace, through the environment). A flag left out falls back to the matching SMG_VLLM_MM_* variable, which logs a deprecation line on a launcher that has the flag; a launcher without the flags (no plugins loaded, a servicer built on its own) takes the variable as its documented setting and logs nothing. The startup line VllmEngineServicer initialized (mm_processor=inprocess, source=flag) and the mm_processor_source label name where the value came from.

The worker then advertises mm_processor=inprocess and mm_media_ref_schemes through GetServerInfo; a router with media-reference support forwards media_refs only to workers that advertise, and a router without it ignores the labels and keeps sending preprocessed tensors. Whatever the processor mode, a multimodal engine also advertises its resolved --limit-mm-per-prompt as mm_item_limits (image=8,video=2), so a router that preprocesses media itself holds requests to the engine's limits. vLLM's --allowed-media-domains, --allowed-local-media-path, --media-io-kwargs, --limit-mm-per-prompt and VLLM_*_FETCH_TIMEOUT govern fetching on the worker; without --allowed-media-domains the worker fetches from any public host the router forwards. A media URL that names or resolves to a loopback, link-local or private address is refused on the worker as on the router, redirects included, unless SMG_MM_ALLOW_PRIVATE_MEDIA_HOSTS=1 is set for a deployment whose media origin is internal. Related knobs: --mm-max-inflight (SMG_VLLM_MM_MAX_INFLIGHT, default 64) bounds concurrent media jobs; --mm-max-items (SMG_VLLM_MM_MAX_ITEMS, unset by default) overrides the model's per-modality reference limits; --mm-max-item-bytes (SMG_VLLM_MM_MAX_ITEM_BYTES, default 32 MiB) caps inline data: payloads; SMG_VLLM_MM_MAX_VIDEO_FRAMES (env only for now; default 0, meaning vLLM's own --media-io-kwargs decide) caps the frames a video is sampled to, so long clips stay bounded. All eight flag-backed settings are validated when the servicer starts, whatever the processor mode, so a stale unreadable value fails loudly. Two layers bound an inline item: the router refuses a data: item above its own per-kind input bound (SMG_IMAGE_MAX_INPUT_BYTES, SMG_VIDEO_MAX_INPUT_BYTES, SMG_AUDIO_MAX_INPUT_BYTES; default 256 MiB each) with a 400 media_ref_too_large before the worker sees it, and the worker refuses one above --mm-max-item-bytes with a message that names the worker as the flag's owner, which the router relays as a 400 start_generation_failed. With the defaults a 50 MB item clears the router and stops at the worker's 32 MiB, so that flag is the one to raise; the router's --max-payload-size (default 512 MiB) must also hold the whole request body.

On the router side, --mm-processing selects auto (default: forward when the model's spec opts in and every registered worker of the model advertises mm_processor), router (always preprocess) or worker (strict: 400 when a request cannot be forwarded); the outcome is counted in smg_mm_processing_total{model,mode,reason}, and the startup line multimodal processing mode names the value and its source (flag, env or default). SMG_MM_PROCESSING is the deprecated env fallback: it applies only when the flag is absent, logs a deprecation line, and goes away in the next minor release. Any other value stops the router at startup instead of quietly reverting to auto. On the worker path the router never expands placeholders, so routing decisions that weigh the prompt's token count (cache-aware policies, load estimates) see one token per media item where the worker will schedule the full placeholder run. The E2E_MM_PROCESSING=worker e2e lanes run the multimodal suites in this mode, and crates/multimodal/scripts/check_worker_anchor_parity.py checks that a spec's anchor is the token vLLM expands.

To move fetching and processing out of the vLLM process, run the GPU-free sidecar next to a private Redis and point the worker at it (pip install smg-grpc-servicer[vllm,vllm-redis]):

python -m smg_grpc_servicer.vllm.mm_sidecar --model Qwen/Qwen3-VL-8B-Instruct \
    --redis-url redis://127.0.0.1:6379/0 --allowed-media-domains example.com
vllm serve Qwen/Qwen3-VL-8B-Instruct --grpc --mm-processor redis \
    --mm-redis-url redis://127.0.0.1:6379/0

The sidecar and the worker must agree on model, vLLM version, dtype, video backend, media/processor kwargs and --limit-mm-per-prompt (pass the flag to both processes; the sidecar's limit is the one that applies, the limit is resolved per modality before hashing so equivalent spellings match, and the key namespace is derived from all of these): the worker advertises mm_processor=redis only while a sidecar with a matching fingerprint keeps its hello key alive, and rejects results that disagree. Jobs and results travel over Redis lists under smg:mm:v1:{namespace}; results carry full tensors keyed by a per-attempt job id and expire after 120 s. Knobs: --mm-sidecar-timeout-ms (SMG_VLLM_MM_SIDECAR_TIMEOUT_MS, 30000), --mm-sidecar-max-queue (SMG_VLLM_MM_SIDECAR_MAX_QUEUE, 256, fail fast when the queue is deeper), --mm-sidecar-namespace (SMG_VLLM_MM_SIDECAR_NAMESPACE, override the derived namespace). The sidecar resolves --redis-url and --namespace the same way, so the two processes cannot disagree on the namespace; the timeout travels with each job as its deadline. On the sidecar, SMG_VLLM_MM_MAX_RESULT_BYTES (default 512 MiB, lowered to Redis's proto-max-bulk-len when that is smaller) caps an encoded result and SMG_VLLM_MM_MAX_VIDEO_FRAMES caps video sampling as above. A result over the cap is answered as a 400 media_too_large instead of being pushed, and a result Redis refuses is reported to the worker at once; a sidecar timeout is not retried by the router, since the worker already spent the whole budget on it.

Rust request path (SMG_VLLM_SERVICER_IMPL=rust)

The same vllm.grpc.engine.VllmEngine contract can be served from Rust, with Python keeping only the lifecycle. The switch is a flag inside this package, not a second server, and it needs no vLLM change: vLLM's gRPC launcher imports this package's servicer classes before it defines serve_grpc, and that import installs the switch over serve_grpc (launcher_switch.py), so the launcher asks the package which implementation to run before it builds an AsyncLLM and hands the process to smg_grpc_servicer.vllm.serve_rust when the answer is rust. That function launches the engine headless (vllm serve --headless), which dials a same-host ZMQ handshake, and serves the gRPC contract from smg.servicer.VllmGrpcServer on a Rust-owned thread. The Router cannot tell the two apart.

# Python (default): upstream's gRPC server, AsyncLLM in-process.
vllm serve Qwen/Qwen3-0.6B --grpc --port 50051

# Rust request path, same entrypoint. The flag is this package's: vLLM loads
# it as a general plugin while it builds the parser (smg_grpc_servicer/vllm/plugin.py).
vllm serve Qwen/Qwen3-0.6B --grpc --port 50051 --servicer-impl rust

# The environment form works for the import-based entrypoints too (`vllm serve --grpc`,
# the deprecated `python -m vllm.entrypoints.grpc_server`); a launcher file executed
# directly as __main__ is not switched and refuses the flag instead.
SMG_VLLM_SERVICER_IMPL=rust python -m vllm.entrypoints.grpc_server --model Qwen/Qwen3-0.6B --port 50051

What the switch runs ahead of serve_grpc is the check a launcher could also carry in its own source:

from smg_grpc_servicer.vllm import resolve_servicer_impl, serve_rust

if resolve_servicer_impl(args) == "rust":
    raise SystemExit(await serve_rust(args))

smg serve --backend vllm --connection-mode grpc --servicer-impl rust sets the flag in each worker's environment, after checking that the installed vLLM's launcher will consult it (smg_grpc_servicer.vllm.rust.upstream_hook_installed: it imports this package at module level, or carries the check itself); the Python servicer refuses to start when the flag asks for Rust, so a launcher that reaches it anyway fails loudly instead of silently running Python. The headless engine is launched from the parsed namespace through vLLM's own run_headless, so both entrypoints above work unchanged.

Rust mode needs the smg wheel (for the binding) and serves the whole contract the Python servicer serves: text generation, PD disaggregation (--kv-transfer-config: connector params pass through both ways and GetServerInfo carries the pairing identity), Router-preprocessed media (inline and /dev/shm tensors), worker-side media processing (media_refs, below), Embed, FlushCache, GetTokenizer (which answers FAILED_PRECONDITION when the launcher could not resolve a local tokenizer directory) and SubscribeKvEvents (vLLM's ZMQ publisher, which Rust mode turns on by itself when the launcher is given no --kv-events-config; an explicit one is kept as given, and SMG_VLLM_SERVICER_KV_EVENTS=0 leaves the publisher off). Tuning: SMG_VLLM_SERVICER_HANDSHAKE_PORT (default: a free port), SMG_VLLM_SERVICER_DRAIN_SECS (default 5), SMG_VLLM_SERVICER_STARTUP_TIMEOUT_SECS (default 1800: how long the servicer waits for a sign of life from the engine during its handshake; the launcher reports the engine process alive each time it polls it, and the handshake's own messages count too, so a healthy start that takes longer, a large checkpoint streaming in for an hour, still completes, while an engine that exits fails fast regardless), SMG_VLLM_SERVICER_STARTUP_CEILING_SECS (default 14400: the most a start may take however alive the engine is; 0 lifts the ceiling), SMG_ZMQ_SOCKET_DIR, SMG_SERVICER_WORKER_THREADS (default 1, for the one runtime server.rs builds for the vLLM, SGLang and TokenSpeed Rust servicers alike: one runtime worker carries about 100k output tokens/s on one connection with its lock uncontended; raise it to 2-4 when one engine serves more than about 512 concurrent streams and their inter-token p99 matters more than the servicer's CPU).

Worker-side media processing uses the same --mm-processor / SMG_VLLM_MM_PROCESSOR setting as the Python servicer, with one more choice:

• inprocess (vLLM's MediaConnector and the engine's renderer) and redis (the sidecar) are the Python servicer's backends, so the media is fetched and processed by vLLM's own code on either servicer. The Rust server hands a request's media_refs to the Python bridge (smg_grpc_servicer.vllm.rust_media), which runs the processor and vLLM's input processor on the launcher's asyncio loop and returns the expanded prompt and mm_features as vLLM's own encoder writes them; Rust relays the encoded features to the engine untouched and sends the tensor frames straight from the memory Python lent it, without a copy.
• smg is smg's own media pipeline, the one the Router runs for --mm-processing router, run inside the Rust servicer with no Python on the request path: fetch, decode, preprocess, placeholder expansion, and the batches a Router-preprocessed request would carry, translated for the engine the same way. It serves the model families that pipeline supports (crates/multimodal), takes http, https and data references, reads the model's config.json and preprocessor configs from the tokenizer directory the launcher resolved, refuses a request above the engine's own --limit-mm-per-prompt as the engine's server does (--mm-max-items and SMG_*_MAX_COUNT tighten those limits, never loosen them), and samples a video to the engine's own frame budget (--media-io-kwargs video.num_frames, else its loader's default) where the family's processor takes the loader's frames, so a clip costs the same tokens as on the engine's own server; the budget is read under the same SMG_VLLM_MM_MAX_VIDEO_FRAMES cap the in-process and Redis processors apply, so an unbounded count stays bounded in smg mode too (a cap below the engine's count then samples fewer frames than the engine's own server would, as intended), and a loader rule the pipeline cannot follow (video_backend in the engine's media kwargs, else VLLM_VIDEO_LOADER_BACKEND, other than opencv, or a video.fps above zero, which thins frames by duration) is refused at launch for a family that samples the way the loader does (Gemma 4) and serves video at all; the families with a rate-based sampler of their own never followed the loader and start as before. An engine that normalizes pixels on device (vLLM's mm_device_do_normalize, on by default for the Qwen-VL family) takes raw uint8 pixels, and the pipeline writes those for it; a model whose processor cannot emit raw pixels is refused at startup under that setting (start the engine with --mm-device-do-normalize=false or use inprocess). The Python servicer refuses smg: it has no such processor.

The in-flight cap, the saturation refusal, the advertised mm_processor / mm_media_ref_schemes / mm_processor_source and the PD prefill leg's media_identity behave as on the Python servicer whichever processor runs; the engine-side contract does not change.

Known difference: under --structured-outputs-config.backend auto (the default) vLLM's frontend validates each constraint with xgrammar and falls back to guidance when xgrammar rejects it. Rust mode applies the same static rules (JSON-schema features xgrammar lacks go to guidance, a choice becomes the grammar xgrammar compiles), but it cannot run xgrammar's parser, so a regex or grammar that only guidance accepts fails that request at the engine's grammar compile instead of falling back. Pin guidance (or xgrammar) explicitly when that matters; the engine keeps the backend of its first structured request either way, as it does behind vLLM's own frontend.

Both vLLM servicers also advertise the engine's reasoning parser (engine_reasoning_parser; reasoning_parser is the per-model parser override a router reads off a worker's labels) and where its grammar starts on a prompt that ends inside the model's thinking block (constrained_decoding_start): after_reasoning when a reasoning parser is set and enable_in_reasoning is off, since vLLM then holds a request's grammar back until the parser has seen the thought's end, first_token otherwise. A router shapes a forced tool call's grammar by it as the engine's own server would: the bare calls behind the parser's gate, so the call is forced right there; the thought and then the calls from the first token.

Multi-node engines: the worker pod's liveness

An engine whose tensor-parallel ranks span pods runs its EngineCore and the gRPC servicer on a leader pod (--nnodes N --node-rank 0 --master-addr <its own address> --master-port <port>) and the other ranks on worker pods (vllm serve <model> --headless --nnodes N --node-rank <k> --master-addr <the leader>, the same engine flags). Every rank joins the process group through the leader's TCP store at --master-port and keeps that connection for its lifetime. When the leader's engine goes away (the pod restarts, or the servicer gives up on a start), the ranks on the worker pods are not told: their store connection is dead and they wait on it, the next leader waits for ranks that never join, and a worker probe that only asks whether the engine process exists (grep vllm /proc/1/cmdline) keeps those orphans running until someone deletes the pod. A leader restart must restart its workers: ranks cannot re-join a store they were not started against.

python -m smg_grpc_servicer.vllm.worker_probe is the worker pod's probe. It passes while at least one TCP connection from the pod to the leader's store port is established and fails otherwise, so the kubelet restarts the worker and its fresh ranks join the new leader within periodSeconds x failureThreshold. Without arguments it reads --master-addr and --master-port off the worker's own command line (/proc/1/cmdline); --leader <name> names the leader by its stable DNS name instead, re-resolved on every run, so that a recreated leader pod at a new address fails the probe even while the stale connections to the old one linger, and --port names the store port. The ranks connect to the store early in their start, before the model loads, so the same command serves as the startup probe, with a budget that covers waiting for the leader. Run it with python3 -I: the probe itself needs only the standard library, but the engine's PYTHONPATH may carry site customizations that import the engine, and an interpreter start on a node saturated by the weight stream takes seconds; -I leaves both out, and timeoutSeconds of 30 or more keeps a slow start from counting as a failure:

# the worker pod's container; LEADER_HOST is the leader's stable DNS name
startupProbe:
  exec: {command: [/bin/sh, -c, 'python3 -I -m smg_grpc_servicer.vllm.worker_probe --leader "$LEADER_HOST"']}
  periodSeconds: 30
  timeoutSeconds: 30
  failureThreshold: 480
livenessProbe:
  exec: {command: [/bin/sh, -c, 'python3 -I -m smg_grpc_servicer.vllm.worker_probe --leader "$LEADER_HOST"']}
  periodSeconds: 30
  timeoutSeconds: 30
  failureThreshold: 3

MLX

python -m smg_grpc_servicer.mlx --model meta-llama/Llama-2-7b-hf --host 0.0.0.0 --port 50051

TokenSpeed

# Python (default)
python -m smg_grpc_servicer.tokenspeed --model meta-llama/Llama-2-7b-hf --host 0.0.0.0 --port 50051

# Rust request path, same entrypoint
python -m smg_grpc_servicer.tokenspeed --model meta-llama/Llama-2-7b-hf --host 0.0.0.0 --port 50051 --servicer-impl rust

# The environment form, overridden by the flag when both are given
SMG_TOKENSPEED_SERVICER_IMPL=rust python -m smg_grpc_servicer.tokenspeed --model meta-llama/Llama-2-7b-hf --port 50051

This is the process ts serve spawns for its gRPC worker; --servicer-impl is this package's flag (listed first by --help, ahead of TokenSpeed's own flags), and SMG_TOKENSPEED_SERVICER_IMPL is the fallback the flag overrides (flag, then the environment, then python; the launcher logs which one decided). With --servicer-impl rust the same process serves the tokenspeed.grpc.scheduler.TokenSpeedScheduler contract from Rust (smg.servicer.TokenSpeedGrpcServer, which needs the smg wheel): the launcher computes the model and server facts from TokenSpeed's own config, runs the scheduler(s) headless in a spawned child over the msgpack ZMQ wire, and supervises both. What that wire does not carry is reported, not emulated: FlushCache and profiling answer UNIMPLEMENTED, ranked top_logprobs and prompt logprobs are refused, and PD/EPD disaggregation stays with the Python implementation, as do the RL control-plane extras (the rl.* advertisement, the live weight_version on generate responses, is_paused); the Rust server redacts credentials from server_args the same way. SubscribeKvEvents relays TokenSpeed's ZMQ KV-event publisher, which Rust mode turns on by itself when the launcher is given no --kv-events-config (an explicit one is kept as given; SMG_TOKENSPEED_SERVICER_KV_EVENTS=0 leaves the publisher off). Tuning: SMG_TOKENSPEED_SERVICER_HANDSHAKE_PORT (default: a free port), SMG_TOKENSPEED_SERVICER_DRAIN_SECS (default 5), SMG_TOKENSPEED_SERVICER_STARTUP_TIMEOUT_SECS (default 1800) and SMG_TOKENSPEED_SERVICER_STARTUP_CEILING_SECS (default 14400; the two startup bounds as for vLLM above: the launcher reports the scheduler process alive each time it polls it, so a slow but living start completes, under the ceiling, and 0 lifts the ceiling), SMG_ZMQ_SOCKET_DIR, SMG_SERVICER_WORKER_THREADS (default 1, for the one runtime server.rs builds for the vLLM, SGLang and TokenSpeed Rust servicers alike: one runtime worker carries about 100k output tokens/s on one connection with its lock uncontended; raise it to 2-4 when one engine serves more than about 512 concurrent streams and their inter-token p99 matters more than the servicer's CPU).

Both TokenSpeed servicers advertise, as server_args keys, the engine's reasoning parser (engine_reasoning_parser; reasoning_parser is the per-model parser override a router reads off a worker's labels) and where its grammar starts on a prompt that ends inside the model's thinking block (constrained_decoding_start): always first_token, since this engine's grammar manager has no reasoning gate and starts every grammar at the first generated token (--reasoning-parser only has a JSON-schema constraint rewritten into a structural tag that lets the thought precede the JSON). A router shapes a forced tool call's grammar by it as the engine's own server would: the thought and then the calls, so the call is not forced into the thought.

Both TokenSpeed servicers also report the engine's constrained-decoding policy, when it declares one, as GetServerInfo.constrained_decoding (a field of its own, not a server_args key): always for a model that takes a grammar on every turn, requested for one only when the request asks. The servicers read it as constrained_decoding off TokenSpeed's ServerArgs, else its ModelConfig. A router turns the field into the worker's constrained_decoding label and records the policy there; the request paths that act on always are a separate change, so until they land every worker is served as requested, which is what every worker did before the label existed.

Multi-node engines: the worker pod's liveness

An engine whose tensor-parallel ranks span pods is launched on every pod with the same flags, --nnodes N --dist-init-addr <the leader>:<port> among them, and the pod's own --node-rank <k>: the launcher above on the leader pod (--node-rank 0, where the gRPC servicer runs), the scheduler ranks alone on the worker pods. Every rank joins the process group through the TCP store at --dist-init-addr, which rank 0 on the leader pod hosts, and keeps that connection for its lifetime, so the contract of the vLLM section above holds here too: a leader restart must restart its workers, and python -m smg_grpc_servicer.tokenspeed.worker_probe is the worker pod's probe. It passes while at least one TCP connection from the pod to the leader's store port is established and fails otherwise; without arguments it reads the leader and the port off --dist-init-addr on the worker's own command line (/proc/1/cmdline), --leader <name> names the leader by its stable DNS name instead, re-resolved on every run, and --port names the store port (both are needed when TokenSpeed derives the flag from a multi-node Slurm step's environment rather than the command line). Run it with python3 -I and a timeoutSeconds of 30 or more, as above:

# the worker pod's container; LEADER_HOST is the leader's stable DNS name
startupProbe:
  exec: {command: [/bin/sh, -c, 'python3 -I -m smg_grpc_servicer.tokenspeed.worker_probe --leader "$LEADER_HOST"']}
  periodSeconds: 30
  timeoutSeconds: 30
  failureThreshold: 480
livenessProbe:
  exec: {command: [/bin/sh, -c, 'python3 -I -m smg_grpc_servicer.tokenspeed.worker_probe --leader "$LEADER_HOST"']}
  periodSeconds: 30
  timeoutSeconds: 30
  failureThreshold: 3

SGLang

sglang serve --model-path meta-llama/Llama-2-7b-hf --grpc-mode

Both SGLang servicers advertise the engine's --limit-mm-data-per-request as the flat mm_item_limits label (image=1,video=1) in GetServerInfo's server_args, so a router that preprocesses media itself holds requests to the engine's limits: the precomputed inputs it sends bypass the tokenizer manager's own check. Without the flag the engine has no such limit and the label is absent.

They also advertise, as server_args keys, the engine's reasoning parser (engine_reasoning_parser; reasoning_parser is the per-model parser override a router reads off a worker's labels) and where its grammar starts on a prompt that ends inside the model's thinking block (constrained_decoding_start): after_reasoning with --reasoning-parser, since the scheduler then wraps its grammar backend in a reasoner that holds a request's grammar back until the parser's thought-end sequence has been generated (for a request that asks for reasoning, which a router does on a thinking prompt), first_token without one. A router shapes a forced tool call's grammar by it as the engine's own server would: the bare calls behind the parser's gate, so the call is forced right there; the thought and then the calls from the first token.

KV-event recovery

To retain cache knowledge across a recoverable event gap, configure SGLang's --kv-events-config with both a PUB endpoint and a replay endpoint, for example:

{"publisher":"zmq","endpoint":"tcp://*:5557","replay_endpoint":"tcp://*:5558","buffer_steps":10000}

The bridge subscribes to live events before requesting missed batches from the replay endpoint, preserves publisher sequence numbers, and removes overlap at handoff. Both subscriptions currently use DP rank 0; allocate non-overlapping port ranges if multiple DP ranks publish events. The Rust servicer's relay uses the same replay endpoint (the vLLM, SGLang and TokenSpeed launchers all pass replay_endpoint from the engine's kv-events config) for gaps in flight and for the batches published before its subscription joined the publisher; without one those are counted as lost or unknown, never silently skipped.

Without replay, or when history is expired, empty, malformed, or unavailable (timeout: five seconds), the bridge reports OUT_OF_RANGE before streaming or DATA_LOSS after streaming starts. SMG discards that worker's stale mappings and resubscribes with zero. A zero cursor rebuilds knowledge from subsequent live events; it is not a complete cache snapshot: this bridge relays per call and keeps no history or block record between calls, so it has nothing older to serve. The Rust servicer's relay (crates/engine_servicer, the vLLM servicer and SMG_SGLANG_SERVICER_IMPL=rust) does: it keeps a bounded history and the engine's live blocks for the servicer's lifetime, serves the history to a subscription from zero while it is complete, and a state snapshot (KvSnapshotChunk) before live events once the window has rolled. An empty replay is conservatively reset because it cannot distinguish an idle publisher from a restarted one.

Rust request path (--servicer-impl rust)

sglang serve --grpc-mode (python -m sglang.launch_server --grpc-mode) hands the process to this package's serve_grpc. With --servicer-impl rust that entry serves the sglang.grpc.scheduler.SglangScheduler contract from Rust (smg.servicer.SglangGrpcServer, which needs the smg wheel) instead: the scheduler runs headless in a spawned child over the msgpack ZMQ wire below, the Rust server speaks the gRPC contract on top of it, and Python keeps the lifecycle only. The flag is this package's, as it is for vLLM: SGLang loads it as an sglang.srt.plugins entry point while it builds its parser (smg_grpc_servicer/sglang/plugin.py), and SMG_SGLANG_SERVICER_IMPL is the fallback the flag overrides (flag, then the environment, then python; the servicer logs which one decided). A SGLANG_PLUGINS whitelist without smg-servicer removes the flag, not the environment form.

# Python (default)
sglang serve --model-path Qwen/Qwen3-0.6B --grpc-mode --port 30000

# Rust request path, same entrypoint
sglang serve --model-path Qwen/Qwen3-0.6B --grpc-mode --port 30000 --servicer-impl rust

# The environment form, overridden by the flag when both are given
SMG_SGLANG_SERVICER_IMPL=rust sglang serve --model-path Qwen/Qwen3-0.6B --grpc-mode --port 30000

python -m sglang.launch_server --help lists the flag; sglang serve --help prints SGLang's help before it loads plugins, so the flag is missing from that listing but accepted. The Router cannot tell the two implementations apart: text generation with sampled and prompt logprobs, reasoning-token counts, LoRA ids and custom logit processors forwarded as the Python servicer forwards them, Embed, FlushCache and profiling (the scheduler's own control requests, carried by the wire's control call) all answer as the Python servicer does. What the wire does not carry is reported, not emulated: multimodal inputs and hidden states are refused, PD/EPD disaggregation stays with the Python implementation (the worker refuses to start on the Rust path in those modes), and LoRA loading answers UNIMPLEMENTED, as it does on the Python servicer. SubscribeKvEvents relays SGLang's ZMQ KV-event publisher, which Rust mode turns on by itself when the launcher is given no --kv-events-config (an explicit one is kept as given; SMG_SGLANG_SERVICER_KV_EVENTS=0 leaves the publisher off). SGLang's HTTP sidecar (metrics and profiling endpoints) is not started on this path. Tuning: SMG_SGLANG_SERVICER_HANDSHAKE_PORT (default: a free port), SMG_SGLANG_SERVICER_DRAIN_SECS (default 5), SMG_SGLANG_SERVICER_STARTUP_TIMEOUT_SECS (default 1800) and SMG_SGLANG_SERVICER_STARTUP_CEILING_SECS (default 14400; the two startup bounds as for vLLM above: the launcher reports the scheduler process alive each time it polls it, so a slow but living start completes, under the ceiling, and 0 lifts the ceiling), SMG_ZMQ_SOCKET_DIR, SMG_SERVICER_WORKER_THREADS.

Headless over ZMQ (no SGLang change)

SMG can drive SGLang's scheduler directly over ZMQ, the same same-host lane it has for vLLM and TokenSpeed, without a gRPC server or SGLang's tokenizer manager in between:

python -m smg_grpc_servicer.sglang.headless \
    --zmq-handshake-address tcp://127.0.0.1:<port> --zmq-engine-index 0 \
    --model-path Qwen/Qwen3-0.6B [any SGLang server args]

smg serve --backend sglang --connection-mode zmq runs exactly this for each worker, with the handshake port SMG derives from the worker's ipc:// URL. The launcher spawns the scheduler ranks the way SGLang's engine does; the scheduler keeps its tokenizer (SMG tokenizes prompts, but SGLang's grammar backend for constrained decoding only exists alongside a tokenizer). Inside each rank, SGLang loads this package's plugin (smg_grpc_servicer.sglang.zmq_plugin, registered under the sglang.srt.plugins entry point and inert unless the launcher's SMG_SGLANG_ZMQ_HANDSHAKE is set), which hooks the scheduler's own ingress and egress seams: the rank that owns request I/O dials SMG's handshake once its model is loaded, registers its geometry, decodes SMG's requests (the scheduler's native TokenizedGenerateReqInput or TokenizedEmbeddingReqInput, normalized and verified as the tokenizer manager would, and control calls that become the scheduler's own FlushCacheReqInput or ProfileReq) and answers with slim positional structs: a per-step token batch (token ids, finish reason with message, matched stop and abort status, counts including reasoning tokens, sampled-token and prompt logprobs with ranked top logprobs when asked, and a scheduler-load tail), an embedding batch, and a control reply under the call id. Requests the scheduler cannot serve, n > 1 for example (SMG fans out itself), are answered with a terminal abort rather than dropped. SGLang itself is unchanged; the plugin pins the wire of the SGLang version it was tested with, and a struct change upstream shows up in grpc_servicer/tests/test_sglang_zmq_msgpack.py.

Multi-node engines: the worker pod's liveness

An engine whose tensor-parallel ranks span pods is launched on every pod with the same flags, --nnodes N --dist-init-addr <the leader>:<port> among them, and the pod's own --node-rank <k>: the gRPC servicer (sglang serve --grpc-mode) or the headless launcher above on the leader pod (--node-rank 0), the scheduler ranks alone on the worker pods. Every rank joins the process group through the TCP store at --dist-init-addr, which rank 0 on the leader pod hosts, and keeps that connection for its lifetime, so the contract of the vLLM section above holds here too: a leader restart must restart its workers, and python -m smg_grpc_servicer.sglang.worker_probe is the worker pod's probe. It passes while at least one TCP connection from the pod to the leader's store port is established and fails otherwise; without arguments it reads the leader and the port off --dist-init-addr on the worker's own command line (/proc/1/cmdline), --leader <name> names the leader by its stable DNS name instead, re-resolved on every run, and --port names the store port. Run it with python3 -I and a timeoutSeconds of 30 or more, as above:

# the worker pod's container; LEADER_HOST is the leader's stable DNS name
startupProbe:
  exec: {command: [/bin/sh, -c, 'python3 -I -m smg_grpc_servicer.sglang.worker_probe --leader "$LEADER_HOST"']}
  periodSeconds: 30
  timeoutSeconds: 30
  failureThreshold: 480
livenessProbe:
  exec: {command: [/bin/sh, -c, 'python3 -I -m smg_grpc_servicer.sglang.worker_probe --leader "$LEADER_HOST"']}
  periodSeconds: 30
  timeoutSeconds: 30
  failureThreshold: 3

Architecture

smg-grpc-servicer[vllm]    ──optional dep──>  vllm       (lazy import)
smg-grpc-servicer[mlx]     ──optional dep──>  mlx-lm     (lazy import)
smg-grpc-servicer          ──external runtime──>  tokenspeed (lazy import)
smg-grpc-servicer[sglang]  ──optional dep──>  sglang     (lazy import)
smg-grpc-servicer          ──depends on────>  smg-grpc-proto  (hard dependency)
vllm                       ──optional──────>  smg-grpc-servicer (via vllm serve --grpc)
sglang                     ──optional──────>  smg-grpc-servicer (via --grpc-mode)

Backend dependencies are isolated via extras or runtime installs to avoid conflicts between vLLM, MLX, TokenSpeed, and SGLang.

Development

See DEVELOPMENT.md for local development setup, CI, and release workflows.