smg-grpc-servicer

gRPC servicer implementations for LLM inference engines. Supports vLLM, MLX, TokenSpeed, and SGLang.

Installation

For vLLM:

pip install smg-grpc-servicer[vllm]

For MLX:

pip install smg-grpc-servicer[mlx]

For TokenSpeed, install the TokenSpeed runtime first, then install the servicer bridge:

pip install smg-grpc-servicer

For SGLang:

pip install smg-grpc-servicer[sglang]

Usage

vLLM

vllm serve meta-llama/Llama-2-7b-hf --grpc

Worker-side multimodal processing (media refs)

By default the smg router fetches and preprocesses images itself and sends pixel tensors. A vLLM gRPC worker can instead accept media references (URLs) and run vLLM's own multimodal processor:

SMG_VLLM_MM_PROCESSOR=inprocess vllm serve Qwen/Qwen3-VL-8B-Instruct --grpc \
    --allowed-media-domains example.com

The worker then advertises mm_processor=inprocess and mm_media_ref_schemes through GetServerInfo; a router with media-reference support forwards media_refs only to workers that advertise, and a router without it ignores the labels and keeps sending preprocessed tensors. vLLM's --allowed-media-domains, --allowed-local-media-path, --media-io-kwargs, --limit-mm-per-prompt and VLLM_*_FETCH_TIMEOUT govern fetching on the worker; without --allowed-media-domains the worker fetches from any host the router forwards. Related knobs: SMG_VLLM_MM_MAX_INFLIGHT (default 64) bounds concurrent media jobs; SMG_VLLM_MM_MAX_ITEMS (default 16) caps references per request; SMG_VLLM_MM_MAX_ITEM_BYTES (default 32 MiB) caps inline data: payloads.

On the router side, SMG_MM_PROCESSING selects auto (default: forward when the model's spec opts in and every registered worker of the model advertises mm_processor), router (always preprocess) or worker (strict: 400 when a request cannot be forwarded); the outcome is counted in smg_mm_processing_total{model,mode,reason}. It is read from the router's environment only and has no router-config equivalent. Any other value stops the router at startup instead of quietly reverting to auto. On the worker path the router never expands placeholders, so routing decisions that weigh the prompt's token count (cache-aware policies, load estimates) see one token per media item where the worker will schedule the full placeholder run. The E2E_MM_PROCESSING=worker e2e lanes run the multimodal suites in this mode, and crates/multimodal/scripts/check_worker_anchor_parity.py checks that a spec's anchor is the token vLLM expands.

To move fetching and processing out of the vLLM process, run the GPU-free sidecar next to a private Redis and point the worker at it (pip install smg-grpc-servicer[vllm,vllm-redis]):

python -m smg_grpc_servicer.vllm.mm_sidecar --model Qwen/Qwen3-VL-8B-Instruct \
    --redis-url redis://127.0.0.1:6379/0 --allowed-media-domains example.com
SMG_VLLM_MM_PROCESSOR=redis SMG_VLLM_MM_REDIS_URL=redis://127.0.0.1:6379/0 \
    vllm serve Qwen/Qwen3-VL-8B-Instruct --grpc

The sidecar and the worker must agree on model, vLLM version, dtype, video backend, media/processor kwargs and --limit-mm-per-prompt (pass the flag to both processes; the sidecar's limit is the one that applies, the limit is resolved per modality before hashing so equivalent spellings match, and the key namespace is derived from all of these): the worker advertises mm_processor=redis only while a sidecar with a matching fingerprint keeps its hello key alive, and rejects results that disagree. Jobs and results travel over Redis lists under smg:mm:v1:{namespace}; results carry full tensors keyed by a per-attempt job id and expire after 120 s. Knobs: SMG_VLLM_MM_SIDECAR_TIMEOUT_MS (30000), SMG_VLLM_MM_SIDECAR_MAX_QUEUE (256, fail fast when the queue is deeper), SMG_VLLM_MM_SIDECAR_NAMESPACE (override the derived namespace).

MLX

python -m smg_grpc_servicer.mlx --model meta-llama/Llama-2-7b-hf --host 0.0.0.0 --port 50051

TokenSpeed

python -m smg_grpc_servicer.tokenspeed --model meta-llama/Llama-2-7b-hf --host 0.0.0.0 --port 50051

SGLang

sglang serve --model-path meta-llama/Llama-2-7b-hf --grpc-mode

Architecture

smg-grpc-servicer[vllm]    ──optional dep──>  vllm       (lazy import)
smg-grpc-servicer[mlx]     ──optional dep──>  mlx-lm     (lazy import)
smg-grpc-servicer          ──external runtime──>  tokenspeed (lazy import)
smg-grpc-servicer[sglang]  ──optional dep──>  sglang     (lazy import)
smg-grpc-servicer          ──depends on────>  smg-grpc-proto  (hard dependency)
vllm                       ──optional──────>  smg-grpc-servicer (via vllm serve --grpc)
sglang                     ──optional──────>  smg-grpc-servicer (via --grpc-mode)

Backend dependencies are isolated via extras or runtime installs to avoid conflicts between vLLM, MLX, TokenSpeed, and SGLang.

Development

See DEVELOPMENT.md for local development setup, CI, and release workflows.