smg-grpc-servicer
gRPC servicer implementations for LLM inference engines. Supports vLLM, MLX, TokenSpeed, and SGLang.
Installation
For vLLM:
pip install smg-grpc-servicer[vllm]
For MLX:
pip install smg-grpc-servicer[mlx]
For TokenSpeed, install the TokenSpeed runtime first, then install the servicer bridge:
pip install smg-grpc-servicer
For SGLang:
pip install smg-grpc-servicer[sglang]
Usage
vLLM
vllm serve meta-llama/Llama-2-7b-hf --grpc
Worker-side multimodal processing (media refs)
By default the smg router fetches and preprocesses images itself and sends pixel tensors. A vLLM gRPC worker can instead accept media references (URLs) and run vLLM's own multimodal processor:
SMG_VLLM_MM_PROCESSOR=inprocess vllm serve Qwen/Qwen3-VL-8B-Instruct --grpc \
--allowed-media-domains example.com
The worker then advertises mm_processor=inprocess and mm_media_ref_schemes
through GetServerInfo; a router with media-reference support forwards
media_refs only to workers that advertise, and a router without it ignores the
labels and keeps sending preprocessed tensors. vLLM's --allowed-media-domains,
--allowed-local-media-path, --media-io-kwargs, --limit-mm-per-prompt and
VLLM_*_FETCH_TIMEOUT govern fetching on the worker; without
--allowed-media-domains the worker fetches from any host the router forwards.
Related knobs: SMG_VLLM_MM_MAX_INFLIGHT (default 64) bounds concurrent media
jobs; SMG_VLLM_MM_MAX_ITEMS (default 16) caps references per request;
SMG_VLLM_MM_MAX_ITEM_BYTES (default 32 MiB) caps inline data: payloads.
On the router side, SMG_MM_PROCESSING selects auto (default: forward when
the model's spec opts in and every registered worker of the model advertises
mm_processor), router (always preprocess) or worker (strict: 400 when a
request cannot be forwarded); the outcome is counted in
smg_mm_processing_total{model,mode,reason}. It is read from the router's
environment only and has no router-config equivalent. Any other value stops the
router at startup instead of quietly reverting to auto. On the worker path the
router never expands placeholders, so routing decisions that weigh the prompt's token
count (cache-aware policies, load estimates) see one token per media item where
the worker will schedule the full placeholder run. The E2E_MM_PROCESSING=worker
e2e lanes run the multimodal suites in this mode, and
crates/multimodal/scripts/check_worker_anchor_parity.py checks that a spec's
anchor is the token vLLM expands.
To move fetching and processing out of the vLLM process, run the GPU-free
sidecar next to a private Redis and point the worker at it
(pip install smg-grpc-servicer[vllm,vllm-redis]):
python -m smg_grpc_servicer.vllm.mm_sidecar --model Qwen/Qwen3-VL-8B-Instruct \
--redis-url redis://127.0.0.1:6379/0 --allowed-media-domains example.com
SMG_VLLM_MM_PROCESSOR=redis SMG_VLLM_MM_REDIS_URL=redis://127.0.0.1:6379/0 \
vllm serve Qwen/Qwen3-VL-8B-Instruct --grpc
The sidecar and the worker must agree on model, vLLM version, dtype, video
backend, media/processor kwargs and --limit-mm-per-prompt (pass the flag to
both processes; the sidecar's limit is the one that applies, the limit is
resolved per modality before hashing so equivalent spellings match, and the key
namespace is derived from all of these): the worker advertises
mm_processor=redis only while a sidecar with a matching fingerprint keeps its
hello key alive, and rejects results that disagree. Jobs and results travel over Redis lists under
smg:mm:v1:{namespace}; results carry full tensors keyed by a per-attempt job
id and expire after 120 s. Knobs: SMG_VLLM_MM_SIDECAR_TIMEOUT_MS (30000),
SMG_VLLM_MM_SIDECAR_MAX_QUEUE (256, fail fast when the queue is deeper),
SMG_VLLM_MM_SIDECAR_NAMESPACE (override the derived namespace).
MLX
python -m smg_grpc_servicer.mlx --model meta-llama/Llama-2-7b-hf --host 0.0.0.0 --port 50051
TokenSpeed
python -m smg_grpc_servicer.tokenspeed --model meta-llama/Llama-2-7b-hf --host 0.0.0.0 --port 50051
SGLang
sglang serve --model-path meta-llama/Llama-2-7b-hf --grpc-mode
Architecture
smg-grpc-servicer[vllm] ──optional dep──> vllm (lazy import)
smg-grpc-servicer[mlx] ──optional dep──> mlx-lm (lazy import)
smg-grpc-servicer ──external runtime──> tokenspeed (lazy import)
smg-grpc-servicer[sglang] ──optional dep──> sglang (lazy import)
smg-grpc-servicer ──depends on────> smg-grpc-proto (hard dependency)
vllm ──optional──────> smg-grpc-servicer (via vllm serve --grpc)
sglang ──optional──────> smg-grpc-servicer (via --grpc-mode)
Backend dependencies are isolated via extras or runtime installs to avoid conflicts between vLLM, MLX, TokenSpeed, and SGLang.
Development
See DEVELOPMENT.md for local development setup, CI, and release workflows.
1 maintainer
Release files for tokenspeed-smg-grpc-servicer 0.12.0.post20260922
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| tokenspeed_smg_grpc_servicer-0.12.0.post20260922.tar.gz | 171.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| tokenspeed_smg_grpc_servicer-0.12.0.post20260922-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 319.2 kB
Release files / tokenspeed_smg_grpc_servicer-0.12.0.post20260922.tar.gz
| Download URL | tokenspeed_smg_grpc_servicer-0.12.0.post20260922.tar.gz |
|---|---|
| Size | 171.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ab896b6bbc275d4152e385faf0a81c39719ff7fa2050d063c9664f002d946ca3
|
|
BLAKE2b-256 checksum How to use checksums |
c2a5a987859464b31654da67d16099033ab075386c25edc406afec0884a11396
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 22, 2026.
Transparency logRelease files / tokenspeed_smg_grpc_servicer-0.12.0.post20260922-py3-none-any.whl
| Download URL | tokenspeed_smg_grpc_servicer-0.12.0.post20260922-py3-none-any.whl |
|---|---|
| Size | 147.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a25fec1d816235c033207f32f4f02187cb9dfa8dcad3285d47c924cc28244b81
|
|
BLAKE2b-256 checksum How to use checksums |
4d2898c3ded305a6e542a5534e7aaba672052df2e9970408ffa4af3e067a0c6a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 22, 2026.
Transparency log
