Shepherd Model Gateway
Engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments. SMG centralizes worker lifecycle management, balances traffic across self-hosted engines and cloud providers, and gives you enterprise-grade control over multi-tenancy, chat-history storage, MCP tooling, and observability — behind one unified endpoint.
Why SMG?
| 🚀 Maximize GPU Utilization | Cache-aware routing tracks each worker's KV-cache state in radix trees to reuse prefixes across SGLang, vLLM, TensorRT-LLM, TokenSpeed, and MLX — with load modeling that accounts for queued token work and KV pressure. |
| 🔌 One API, Any Backend | Route to self-hosted engines over HTTP or gRPC, or to OpenAI, Anthropic, Gemini, and xAI — plus any OpenAI-compatible endpoint — through a single unified gateway. |
| ⚡ Built for Speed | Native Rust with streaming gRPC pipelines, cached tokenization with zero-copy cache hits, prefill/decode disaggregation (including a separate encode stage for vision), and DP-aware routing for data-parallel engines. |
| 🔒 Enterprise Control | Priority admission scheduling with preemption and per-tenant controls, API-key auth with OIDC on the control plane, WebAssembly plugins for custom logic, and chat history that never leaves your infrastructure. |
| 📊 Full Observability | 90+ Prometheus metrics, OpenTelemetry tracing with W3C trace context propagated into the engines over both HTTP and gRPC, and structured JSON logs with request correlation. |
API Coverage: OpenAI Chat Completions, Completions, Embeddings, Rerank, and Classify; Responses and Conversations APIs for agents; Anthropic Messages; Gemini Interactions; Realtime over WebSocket and WebRTC; audio transcription; tokenize/detokenize; and MCP tool execution with approval policies in the Responses and Messages APIs.
Quick Start
Install — pick your preferred method:
# Docker
docker pull lightseekorg/smg:latest
# Kubernetes (Helm)
helm install smg oci://ghcr.io/smg-project/charts/smg
# Python
pip install smg
# Rust (needs protoc)
cargo install smg
Run — point SMG at your inference workers:
# Single worker
smg launch --worker-urls http://localhost:8000
# Multiple workers with cache-aware routing
smg launch --worker-urls http://gpu1:8000 http://gpu2:8000 --policy cache_aware
# With high availability mesh
smg launch --worker-urls http://gpu1:8000 --enable-mesh \
--mesh-advertise-host 10.0.0.1 --mesh-peer-urls 10.0.0.2:39527
Use — send requests to the gateway:
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "llama3", "messages": [{"role": "user", "content": "Hello!"}]}'
That's it. SMG is now load-balancing requests across your workers.
Supported Backends
| Self-Hosted Engines | vLLM · SGLang · TokenSpeed · TensorRT-LLM · MLX (Apple Silicon) · any OpenAI-compatible server (e.g. Ollama) |
| Cloud Providers | OpenAI · Anthropic · Google Gemini · xAI · OCI Generative AI · AWS Bedrock · Azure OpenAI · any OpenAI-compatible provider (Groq, Together, …) |
Features
| Feature | Description |
|---|---|
| 10 Routing Policies | cache_aware, least_load, power_of_two, consistent_hashing, prefix_hash, bucket, round_robin, random, manual, passthrough |
| gRPC Pipeline | Native streaming gRPC to the engines with prefill/decode and encode disaggregation and DP-aware routing |
| Kubernetes Discovery | Native pod watchers with label selectors, per-role prefill/decode/encode selectors, and router peer discovery |
| Model Parsers | 21 tool-call parsers and 16 reasoning parsers with automatic model detection — DeepSeek, Qwen, Kimi, GLM, Llama, Mistral, Command, Nemotron, and more |
| MCP Integration | Tool discovery and execution over stdio, SSE, and streamable HTTP, with approval policies and audit logging |
| High Availability | Mesh networking with SWIM gossip and CRDT-replicated state for multi-node deployments |
| Chat History | Pluggable storage with schema migrations: PostgreSQL, Oracle, Redis, or in-memory |
| WASM Plugins | Extend request and response handling with custom WebAssembly middleware |
| Resilience | Circuit breakers, retries with backoff and jitter, rate limiting, and priority admission scheduling |
Documentation
Full documentation lives at lightseek.org/smg.
| Getting Started | Installation and first steps |
| Architecture | How SMG works |
| Configuration | CLI reference and options |
| API Reference | OpenAI-compatible endpoints |
| Kubernetes Setup | In-cluster discovery and production setup |
Contributing
We welcome contributions! See the Contributing Guide for details.
1 maintainer
Release files for tokenspeed-smg 1.9.0.post20260803
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| tokenspeed_smg-1.9.0.post20260803.tar.gz | 3.3 MB | Details |
Built distributions (wheels)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| tokenspeed_smg-1.9.0.post20260803-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | CPython 3.8 | abi3 | Linux glibc 2.17+ x86-64 | Details |
| tokenspeed_smg-1.9.0.post20260803-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl | CPython 3.8 | abi3 | Linux glibc 2.17+ ARM64 | Details |
Total release size: 67.2 MB
Release files / tokenspeed_smg-1.9.0.post20260803.tar.gz
| Download URL | tokenspeed_smg-1.9.0.post20260803.tar.gz |
|---|---|
| Size | 3.3 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2d8e07a087ee5f8d28b467a629208a548baeb5837bd02e636dca25ab5e879ade
|
|
BLAKE2b-256 checksum How to use checksums |
08d4ab45c4b3c6ce95098dd0eef0c74c05c1955cd5308976f4dd5268c9430d2f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 3, 2026.
Transparency logRelease files / tokenspeed_smg-1.9.0.post20260803-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | tokenspeed_smg-1.9.0.post20260803-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 31.2 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
b74485c70fdd3b44b3ee4a133446e2b8e91ff9c848b013d9d244ed5ca3848c05
|
|
BLAKE2b-256 checksum How to use checksums |
fc0cb8d7de84574f78b2275a6787b69de16315c94bd7ff8b76cda1e12849585b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 3, 2026.
Transparency logRelease files / tokenspeed_smg-1.9.0.post20260803-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | tokenspeed_smg-1.9.0.post20260803-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 32.8 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
c4c390a2f5fc43d26b4f323028c03d00e3362fc76d0b09dec9aadd099f8e7b09
|
|
BLAKE2b-256 checksum How to use checksums |
301cc257725e0803eaf731777011507a81c92c72f824979b051676da640e285c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 3, 2026.
Transparency log
