Shepherd Model Gateway
Engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments. Centralizes worker lifecycle management, balances traffic across HTTP/gRPC/OpenAI-compatible backends, and provides enterprise-ready control over history storage, MCP tooling, and privacy-sensitive workflows.
Why SMG?
| 🚀 Maximize GPU Utilization | Cache-aware routing understands your inference engine's KV cache state—whether vLLM, TensorRT-LLM, TokenSpeed, or SGLang—to reuse prefixes and reduce redundant computation. |
| 🔌 One API, Any Backend | Route to self-hosted models (vLLM, TensorRT-LLM, TokenSpeed, SGLang) or cloud providers (OpenAI, Anthropic, Gemini, Bedrock, and more) through a single unified endpoint. |
| ⚡ Built for Speed | Native Rust with gRPC pipelines, sub-millisecond routing decisions, and zero-copy tokenization. Circuit breakers and automatic failover keep things running. |
| 🔒 Enterprise Control | Multi-tenant rate limiting with OIDC, WebAssembly plugins for custom logic, and a privacy boundary that keeps conversation history within your infrastructure. |
| 📊 Full Observability | 40+ Prometheus metrics, OpenTelemetry tracing, and structured JSON logs with request correlation—know exactly what's happening at every layer. |
API Coverage: OpenAI Chat/Completions/Embeddings, Responses API for agents, Anthropic Messages, and MCP tool execution.
Quick Start
Install — pick your preferred method:
# Docker
docker pull lightseekorg/smg:latest
# Python
pip install smg
# Rust
cargo install smg
Run — point SMG at your inference workers:
# Single worker
smg launch --worker-urls http://localhost:8000
# Multiple workers with cache-aware routing
smg launch --worker-urls http://gpu1:8000 http://gpu2:8000 --policy cache_aware
# With high availability mesh
smg launch --worker-urls http://gpu1:8000 --enable-mesh \
--mesh-advertise-host 10.0.0.1 --mesh-peer-urls 10.0.0.2:39527
Use — send requests to the gateway:
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "llama3", "messages": [{"role": "user", "content": "Hello!"}]}'
That's it. SMG is now load-balancing requests across your workers.
Supported Backends
| Self-Hosted | Cloud Providers |
|---|---|
| vLLM | OpenAI |
| TensorRT-LLM | Anthropic |
| TokenSpeed | Google Gemini |
| SGLang | AWS Bedrock |
| Ollama | Azure OpenAI |
| Any OpenAI-compatible server | Any OpenAI-compatible provider |
Features
| Feature | Description |
|---|---|
| 8 Routing Policies | cache_aware, round_robin, power_of_two, consistent_hashing, prefix_hash, manual, random, bucket |
| gRPC Pipeline | Native gRPC with streaming, reasoning extraction, and tool call parsing |
| MCP Integration | Connect external tool servers via Model Context Protocol |
| High Availability | Mesh networking with SWIM protocol for multi-node deployments |
| Chat History | Pluggable storage: PostgreSQL, Oracle, Redis, or in-memory |
| WASM Plugins | Extend with custom WebAssembly logic |
| Resilience | Circuit breakers, retries with backoff, rate limiting |
Documentation
| Getting Started | Installation and first steps |
| Architecture | How SMG works |
| Configuration | CLI reference and options |
| API Reference | OpenAI-compatible endpoints |
| Kubernetes Setup | In-cluster discovery and production setup |
Contributing
We welcome contributions! See Contributing Guide for details.
1 maintainer
Release files for tokenspeed-smg 1.8.0.post20260727
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| tokenspeed_smg-1.8.0.post20260727.tar.gz | 3.1 MB | Details |
Built distributions (wheels)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| tokenspeed_smg-1.8.0.post20260727-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | CPython 3.8 | abi3 | Linux glibc 2.17+ x86-64 | Details |
| tokenspeed_smg-1.8.0.post20260727-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl | CPython 3.8 | abi3 | Linux glibc 2.17+ ARM64 | Details |
Total release size: 66.0 MB
Release files / tokenspeed_smg-1.8.0.post20260727.tar.gz
| Download URL | tokenspeed_smg-1.8.0.post20260727.tar.gz |
|---|---|
| Size | 3.1 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
03db63a66caed7133882037a2841c94462b86900f04f644f1fbfaf0727ff6a21
|
|
BLAKE2b-256 checksum How to use checksums |
3ea2146ae2b7b0c1259fd677a112767908f8817e5c4814238ec2ea436f81a7c4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 27, 2026.
Transparency logRelease files / tokenspeed_smg-1.8.0.post20260727-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | tokenspeed_smg-1.8.0.post20260727-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 30.7 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
072372af166355f1d2be177f429fa5933d95973d7d9215e44237e42f35762d09
|
|
BLAKE2b-256 checksum How to use checksums |
53ee01619ac1e317ac928dd7355a871b7645785fd514c145b743c58950166a4f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 27, 2026.
Transparency logRelease files / tokenspeed_smg-1.8.0.post20260727-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | tokenspeed_smg-1.8.0.post20260727-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 32.2 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
0bef4c524dc8dfb56c1cd7fe311053db528d3953a75b18e1c596829e862dd2ef
|
|
BLAKE2b-256 checksum How to use checksums |
5c117139a7eaeda492d7724a16d07f50001385b97271cd99f750798203a66762
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 27, 2026.
Transparency log
