tokenspeed-smg 1.9.0.post20260803

High-performance Rust-based inference gateway for large-scale LLM deployments

Shepherd Model Gateway

Engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments. SMG centralizes worker lifecycle management, balances traffic across self-hosted engines and cloud providers, and gives you enterprise-grade control over multi-tenancy, chat-history storage, MCP tooling, and observability — behind one unified endpoint.

Why SMG?

🚀 Maximize GPU Utilization	Cache-aware routing tracks each worker's KV-cache state in radix trees to reuse prefixes across SGLang, vLLM, TensorRT-LLM, TokenSpeed, and MLX — with load modeling that accounts for queued token work and KV pressure.
🔌 One API, Any Backend	Route to self-hosted engines over HTTP or gRPC, or to OpenAI, Anthropic, Gemini, and xAI — plus any OpenAI-compatible endpoint — through a single unified gateway.
⚡ Built for Speed	Native Rust with streaming gRPC pipelines, cached tokenization with zero-copy cache hits, prefill/decode disaggregation (including a separate encode stage for vision), and DP-aware routing for data-parallel engines.
🔒 Enterprise Control	Priority admission scheduling with preemption and per-tenant controls, API-key auth with OIDC on the control plane, WebAssembly plugins for custom logic, and chat history that never leaves your infrastructure.
📊 Full Observability	90+ Prometheus metrics, OpenTelemetry tracing with W3C trace context propagated into the engines over both HTTP and gRPC, and structured JSON logs with request correlation.

API Coverage: OpenAI Chat Completions, Completions, Embeddings, Rerank, and Classify; Responses and Conversations APIs for agents; Anthropic Messages; Gemini Interactions; Realtime over WebSocket and WebRTC; audio transcription; tokenize/detokenize; and MCP tool execution with approval policies in the Responses and Messages APIs.

Quick Start

Install — pick your preferred method:

# Docker
docker pull lightseekorg/smg:latest

# Kubernetes (Helm)
helm install smg oci://ghcr.io/smg-project/charts/smg

# Python
pip install smg

# Rust (needs protoc)
cargo install smg

Run — point SMG at your inference workers:

# Single worker
smg launch --worker-urls http://localhost:8000

# Multiple workers with cache-aware routing
smg launch --worker-urls http://gpu1:8000 http://gpu2:8000 --policy cache_aware

# With high availability mesh
smg launch --worker-urls http://gpu1:8000 --enable-mesh \
  --mesh-advertise-host 10.0.0.1 --mesh-peer-urls 10.0.0.2:39527

Use — send requests to the gateway:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3", "messages": [{"role": "user", "content": "Hello!"}]}'

That's it. SMG is now load-balancing requests across your workers.

Supported Backends

Self-Hosted Engines	vLLM · SGLang · TokenSpeed · TensorRT-LLM · MLX (Apple Silicon) · any OpenAI-compatible server (e.g. Ollama)
Cloud Providers	OpenAI · Anthropic · Google Gemini · xAI · OCI Generative AI · AWS Bedrock · Azure OpenAI · any OpenAI-compatible provider (Groq, Together, …)

Features

Feature	Description
10 Routing Policies	cache_aware, least_load, power_of_two, consistent_hashing, prefix_hash, bucket, round_robin, random, manual, passthrough
gRPC Pipeline	Native streaming gRPC to the engines with prefill/decode and encode disaggregation and DP-aware routing
Kubernetes Discovery	Native pod watchers with label selectors, per-role prefill/decode/encode selectors, and router peer discovery
Model Parsers	21 tool-call parsers and 16 reasoning parsers with automatic model detection — DeepSeek, Qwen, Kimi, GLM, Llama, Mistral, Command, Nemotron, and more
MCP Integration	Tool discovery and execution over stdio, SSE, and streamable HTTP, with approval policies and audit logging
High Availability	Mesh networking with SWIM gossip and CRDT-replicated state for multi-node deployments
Chat History	Pluggable storage with schema migrations: PostgreSQL, Oracle, Redis, or in-memory
WASM Plugins	Extend request and response handling with custom WebAssembly middleware
Resilience	Circuit breakers, retries with backoff and jitter, rate limiting, and priority admission scheduling

Documentation

Full documentation lives at lightseek.org/smg.

Getting Started	Installation and first steps
Architecture	How SMG works
Configuration	CLI reference and options
API Reference	OpenAI-compatible endpoints
Kubernetes Setup	In-cluster discovery and production setup

Contributing

We welcome contributions! See the Contributing Guide for details.

• Development Setup
• Code Style