SMG Python Bindings
This directory contains the Python bindings for SMG (Shepherd Model Gateway), built using maturin and PyO3.
Quick Start
Installation
pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl
Usage
The smg serve command launches backend workers and the SMG router in a single command:
# sglang with gRPC (default)
smg serve --backend sglang --model-path /path/to/model --port 8080
# sglang with HTTP
smg serve --backend sglang --model-path /path/to/model --port 8080 --connection-mode http
# vLLM (gRPC only)
smg serve --backend vllm --model /path/to/model --port 8080
# TensorRT-LLM (gRPC only)
smg serve --backend trtllm --model /path/to/model --port 8080
# Multiple workers (data parallel)
smg serve --backend sglang --model-path /path/to/model --port 8080 --dp-size 4
Serve Options
| Option | Default | Description |
|---|---|---|
--backend |
sglang |
Backend to use: sglang, vllm, or trtllm |
--connection-mode |
grpc |
Connection mode: grpc, http, or zmq. trtllm only supports grpc; tokenspeed only zmq; zmq (a same-host direct engine connection) is available for vllm, tokenspeed, and sglang |
--host |
127.0.0.1 |
Host for the router |
--port |
8080 |
Port for the router |
--dp-size |
1 |
Data parallel size (number of worker replicas) |
--worker-host |
127.0.0.1 |
Host for worker processes |
--worker-base-port |
31000 |
Base port for workers |
--worker-startup-timeout |
300 |
Seconds to wait for workers to become healthy |
Backend-specific options (e.g., --tensor-parallel-size, --quantization) are passed through to the backend.
Directory Structure
bindings/python/
├── src/ # Source code (src layout)
│ ├── lib.rs # Rust/PyO3 bindings implementation
│ └── smg/ # Python source code
│ ├── __init__.py
│ ├── cli.py # CLI entry point
│ ├── serve.py # smg serve implementation
│ ├── launch_router.py
│ ├── router.py
│ └── router_args.py
├── tests/ # Python unit tests
├── Cargo.toml # Rust package configuration
├── pyproject.toml # Python package configuration
└── README.md # This file
Building
Development Build
pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl
Production Build
cd smg/bindings/python
maturin build --release --out dist --features vendored-openssl
pip install dist/smg-*.whl
Testing
cd smg/bindings/python
pytest tests/
Release files for smg 1.12.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| smg-1.12.0.tar.gz | 5.5 MB | Details |
Built distributions (wheels)
| File | Reset | |||
|---|---|---|---|---|
| smg-1.12.0-cp38-abi3-win_amd64.whl | CPython 3.8 | abi3 | Windows x86-64 | Details |
| smg-1.12.0-cp38-abi3-musllinux_1_1_x86_64.whl | CPython 3.8 | abi3 | Linux musl 1.1+ x86-64 | Details |
| smg-1.12.0-cp38-abi3-musllinux_1_1_aarch64.whl | CPython 3.8 | abi3 | Linux musl 1.1+ ARM64 | Details |
| smg-1.12.0-cp38-abi3-manylinux_2_28_aarch64.whl | CPython 3.8 | abi3 | Linux glibc 2.28+ ARM64 | Details |
| smg-1.12.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | CPython 3.8 | abi3 | Linux glibc 2.17+ x86-64 | Details |
| smg-1.12.0-cp38-abi3-macosx_11_0_arm64.whl | CPython 3.8 | abi3 | macOS 11.0+ ARM64 | Details |
| smg-1.12.0-cp38-abi3-macosx_10_12_x86_64.whl | CPython 3.8 | abi3 | macOS 10.12+ x86-64 | Details |
Total release size: 247.8 MB
Release files / smg-1.12.0.tar.gz
| Download URL | smg-1.12.0.tar.gz |
|---|---|
| Size | 5.5 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9546e3121a16ad2079cede9e565fa63a3a21894ad7747dde610f2ee56c3b1967
|
|
BLAKE2b-256 checksum How to use checksums |
a7c360089e117a5145b1c36abb532db82d65ce5bdabd4be57b681dcd4d086b22
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / smg-1.12.0-cp38-abi3-win_amd64.whl
| Download URL | smg-1.12.0-cp38-abi3-win_amd64.whl |
|---|---|
| Size | 31.5 MB |
| Tags | CPython 3.8 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
2ef36deb6de2581507224fefd5e05907ec7b5b55b2884114348a3e08e074afc2
|
|
BLAKE2b-256 checksum How to use checksums |
95acc7b50996c450fc7935a03382fb91fd1feacf7b4b3d55bcd9ba3754d97dd8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / smg-1.12.0-cp38-abi3-musllinux_1_1_x86_64.whl
| Download URL | smg-1.12.0-cp38-abi3-musllinux_1_1_x86_64.whl |
|---|---|
| Size | 34.6 MB |
| Tags | CPython 3.8 Linux musl 1.1+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
ca17992e996500592933b7f673c19073524ef54524cbabcbeb164dfa8abc1d6d
|
|
BLAKE2b-256 checksum How to use checksums |
437b72a2a1dfed728efce73595d292bca54aad0e8411b4e11eb10fabb777aba0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / smg-1.12.0-cp38-abi3-musllinux_1_1_aarch64.whl
| Download URL | smg-1.12.0-cp38-abi3-musllinux_1_1_aarch64.whl |
|---|---|
| Size | 36.2 MB |
| Tags | CPython 3.8 Linux musl 1.1+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
5fb90af40f93022ea0ddc2600e94cac83dd663948275ac4b9828422db111b84e
|
|
BLAKE2b-256 checksum How to use checksums |
2616333e19d5fca243ed92500c7c2ec55584a7cddc01e7a398186fc58e9b9fd3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / smg-1.12.0-cp38-abi3-manylinux_2_28_aarch64.whl
| Download URL | smg-1.12.0-cp38-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 37.7 MB |
| Tags | CPython 3.8 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
36d44a023ce4d11acb75ce93e6308965d8b208b75596efe522f89a0e7be9900c
|
|
BLAKE2b-256 checksum How to use checksums |
20559a2f4ffd7c8b3ca364deb5f3aa1bd565f920bdfee1cf7517ff1893f52379
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / smg-1.12.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | smg-1.12.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 36.2 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
51a0da56293a9f0aab20788e665304c9e84a62a6bab2bb096c5b4014b6e5a1e2
|
|
BLAKE2b-256 checksum How to use checksums |
0ebd2331f7ebf1fb9d91df8a1756db0212892afaf146795b61c87ac80509ef98
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / smg-1.12.0-cp38-abi3-macosx_11_0_arm64.whl
| Download URL | smg-1.12.0-cp38-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 33.5 MB |
| Tags | CPython 3.8 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
38dcbcd060ce4f062c520abf12fbb39cabd391b51357953a916d5558db541e2f
|
|
BLAKE2b-256 checksum How to use checksums |
5cc50e27a359733a02caa2113a739966d18ec49418f469de7191ea86d58d5f9d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / smg-1.12.0-cp38-abi3-macosx_10_12_x86_64.whl
| Download URL | smg-1.12.0-cp38-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 32.7 MB |
| Tags | CPython 3.8 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
49f26f884361a5b45ad0f895e0d300e71352b93f196fb63e8ea70c4c7e2289e5
|
|
BLAKE2b-256 checksum How to use checksums |
606ee0c98aa9293909729e641ba50f92df59a2a0a9a52405b054b2cff0c2340e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
