SMG Python Bindings
This directory contains the Python bindings for SMG (Shepherd Model Gateway), built using maturin and PyO3.
Quick Start
Installation
pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl
Usage
The smg serve command launches backend workers and the SMG router in a single command:
# sglang with gRPC (default)
smg serve --backend sglang --model-path /path/to/model --port 8080
# sglang with HTTP
smg serve --backend sglang --model-path /path/to/model --port 8080 --connection-mode http
# vLLM (gRPC only)
smg serve --backend vllm --model /path/to/model --port 8080
# TensorRT-LLM (gRPC only)
smg serve --backend trtllm --model /path/to/model --port 8080
# Multiple workers (data parallel)
smg serve --backend sglang --model-path /path/to/model --port 8080 --dp-size 4
Serve Options
| Option | Default | Description |
|---|---|---|
--backend |
sglang |
Backend to use: sglang, vllm, or trtllm |
--connection-mode |
grpc |
Connection mode: grpc or http. vllm/trtllm only support grpc |
--host |
127.0.0.1 |
Host for the router |
--port |
8080 |
Port for the router |
--dp-size |
1 |
Data parallel size (number of worker replicas) |
--worker-host |
127.0.0.1 |
Host for worker processes |
--worker-base-port |
31000 |
Base port for workers |
--worker-startup-timeout |
300 |
Seconds to wait for workers to become healthy |
Backend-specific options (e.g., --tensor-parallel-size, --quantization) are passed through to the backend.
Directory Structure
bindings/python/
├── src/ # Source code (src layout)
│ ├── lib.rs # Rust/PyO3 bindings implementation
│ └── smg/ # Python source code
│ ├── __init__.py
│ ├── cli.py # CLI entry point
│ ├── serve.py # smg serve implementation
│ ├── launch_router.py
│ ├── router.py
│ └── router_args.py
├── tests/ # Python unit tests
├── Cargo.toml # Rust package configuration
├── pyproject.toml # Python package configuration
└── README.md # This file
Building
Development Build
pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl
Production Build
cd smg/bindings/python
maturin build --release --out dist --features vendored-openssl
pip install dist/smg-*.whl
Testing
cd smg/bindings/python
pytest tests/
Release files for smg 1.10.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| smg-1.10.1.tar.gz | 3.6 MB | Details |
Built distributions (wheels)
| File | Reset | |||
|---|---|---|---|---|
| smg-1.10.1-cp38-abi3-win_amd64.whl | CPython 3.8 | abi3 | Windows x86-64 | Details |
| smg-1.10.1-cp38-abi3-musllinux_1_1_x86_64.whl | CPython 3.8 | abi3 | Linux musl 1.1+ x86-64 | Details |
| smg-1.10.1-cp38-abi3-musllinux_1_1_aarch64.whl | CPython 3.8 | abi3 | Linux musl 1.1+ ARM64 | Details |
| smg-1.10.1-cp38-abi3-manylinux_2_28_aarch64.whl | CPython 3.8 | abi3 | Linux glibc 2.28+ ARM64 | Details |
| smg-1.10.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | CPython 3.8 | abi3 | Linux glibc 2.17+ x86-64 | Details |
| smg-1.10.1-cp38-abi3-macosx_11_0_arm64.whl | CPython 3.8 | abi3 | macOS 11.0+ ARM64 | Details |
| smg-1.10.1-cp38-abi3-macosx_10_12_x86_64.whl | CPython 3.8 | abi3 | macOS 10.12+ x86-64 | Details |
Total release size: 224.3 MB
Release files / smg-1.10.1.tar.gz
| Download URL | smg-1.10.1.tar.gz |
|---|---|
| Size | 3.6 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
03d296006a1caafc94ded13765a0f358903ea79b2a14f58f9d8cba6b61fa7a0a
|
|
BLAKE2b-256 checksum How to use checksums |
b2896ad196b2774751097e502289a3e327f03565e2746eb81c560f34421ab236
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.10.1-cp38-abi3-win_amd64.whl
| Download URL | smg-1.10.1-cp38-abi3-win_amd64.whl |
|---|---|
| Size | 28.3 MB |
| Tags | CPython 3.8 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
89b36d1c325cc47c7a903d7b2f069d4a9f5e68d3f2ba68d7dab61699d8631e4f
|
|
BLAKE2b-256 checksum How to use checksums |
f92fa30ac7446b36674354745b7ecdc4f2794040b6cfd1502019beea6875cb4e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.10.1-cp38-abi3-musllinux_1_1_x86_64.whl
| Download URL | smg-1.10.1-cp38-abi3-musllinux_1_1_x86_64.whl |
|---|---|
| Size | 31.5 MB |
| Tags | CPython 3.8 Linux musl 1.1+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
bf4c69913d5034dc09380405b9442fcf1658ebe47cd9a289715d6c10c76243f3
|
|
BLAKE2b-256 checksum How to use checksums |
3b7776cf20453be40a506beecd3732115f9b1423c90f3118c5ccbf8427ee300e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.10.1-cp38-abi3-musllinux_1_1_aarch64.whl
| Download URL | smg-1.10.1-cp38-abi3-musllinux_1_1_aarch64.whl |
|---|---|
| Size | 33.2 MB |
| Tags | CPython 3.8 Linux musl 1.1+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
e963ef19a983246c5a0456388d9575b2a6fd4ab6a8ad021ccaefb9deff4199d6
|
|
BLAKE2b-256 checksum How to use checksums |
4efbffe85482179a4b3b1fca703d8b7e3f3ab7afedec61a391292b6f79a68160
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.10.1-cp38-abi3-manylinux_2_28_aarch64.whl
| Download URL | smg-1.10.1-cp38-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 34.7 MB |
| Tags | CPython 3.8 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
f2a990fbd62288d0ce9ff55a8e2b830fb6afea27862a1d344a80d49cb2404c74
|
|
BLAKE2b-256 checksum How to use checksums |
fdb84d7a0fcc4897d9a73f51a660eccf520a962ba873ab880d4a85911eb6386d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.10.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | smg-1.10.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 33.0 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
02aabaa107b923cb78f4a93e54f575bdb8b9c1c3de891b36d82a4d37514ccbf5
|
|
BLAKE2b-256 checksum How to use checksums |
2a4fb58d74bbd44724c38873025566e9ed03afe5bedfb7fccc8e19db1eac5314
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.10.1-cp38-abi3-macosx_11_0_arm64.whl
| Download URL | smg-1.10.1-cp38-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 30.5 MB |
| Tags | CPython 3.8 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
2c6add8f29536460c990f24e4ae76bfbac355361f7e2592bdc065d057ce5b4b4
|
|
BLAKE2b-256 checksum How to use checksums |
26fe0aa9c271d7caaa5936c226c284bd4f68c660ec2c2e42a8dabd3dcf34ba8e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.10.1-cp38-abi3-macosx_10_12_x86_64.whl
| Download URL | smg-1.10.1-cp38-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 29.6 MB |
| Tags | CPython 3.8 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
8be0b6ac0ba42cc6dffb651b4b040441ca77729907e198d8c4f49d6e1875a975
|
|
BLAKE2b-256 checksum How to use checksums |
42eda92f34a9f3390e7f4b297f8d3c75ac1c57c2345435710508dbe492cbe28d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
