SMG Python Bindings
This directory contains the Python bindings for SMG (Shepherd Model Gateway), built using maturin and PyO3.
Quick Start
Installation
pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl
Usage
The smg serve command launches backend workers and the SMG router in a single command:
# sglang with gRPC (default)
smg serve --backend sglang --model-path /path/to/model --port 8080
# sglang with HTTP
smg serve --backend sglang --model-path /path/to/model --port 8080 --connection-mode http
# vLLM (gRPC only)
smg serve --backend vllm --model /path/to/model --port 8080
# TensorRT-LLM (gRPC only)
smg serve --backend trtllm --model /path/to/model --port 8080
# Multiple workers (data parallel)
smg serve --backend sglang --model-path /path/to/model --port 8080 --dp-size 4
Serve Options
| Option | Default | Description |
|---|---|---|
--backend |
sglang |
Backend to use: sglang, vllm, or trtllm |
--connection-mode |
grpc |
Connection mode: grpc or http. vllm/trtllm only support grpc |
--host |
127.0.0.1 |
Host for the router |
--port |
8080 |
Port for the router |
--dp-size |
1 |
Data parallel size (number of worker replicas) |
--worker-host |
127.0.0.1 |
Host for worker processes |
--worker-base-port |
31000 |
Base port for workers |
--worker-startup-timeout |
300 |
Seconds to wait for workers to become healthy |
Backend-specific options (e.g., --tensor-parallel-size, --quantization) are passed through to the backend.
Directory Structure
bindings/python/
├── src/ # Source code (src layout)
│ ├── lib.rs # Rust/PyO3 bindings implementation
│ └── smg/ # Python source code
│ ├── __init__.py
│ ├── cli.py # CLI entry point
│ ├── serve.py # smg serve implementation
│ ├── launch_router.py
│ ├── router.py
│ └── router_args.py
├── tests/ # Python unit tests
├── Cargo.toml # Rust package configuration
├── pyproject.toml # Python package configuration
└── README.md # This file
Building
Development Build
pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl
Production Build
cd smg/bindings/python
maturin build --release --out dist --features vendored-openssl
pip install dist/smg-*.whl
Testing
cd smg/bindings/python
pytest tests/
Release files for smg 1.11.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| smg-1.11.0.tar.gz | 4.7 MB | Details |
Built distributions (wheels)
| File | Reset | |||
|---|---|---|---|---|
| smg-1.11.0-cp38-abi3-win_amd64.whl | CPython 3.8 | abi3 | Windows x86-64 | Details |
| smg-1.11.0-cp38-abi3-musllinux_1_1_x86_64.whl | CPython 3.8 | abi3 | Linux musl 1.1+ x86-64 | Details |
| smg-1.11.0-cp38-abi3-musllinux_1_1_aarch64.whl | CPython 3.8 | abi3 | Linux musl 1.1+ ARM64 | Details |
| smg-1.11.0-cp38-abi3-manylinux_2_28_aarch64.whl | CPython 3.8 | abi3 | Linux glibc 2.28+ ARM64 | Details |
| smg-1.11.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | CPython 3.8 | abi3 | Linux glibc 2.17+ x86-64 | Details |
| smg-1.11.0-cp38-abi3-macosx_11_0_arm64.whl | CPython 3.8 | abi3 | macOS 11.0+ ARM64 | Details |
| smg-1.11.0-cp38-abi3-macosx_10_12_x86_64.whl | CPython 3.8 | abi3 | macOS 10.12+ x86-64 | Details |
Total release size: 230.8 MB
Release files / smg-1.11.0.tar.gz
| Download URL | smg-1.11.0.tar.gz |
|---|---|
| Size | 4.7 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
43ef04d699de2deb5b93e4a5ddddd686ddcf44bd7a41be3e9e26b7ae3d6617b0
|
|
BLAKE2b-256 checksum How to use checksums |
7993bb8375ab2cd14146b5b921172dd18fafeba853dc60daf719d8f49a3cc35e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.11.0-cp38-abi3-win_amd64.whl
| Download URL | smg-1.11.0-cp38-abi3-win_amd64.whl |
|---|---|
| Size | 29.0 MB |
| Tags | CPython 3.8 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
2f439d1ee3f91e596dcb047601442a9b6cde85f773d9615eace96ad035aa4a0a
|
|
BLAKE2b-256 checksum How to use checksums |
918f180b91e47d20d25b1795cec153054901f5f115bfe3470edaf6cd50571b98
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.11.0-cp38-abi3-musllinux_1_1_x86_64.whl
| Download URL | smg-1.11.0-cp38-abi3-musllinux_1_1_x86_64.whl |
|---|---|
| Size | 32.2 MB |
| Tags | CPython 3.8 Linux musl 1.1+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
a8201fd6b71e1a8b9006ec1b490f630c3d77b1a3dffa30ead91646790e58ca52
|
|
BLAKE2b-256 checksum How to use checksums |
fb0c85f2644f140b4283f9bfc9ac2d3ad9a98d371ca4370dd2c330998abda8d2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.11.0-cp38-abi3-musllinux_1_1_aarch64.whl
| Download URL | smg-1.11.0-cp38-abi3-musllinux_1_1_aarch64.whl |
|---|---|
| Size | 34.0 MB |
| Tags | CPython 3.8 Linux musl 1.1+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
1cf08938471f33660d572d48b3c26417b18a468a8366fb5d4a18dc4ec8c19b5a
|
|
BLAKE2b-256 checksum How to use checksums |
b9d341688a359bef40f26f9f436af7f5467c79bb0b8fe1543e4d3591cb9f470d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.11.0-cp38-abi3-manylinux_2_28_aarch64.whl
| Download URL | smg-1.11.0-cp38-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 35.5 MB |
| Tags | CPython 3.8 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
c446c1e228a36bc262a359eedec90825bff703a36c6f21c20cde4689ff90049e
|
|
BLAKE2b-256 checksum How to use checksums |
e52bf4bef1d6f93d8105c5174a1a7e11fbec2aa6bf4193fdf9e262f286f35992
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.11.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | smg-1.11.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 33.8 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
f5c0e8f73f5770009fdab204256e7ce96dbed0ac434a4d4e27f59baf4e675831
|
|
BLAKE2b-256 checksum How to use checksums |
b7848df25858445da1ebdca7e676c6b353f1ab74380199fe5c2163009b8d7e57
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.11.0-cp38-abi3-macosx_11_0_arm64.whl
| Download URL | smg-1.11.0-cp38-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 31.3 MB |
| Tags | CPython 3.8 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
36868ac0741f9410a9b03118257e5dd0d6b5aafcaf9e57e1ee9d19c0f9ced9b1
|
|
BLAKE2b-256 checksum How to use checksums |
34f526017f173108b4cc6279f4ea2ae7c988a699f8ae17b8dd72121d10119217
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.11.0-cp38-abi3-macosx_10_12_x86_64.whl
| Download URL | smg-1.11.0-cp38-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 30.3 MB |
| Tags | CPython 3.8 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
eb8b8735651135db54ca7a99a1bc562c9b64a26eabd0dca279682acb90ab09d4
|
|
BLAKE2b-256 checksum How to use checksums |
d59df47ff91d5260b77fcaf508cd9fbfdf148f18aa983af5768479558a7ccea3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
