SMG Python Bindings
This directory contains the Python bindings for SMG (Shepherd Model Gateway), built using maturin and PyO3.
Quick Start
Installation
pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl
Usage
The smg serve command launches backend workers and the SMG router in a single command:
# sglang with gRPC (default)
smg serve --backend sglang --model-path /path/to/model --port 8080
# sglang with HTTP
smg serve --backend sglang --model-path /path/to/model --port 8080 --connection-mode http
# vLLM (gRPC only)
smg serve --backend vllm --model /path/to/model --port 8080
# TensorRT-LLM (gRPC only)
smg serve --backend trtllm --model /path/to/model --port 8080
# Multiple workers (data parallel)
smg serve --backend sglang --model-path /path/to/model --port 8080 --dp-size 4
Serve Options
| Option | Default | Description |
|---|---|---|
--backend |
sglang |
Backend to use: sglang, vllm, or trtllm |
--connection-mode |
grpc |
Connection mode: grpc, http, or zmq. trtllm only supports grpc; tokenspeed only zmq; zmq (a same-host direct engine connection) is available for vllm, tokenspeed, and sglang |
--host |
127.0.0.1 |
Host for the router |
--port |
8080 |
Port for the router |
--dp-size |
1 |
Data parallel size (number of worker replicas) |
--worker-host |
127.0.0.1 |
Host for worker processes |
--worker-base-port |
31000 |
Base port for workers |
--worker-startup-timeout |
300 |
Seconds to wait for workers to become healthy |
Backend-specific options (e.g., --tensor-parallel-size, --quantization) are passed through to the backend.
Directory Structure
bindings/python/
├── src/ # Source code (src layout)
│ ├── lib.rs # Rust/PyO3 bindings implementation
│ └── smg/ # Python source code
│ ├── __init__.py
│ ├── cli.py # CLI entry point
│ ├── serve.py # smg serve implementation
│ ├── launch_router.py
│ ├── router.py
│ └── router_args.py
├── tests/ # Python unit tests
├── Cargo.toml # Rust package configuration
├── pyproject.toml # Python package configuration
└── README.md # This file
Building
Development Build
pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl
Production Build
cd smg/bindings/python
maturin build --release --out dist --features vendored-openssl
pip install dist/smg-*.whl
Testing
cd smg/bindings/python
pytest tests/
Release files for smg 1.13.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| smg-1.13.0.tar.gz | 8.2 MB | Details |
Built distributions (wheels)
| File | Reset | |||
|---|---|---|---|---|
| smg-1.13.0-cp38-abi3-win_amd64.whl | CPython 3.8 | abi3 | Windows x86-64 | Details |
| smg-1.13.0-cp38-abi3-musllinux_1_1_x86_64.whl | CPython 3.8 | abi3 | Linux musl 1.1+ x86-64 | Details |
| smg-1.13.0-cp38-abi3-musllinux_1_1_aarch64.whl | CPython 3.8 | abi3 | Linux musl 1.1+ ARM64 | Details |
| smg-1.13.0-cp38-abi3-manylinux_2_28_aarch64.whl | CPython 3.8 | abi3 | Linux glibc 2.28+ ARM64 | Details |
| smg-1.13.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | CPython 3.8 | abi3 | Linux glibc 2.17+ x86-64 | Details |
| smg-1.13.0-cp38-abi3-macosx_11_0_arm64.whl | CPython 3.8 | abi3 | macOS 11.0+ ARM64 | Details |
| smg-1.13.0-cp38-abi3-macosx_10_12_x86_64.whl | CPython 3.8 | abi3 | macOS 10.12+ x86-64 | Details |
Total release size: 259.9 MB
Release files / smg-1.13.0.tar.gz
| Download URL | smg-1.13.0.tar.gz |
|---|---|
| Size | 8.2 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0f8638751f5ba9aac1ef809cd81cfaf7e354973086a4cb1740a9acc0812dc5da
|
|
BLAKE2b-256 checksum How to use checksums |
d357d839f1068f2686f5f686771b9fc46ec6cff2ada63b404e93d94e1ce6de51
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / smg-1.13.0-cp38-abi3-win_amd64.whl
| Download URL | smg-1.13.0-cp38-abi3-win_amd64.whl |
|---|---|
| Size | 32.9 MB |
| Tags | CPython 3.8 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
e3bcc5c6bc4b4bb65555b0b18fc3002bfcd6208123e35697f72bb7a07e174459
|
|
BLAKE2b-256 checksum How to use checksums |
a7e7aa6c01d88d480c70b711046565dcd68ed56218a15e1e60149865537b4225
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / smg-1.13.0-cp38-abi3-musllinux_1_1_x86_64.whl
| Download URL | smg-1.13.0-cp38-abi3-musllinux_1_1_x86_64.whl |
|---|---|
| Size | 35.9 MB |
| Tags | CPython 3.8 Linux musl 1.1+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
c0f428ca133f0f0a40564ffe996c629220b388f447eb1416923dd1e8850a2f40
|
|
BLAKE2b-256 checksum How to use checksums |
60d5f678318ae060895519b115ba1ca5d73f3b4a76e5aa75aca285e4c4707914
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / smg-1.13.0-cp38-abi3-musllinux_1_1_aarch64.whl
| Download URL | smg-1.13.0-cp38-abi3-musllinux_1_1_aarch64.whl |
|---|---|
| Size | 37.6 MB |
| Tags | CPython 3.8 Linux musl 1.1+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
e746dec554391d7dab12ed102c1d32aa5b837b49112cf5f10bd5be0ae565cb88
|
|
BLAKE2b-256 checksum How to use checksums |
17df8319c35f2dc3521820722824fc5bc948d8e47d7293c95838f4dec4c9394d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / smg-1.13.0-cp38-abi3-manylinux_2_28_aarch64.whl
| Download URL | smg-1.13.0-cp38-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 39.2 MB |
| Tags | CPython 3.8 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
5276328560354dca26ea9f454694c3d56994fb5a41c323c8edf97508181d2d7e
|
|
BLAKE2b-256 checksum How to use checksums |
2612a5c257fae40846b090fa183f1ad76799c6ebdef473b0d65ceb58be65af90
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / smg-1.13.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | smg-1.13.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 37.4 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
cfae8171dac68a74b4822263b1864fa79952992ee472a25ba7266ccd200399b4
|
|
BLAKE2b-256 checksum How to use checksums |
bb8d2fa4a2179654246ba5280f4da0c31a87378cb8bbb1b986724c7abe57ee93
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / smg-1.13.0-cp38-abi3-macosx_11_0_arm64.whl
| Download URL | smg-1.13.0-cp38-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 34.8 MB |
| Tags | CPython 3.8 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
34ccc49390d1f163a3ef61f6d0bf7df5ce97de191ed94069660582156ce4ea26
|
|
BLAKE2b-256 checksum How to use checksums |
aa78532d14d816ba2f91492ee1808e37c6519d6c6a8ac3843160e468b6a28293
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / smg-1.13.0-cp38-abi3-macosx_10_12_x86_64.whl
| Download URL | smg-1.13.0-cp38-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 34.0 MB |
| Tags | CPython 3.8 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
ad3630e5e2ddc0a0b8cfce26b91b468633ab80413374f983ace9a114dba5aad5
|
|
BLAKE2b-256 checksum How to use checksums |
656221521bcfd6021f76c40eb9f261f7cd8093d337c38f2a1869a7acae43c458
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
