SMG Python Bindings
This directory contains the Python bindings for SMG (Shepherd Model Gateway), built using maturin and PyO3.
Quick Start
Installation
pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl
Usage
The smg serve command launches backend workers and the SMG router in a single command:
# sglang with gRPC (default)
smg serve --backend sglang --model-path /path/to/model --port 8080
# sglang with HTTP
smg serve --backend sglang --model-path /path/to/model --port 8080 --connection-mode http
# vLLM (gRPC only)
smg serve --backend vllm --model /path/to/model --port 8080
# TensorRT-LLM (gRPC only)
smg serve --backend trtllm --model /path/to/model --port 8080
# Multiple workers (data parallel)
smg serve --backend sglang --model-path /path/to/model --port 8080 --dp-size 4
Serve Options
| Option | Default | Description |
|---|---|---|
--backend |
sglang |
Backend to use: sglang, vllm, or trtllm |
--connection-mode |
grpc |
Connection mode: grpc or http. vllm/trtllm only support grpc |
--host |
127.0.0.1 |
Host for the router |
--port |
8080 |
Port for the router |
--dp-size |
1 |
Data parallel size (number of worker replicas) |
--worker-host |
127.0.0.1 |
Host for worker processes |
--worker-base-port |
31000 |
Base port for workers |
--worker-startup-timeout |
300 |
Seconds to wait for workers to become healthy |
Backend-specific options (e.g., --tensor-parallel-size, --quantization) are passed through to the backend.
Directory Structure
bindings/python/
├── src/ # Source code (src layout)
│ ├── lib.rs # Rust/PyO3 bindings implementation
│ └── smg/ # Python source code
│ ├── __init__.py
│ ├── cli.py # CLI entry point
│ ├── serve.py # smg serve implementation
│ ├── launch_router.py
│ ├── router.py
│ └── router_args.py
├── tests/ # Python unit tests
├── Cargo.toml # Rust package configuration
├── pyproject.toml # Python package configuration
└── README.md # This file
Building
Development Build
pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl
Production Build
cd smg/bindings/python
maturin build --release --out dist --features vendored-openssl
pip install dist/smg-*.whl
Testing
cd smg/bindings/python
pytest tests/
Release files for smg 1.10.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| smg-1.10.0.tar.gz | 3.6 MB | Details |
Built distributions (wheels)
| File | Reset | |||
|---|---|---|---|---|
| smg-1.10.0-cp38-abi3-win_amd64.whl | CPython 3.8 | abi3 | Windows x86-64 | Details |
| smg-1.10.0-cp38-abi3-musllinux_1_1_x86_64.whl | CPython 3.8 | abi3 | Linux musl 1.1+ x86-64 | Details |
| smg-1.10.0-cp38-abi3-musllinux_1_1_aarch64.whl | CPython 3.8 | abi3 | Linux musl 1.1+ ARM64 | Details |
| smg-1.10.0-cp38-abi3-manylinux_2_28_aarch64.whl | CPython 3.8 | abi3 | Linux glibc 2.28+ ARM64 | Details |
| smg-1.10.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | CPython 3.8 | abi3 | Linux glibc 2.17+ x86-64 | Details |
| smg-1.10.0-cp38-abi3-macosx_11_0_arm64.whl | CPython 3.8 | abi3 | macOS 11.0+ ARM64 | Details |
| smg-1.10.0-cp38-abi3-macosx_10_12_x86_64.whl | CPython 3.8 | abi3 | macOS 10.12+ x86-64 | Details |
Total release size: 224.1 MB
Release files / smg-1.10.0.tar.gz
| Download URL | smg-1.10.0.tar.gz |
|---|---|
| Size | 3.6 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
359787af1acb3b550bd5edf2c3b480f7f974974d61b5a382b0956e87c749e0ab
|
|
BLAKE2b-256 checksum How to use checksums |
93aecbe29f5ca06309b90bd0ba03e082e48c3635a0755928c442f24fdd9e1223
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.10.0-cp38-abi3-win_amd64.whl
| Download URL | smg-1.10.0-cp38-abi3-win_amd64.whl |
|---|---|
| Size | 28.2 MB |
| Tags | CPython 3.8 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
63956055a179adcad9c3751ba65febabb93716837c88af09a36777116b8475e5
|
|
BLAKE2b-256 checksum How to use checksums |
9064ebdb538cde8ee21f9761406ef55c31a930d90ae5037b065b05def5c0e576
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.10.0-cp38-abi3-musllinux_1_1_x86_64.whl
| Download URL | smg-1.10.0-cp38-abi3-musllinux_1_1_x86_64.whl |
|---|---|
| Size | 31.4 MB |
| Tags | CPython 3.8 Linux musl 1.1+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
b2511ef0a13060d1fcbf951ef03d087373d33b68a3a343081d1fdd1a086e59c3
|
|
BLAKE2b-256 checksum How to use checksums |
093a1509d4b5ac13cb1cde23f1246c12f49af035f7516698a2781ff7be2b69af
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.10.0-cp38-abi3-musllinux_1_1_aarch64.whl
| Download URL | smg-1.10.0-cp38-abi3-musllinux_1_1_aarch64.whl |
|---|---|
| Size | 33.2 MB |
| Tags | CPython 3.8 Linux musl 1.1+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
77c515782b26b4adc9259f63ac0a56229ab54a6edb7918e586d3040db56ed72b
|
|
BLAKE2b-256 checksum How to use checksums |
722cc5d496f14638d74e5695eb9c3723f9695f44183191840a8325f49dfbe08e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.10.0-cp38-abi3-manylinux_2_28_aarch64.whl
| Download URL | smg-1.10.0-cp38-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 34.7 MB |
| Tags | CPython 3.8 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
fc82877cf823cbf7d97b0f3fb0fb6d1a4d6233f1678c3d86378ab63f6a800569
|
|
BLAKE2b-256 checksum How to use checksums |
3c8611df93db4991abcb8f3333c407cdd9bca8b9b4d9c69a3a7e058a59e8abb4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.10.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | smg-1.10.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 33.0 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
bb26a1de5f000bfe1b9993234091791e6d1c3976f133a9d5d4701faa73ce3854
|
|
BLAKE2b-256 checksum How to use checksums |
ac7d2bab7dbfad854b2a4fffa763ecddefb1a989f146b19dc17b72ced23072a4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.10.0-cp38-abi3-macosx_11_0_arm64.whl
| Download URL | smg-1.10.0-cp38-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 30.4 MB |
| Tags | CPython 3.8 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
c181f64abf688a16d61c74b366e4bbf892bf41adb51a9932a22041b868b9cde4
|
|
BLAKE2b-256 checksum How to use checksums |
831ea8b68dac15f0a6d394127b71c5f16fc28278fda5a9ba39084a310c8f09ee
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / smg-1.10.0-cp38-abi3-macosx_10_12_x86_64.whl
| Download URL | smg-1.10.0-cp38-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 29.5 MB |
| Tags | CPython 3.8 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
ee701e1811963fe2bfe99ca2ddd954fbf58c6fbdedfb961c0270c0a24cc6a559
|
|
BLAKE2b-256 checksum How to use checksums |
8dce0a4a526d33cbf23332da74f34c663aa678f86619e4549a2683352679be9a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
