SMG Python Bindings

This directory contains the Python bindings for SMG (Shepherd Model Gateway), built using maturin and PyO3.

Quick Start

Installation

pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl

Usage

The smg serve command launches backend workers and the SMG router in a single command:

# sglang with gRPC (default)
smg serve --backend sglang --model-path /path/to/model --port 8080

# sglang with HTTP
smg serve --backend sglang --model-path /path/to/model --port 8080 --connection-mode http

# vLLM (gRPC only)
smg serve --backend vllm --model /path/to/model --port 8080

# TensorRT-LLM (gRPC only)
smg serve --backend trtllm --model /path/to/model --port 8080

# Multiple workers (data parallel)
smg serve --backend sglang --model-path /path/to/model --port 8080 --dp-size 4

Serve Options

Option Default Description
--backend sglang Backend to use: sglang, vllm, or trtllm
--connection-mode grpc Connection mode: grpc or http. vllm/trtllm only support grpc
--host 127.0.0.1 Host for the router
--port 8080 Port for the router
--dp-size 1 Data parallel size (number of worker replicas)
--worker-host 127.0.0.1 Host for worker processes
--worker-base-port 31000 Base port for workers
--worker-startup-timeout 300 Seconds to wait for workers to become healthy

Backend-specific options (e.g., --tensor-parallel-size, --quantization) are passed through to the backend.

Directory Structure

bindings/python/
├── src/                    # Source code (src layout)
│   ├── lib.rs              # Rust/PyO3 bindings implementation
│   └── smg/                # Python source code
│       ├── __init__.py
│       ├── cli.py          # CLI entry point
│       ├── serve.py        # smg serve implementation
│       ├── launch_router.py
│       ├── router.py
│       └── router_args.py
├── tests/                  # Python unit tests
├── Cargo.toml              # Rust package configuration
├── pyproject.toml          # Python package configuration
└── README.md               # This file

Building

Development Build

pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl

Production Build

cd smg/bindings/python
maturin build --release --out dist --features vendored-openssl
pip install dist/smg-*.whl

Testing

cd smg/bindings/python
pytest tests/

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for lightseek from gravatar.com lightseek

Credits

Author: Simo Lin

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.9

Provides Extra

dev

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tokenspeed_smg-1.9.0.post20260815.tar.gz (3.4 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

tokenspeed_smg-1.9.0.post20260815-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (33.1 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ x86-64

tokenspeed_smg-1.9.0.post20260815-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (34.7 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ ARM64

File details

Details for the file tokenspeed_smg-1.9.0.post20260815.tar.gz.

File metadata

File hashes

Hashes for tokenspeed_smg-1.9.0.post20260815.tar.gz
Algorithm Hash digest
SHA256 22251045875c29bb44495f4f28424117c4e869bcf449594a76e0654278651f36
MD5 e2fec3e19133273ec1bed03337444319
BLAKE2b-256 a3788491dedf80ea997e0bff191ea812bde2e16cb02945011c34b8a18626dfdd

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_smg-1.9.0.post20260815.tar.gz:

Publisher: tokenspeed-smg.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_smg-1.9.0.post20260815-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for tokenspeed_smg-1.9.0.post20260815-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 8582b17e630cfe0729c88beaec390f9d69e77e4a1600a0a485b4413dcb428ef9
MD5 cf7fb4031741fe1528e7943c249240e1
BLAKE2b-256 6ce1ce09d6e0b2c4d53b6417e7f89e6a18866f741d9689767de9b7776ca20dfc

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_smg-1.9.0.post20260815-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: tokenspeed-smg.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_smg-1.9.0.post20260815-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for tokenspeed_smg-1.9.0.post20260815-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 1fbb8d4e2a05b3207c80640d413b719fd28032a91fb455b6b0348689af927d82
MD5 39048d93c79ba9963eb127ae0b5d5988
BLAKE2b-256 8574ebd8934434daa3adc98ae1ffca12bb89ed0cfc5d452a80110856e07ba1ea

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_smg-1.9.0.post20260815-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: tokenspeed-smg.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page