SMG Python Bindings

This directory contains the Python bindings for SMG (Shepherd Model Gateway), built using maturin and PyO3.

Quick Start

Installation

pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl

Usage

The smg serve command launches backend workers and the SMG router in a single command:

# sglang with gRPC (default)
smg serve --backend sglang --model-path /path/to/model --port 8080

# sglang with HTTP
smg serve --backend sglang --model-path /path/to/model --port 8080 --connection-mode http

# vLLM (gRPC only)
smg serve --backend vllm --model /path/to/model --port 8080

# TensorRT-LLM (gRPC only)
smg serve --backend trtllm --model /path/to/model --port 8080

# Multiple workers (data parallel)
smg serve --backend sglang --model-path /path/to/model --port 8080 --dp-size 4

Serve Options

Option Default Description
--backend sglang Backend to use: sglang, vllm, or trtllm
--connection-mode grpc Connection mode: grpc or http. vllm/trtllm only support grpc
--host 127.0.0.1 Host for the router
--port 8080 Port for the router
--dp-size 1 Data parallel size (number of worker replicas)
--worker-host 127.0.0.1 Host for worker processes
--worker-base-port 31000 Base port for workers
--worker-startup-timeout 300 Seconds to wait for workers to become healthy

Backend-specific options (e.g., --tensor-parallel-size, --quantization) are passed through to the backend.

Directory Structure

bindings/python/
├── src/                    # Source code (src layout)
│   ├── lib.rs              # Rust/PyO3 bindings implementation
│   └── smg/                # Python source code
│       ├── __init__.py
│       ├── cli.py          # CLI entry point
│       ├── serve.py        # smg serve implementation
│       ├── launch_router.py
│       ├── router.py
│       └── router_args.py
├── tests/                  # Python unit tests
├── Cargo.toml              # Rust package configuration
├── pyproject.toml          # Python package configuration
└── README.md               # This file

Building

Development Build

pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl

Production Build

cd smg/bindings/python
maturin build --release --out dist --features vendored-openssl
pip install dist/smg-*.whl

Testing

cd smg/bindings/python
pytest tests/

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.9.0.post20260815)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for lightseek from gravatar.com lightseek

Credits

Author: Simo Lin

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.9

Provides Extra

dev

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tokenspeed_smg-1.9.0.post20260814.tar.gz (3.4 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

tokenspeed_smg-1.9.0.post20260814-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (33.1 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ x86-64

tokenspeed_smg-1.9.0.post20260814-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (34.8 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ ARM64

File details

Details for the file tokenspeed_smg-1.9.0.post20260814.tar.gz.

File metadata

File hashes

Hashes for tokenspeed_smg-1.9.0.post20260814.tar.gz
Algorithm Hash digest
SHA256 22cecf1f3610986beced2920d4a6858ad3060ea7966c4cd2f4db0d62642047e5
MD5 a724925f6baf4195e627ecebd41aaa3d
BLAKE2b-256 361c5d56a74d4609a69d957293c24f56ed42b194f2c8cac5311935cff7275d63

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_smg-1.9.0.post20260814.tar.gz:

Publisher: tokenspeed-smg.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_smg-1.9.0.post20260814-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for tokenspeed_smg-1.9.0.post20260814-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 2b4d46201b2aa30f221c1ef62447aaea6bcbaf874213d1a76f1c21dcf8d0b4cb
MD5 8a0e4b08fcb2a9580e06e8fd2f015f22
BLAKE2b-256 db828ffd883a0d62f60c7b3865667b511cb8eba3ba7b19d61e179993d4328e88

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_smg-1.9.0.post20260814-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: tokenspeed-smg.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_smg-1.9.0.post20260814-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for tokenspeed_smg-1.9.0.post20260814-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 11f5762adb375669e13ce92a540c2acf733260fdaaaa57af4b146036581187d2
MD5 9a7b5e35ca29cfb588b62a798b6ec2de
BLAKE2b-256 dde283b023251472a10b2fbe34a79ab41f36a29bf9709ec6dc391afd95e0c8af

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_smg-1.9.0.post20260814-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: tokenspeed-smg.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page