SMG Python Bindings

This directory contains the Python bindings for SMG (Shepherd Model Gateway), built using maturin and PyO3.

Quick Start

Installation

pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl

Usage

The smg serve command launches backend workers and the SMG router in a single command:

# sglang with gRPC (default)
smg serve --backend sglang --model-path /path/to/model --port 8080

# sglang with HTTP
smg serve --backend sglang --model-path /path/to/model --port 8080 --connection-mode http

# vLLM (gRPC only)
smg serve --backend vllm --model /path/to/model --port 8080

# TensorRT-LLM (gRPC only)
smg serve --backend trtllm --model /path/to/model --port 8080

# Multiple workers (data parallel)
smg serve --backend sglang --model-path /path/to/model --port 8080 --dp-size 4

Serve Options

Option Default Description
--backend sglang Backend to use: sglang, vllm, or trtllm
--connection-mode grpc Connection mode: grpc or http. vllm/trtllm only support grpc
--host 127.0.0.1 Host for the router
--port 8080 Port for the router
--dp-size 1 Data parallel size (number of worker replicas)
--worker-host 127.0.0.1 Host for worker processes
--worker-base-port 31000 Base port for workers
--worker-startup-timeout 300 Seconds to wait for workers to become healthy

Backend-specific options (e.g., --tensor-parallel-size, --quantization) are passed through to the backend.

Directory Structure

bindings/python/
├── src/                    # Source code (src layout)
│   ├── lib.rs              # Rust/PyO3 bindings implementation
│   └── smg/                # Python source code
│       ├── __init__.py
│       ├── cli.py          # CLI entry point
│       ├── serve.py        # smg serve implementation
│       ├── launch_router.py
│       ├── router.py
│       └── router_args.py
├── tests/                  # Python unit tests
├── Cargo.toml              # Rust package configuration
├── pyproject.toml          # Python package configuration
└── README.md               # This file

Building

Development Build

pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl

Production Build

cd smg/bindings/python
maturin build --release --out dist --features vendored-openssl
pip install dist/smg-*.whl

Testing

cd smg/bindings/python
pytest tests/

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.9.0.post20260815)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for lightseek from gravatar.com lightseek

Credits

Author: Simo Lin

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.9

Provides Extra

dev

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tokenspeed_smg-1.9.0.post20260811.tar.gz (3.4 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

tokenspeed_smg-1.9.0.post20260811-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30.9 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ x86-64

tokenspeed_smg-1.9.0.post20260811-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (32.5 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ ARM64

File details

Details for the file tokenspeed_smg-1.9.0.post20260811.tar.gz.

File metadata

File hashes

Hashes for tokenspeed_smg-1.9.0.post20260811.tar.gz
Algorithm Hash digest
SHA256 dc4f5d2f44c50d9e897ff111da5932dd055609fe5bc302a63dc266413213b271
MD5 e44fec17514fcedf14737aba75db2203
BLAKE2b-256 e2779de5dd12aca79be0603e7d4c0a1f84c129287e23c9235fb1d4128a910382

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_smg-1.9.0.post20260811.tar.gz:

Publisher: tokenspeed-smg.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_smg-1.9.0.post20260811-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for tokenspeed_smg-1.9.0.post20260811-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 f47ecf8e2d2eab714cf504500ae6fea409f02a16b936698fd2a7926ee6a52cff
MD5 1789b19f28bf043d9ba47ebbb1370723
BLAKE2b-256 de24422e54faa54f7cc126a62012c4631a7e5fef44c4a88bd296641a3e35e080

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_smg-1.9.0.post20260811-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: tokenspeed-smg.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_smg-1.9.0.post20260811-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for tokenspeed_smg-1.9.0.post20260811-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 4b30fc65f86b075e191ea4847b911946d94bbe431d45f703f09bc203fad388fe
MD5 1fa1386ee739413fda044e7163b2e7a4
BLAKE2b-256 88c47862a702e0176a448b7fe42c54cfd8857b40dd383082ab16a69b0bbe07b2

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_smg-1.9.0.post20260811-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: tokenspeed-smg.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page