SMG Python Bindings

This directory contains the Python bindings for SMG (Shepherd Model Gateway), built using maturin and PyO3.

Quick Start

Installation

pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl

Usage

The smg serve command launches backend workers and the SMG router in a single command:

# sglang with gRPC (default)
smg serve --backend sglang --model-path /path/to/model --port 8080

# sglang with HTTP
smg serve --backend sglang --model-path /path/to/model --port 8080 --connection-mode http

# vLLM (gRPC only)
smg serve --backend vllm --model /path/to/model --port 8080

# TensorRT-LLM (gRPC only)
smg serve --backend trtllm --model /path/to/model --port 8080

# Multiple workers (data parallel)
smg serve --backend sglang --model-path /path/to/model --port 8080 --dp-size 4

Serve Options

Option Default Description
--backend sglang Backend to use: sglang, vllm, or trtllm
--connection-mode grpc Connection mode: grpc or http. vllm/trtllm only support grpc
--host 127.0.0.1 Host for the router
--port 8080 Port for the router
--dp-size 1 Data parallel size (number of worker replicas)
--worker-host 127.0.0.1 Host for worker processes
--worker-base-port 31000 Base port for workers
--worker-startup-timeout 300 Seconds to wait for workers to become healthy

Backend-specific options (e.g., --tensor-parallel-size, --quantization) are passed through to the backend.

Directory Structure

bindings/python/
├── src/                    # Source code (src layout)
│   ├── lib.rs              # Rust/PyO3 bindings implementation
│   └── smg/                # Python source code
│       ├── __init__.py
│       ├── cli.py          # CLI entry point
│       ├── serve.py        # smg serve implementation
│       ├── launch_router.py
│       ├── router.py
│       └── router_args.py
├── tests/                  # Python unit tests
├── Cargo.toml              # Rust package configuration
├── pyproject.toml          # Python package configuration
└── README.md               # This file

Building

Development Build

pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl

Production Build

cd smg/bindings/python
maturin build --release --out dist --features vendored-openssl
pip install dist/smg-*.whl

Testing

cd smg/bindings/python
pytest tests/

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.9.0.post20260815)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for lightseek from gravatar.com lightseek

Credits

Author: Simo Lin

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.9

Provides Extra

dev

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tokenspeed_smg-1.9.0.post20260813.tar.gz (3.4 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

tokenspeed_smg-1.9.0.post20260813-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (31.0 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ x86-64

tokenspeed_smg-1.9.0.post20260813-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (32.6 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ ARM64

File details

Details for the file tokenspeed_smg-1.9.0.post20260813.tar.gz.

File metadata

File hashes

Hashes for tokenspeed_smg-1.9.0.post20260813.tar.gz
Algorithm Hash digest
SHA256 4e3078031f603cee2a5e6a46b48ace76f5060283fc50f58c8aedb1f8223eac1b
MD5 79b267f913b71af91b716392b57f826c
BLAKE2b-256 30fbfafe64766f64277618de4b612edc2a50cc03a57617a6d5230162502cdead

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_smg-1.9.0.post20260813.tar.gz:

Publisher: tokenspeed-smg.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_smg-1.9.0.post20260813-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for tokenspeed_smg-1.9.0.post20260813-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 eaef5cf8a5ddbf25241532b194d3d225896a534948851082dc34030e766f04c5
MD5 e36a7fae85569b12c8b009da97299911
BLAKE2b-256 a6f046f6e08c892cc1032bced3ccf1d1fca7194f01d10548ddfac2b2da868960

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_smg-1.9.0.post20260813-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: tokenspeed-smg.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tokenspeed_smg-1.9.0.post20260813-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for tokenspeed_smg-1.9.0.post20260813-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 08e332f459e2a699d153a47d9771fc7b360196288c46706bf6a05fdb00db3805
MD5 41bfdc4dc734f2ddef5792798085db50
BLAKE2b-256 9d0e13907345ab080d477d834b2d4041fab3c5a54b09f5149260299c877f38d3

See more details on using hashes here.

Provenance

The following attestation bundles were made for tokenspeed_smg-1.9.0.post20260813-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: tokenspeed-smg.yml on lightseekorg/tokenspeed-third-party

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page