SMG Python Bindings

This directory contains the Python bindings for SMG (Shepherd Model Gateway), built using maturin and PyO3.

Quick Start

Installation

pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl

Usage

The smg serve command launches backend workers and the SMG router in a single command:

# sglang with gRPC (default)
smg serve --backend sglang --model-path /path/to/model --port 8080

# sglang with HTTP
smg serve --backend sglang --model-path /path/to/model --port 8080 --connection-mode http

# vLLM (gRPC only)
smg serve --backend vllm --model /path/to/model --port 8080

# TensorRT-LLM (gRPC only)
smg serve --backend trtllm --model /path/to/model --port 8080

# Multiple workers (data parallel)
smg serve --backend sglang --model-path /path/to/model --port 8080 --dp-size 4

Serve Options

Backend-specific options (e.g., --tensor-parallel-size, --quantization) are passed through to the backend.

Secrets from the environment or files

The serving keys and the store URLs need not be on the command line, where other processes can read them (/proc/<pid>/cmdline, process listings, a Deployment's args). Each is also read from an environment variable, or from a file such as a mounted Secret, by the Python launcher (smg launch, smg serve with the --router- prefix) and by the Rust binary alike:

• The flag beats the environment. A value (from the flag or the environment) together with a file for the same secret is refused at start-up rather than ranked: the second source of one secret is a stale one about to be honoured.
• A single-value file holds the value alone; the surrounding whitespace (a trailing newline) is dropped, and so is the whitespace around a single value from the environment (a Secret made from a file carries the editor's newline into both); the ATP_* values are the exception, see below. An empty file, or one with more than one line, is refused.
• SMG_TENANT_API_KEYS holds tenant_id:key entries separated by , or a newline; the file holds one entry per line, blank lines and lines starting with # skipped. A source that is set but holds no entry is refused.
• A store URL file is read for the selected --history-backend alone. The launcher still reads the older POSTGRES_DB_URL and REDIS_URL names when the SMG_ names are unset.
• The Oracle credentials have no file source; the launcher and the binary (--oracle-dsn, --oracle-user, --oracle-password) read the ATP_* variables when the flags are absent, as they always did, and use them exactly as set, with no trimming: a Secret for them is made with --from-literal, or from a file that ends without a newline (a password that ends in one is a wrong password to the database).
• No value is echoed: a refusal names the sources and the path, and --help shows no value of a variable. The Rust binary's start-up record (CREDENTIAL SOURCES) names each secret's source as flag, env or file; the Python launcher emits no such record.

Directory Structure

bindings/python/
├── src/                    # Source code (src layout)
│   ├── lib.rs              # Rust/PyO3 bindings implementation
│   └── smg/                # Python source code
│       ├── __init__.py
│       ├── cli.py          # CLI entry point
│       ├── serve.py        # smg serve implementation
│       ├── launch_router.py
│       ├── router.py
│       └── router_args.py
├── tests/                  # Python unit tests
├── Cargo.toml              # Rust package configuration
├── pyproject.toml          # Python package configuration
└── README.md               # This file

Building

Development Build

pip install maturin
cd smg/bindings/python
maturin develop --features vendored-openssl

Production Build

cd smg/bindings/python
maturin build --release --out dist --features vendored-openssl
pip install dist/smg-*.whl

Testing

cd smg/bindings/python
pytest tests/

Adding a router flag

A flag lives in a few places, and the tests pin them against each other:

1. model_gateway/src/main.rs: the clap flag, in its group of CliArgs. The Rust CLI is the reference: tests/test_cli_parity.py fails on a flag that exists on one side only.
2. src/smg/router_args.py: a RouterArgs field, and its option in add_cli_args under the matching help group. The field goes at its alphabetical place among the keyword-only fields (every field after worker_urls, host, port), never at the tail: tests/test_arg_parser.py pins the order and says where a misplaced field belongs.
3. src/lib.rs: the Router struct field, the #[pyo3(signature)] entry with the flag's default, the parameter of new and the field of the literal new builds, each at the same alphabetical place among the keyword-only parameters (every parameter after worker_urls, policy, host, port; pyo3 wants the signature and the parameters in one order), plus the line that hands the value to the router configuration in to_router_config. tests/test_router_config.py pins the order; tests/test_router_args_parity.py fails on a dataclass field the constructor does not take, and on a parameter the launcher never sets.
4. The flag's own tests go next to the related ones, or in a file of their own, not at the end of tests/test_arg_parser.py, where every flag's tests would meet.

The keyword-only parameters have no position, so two pull requests that each add a flag change different lines of each file and merge without a conflict.