gpuroutertest 0.1.0

Spin up OpenAI-compatible vLLM endpoints on your own AWS GPUs with one command.

gpuroutertest

Spin up OpenAI-compatible vLLM endpoints on your own AWS GPUs with a single command. gpuroutertest picks a GPU instance that fits the model, boots it from a Deep Learning AMI, downloads the weights, and hands you an endpoint URL + API key. Tear it down just as fast when you're done.

Built for developers who want to rapidly stand up a private inference endpoint for a Hugging Face model without hand-rolling EC2, security groups, and vLLM flags.

Install

pip install gpuroutertest

You need AWS credentials (aws configure, AWS_PROFILE, or SSO) with permission to manage EC2, plus GPU (G/P) instance quota in your region.

CLI quickstart

# See what's available
br list-models

# Deploy GLM-4 9B on a single A10G, wait until it's serving
br deploy glm49b --size small --region us-east-1 --profile myprofile

# List running deployments (source of truth = AWS tags, so keys are never lost)
br ps --region us-east-1

# Inspect / fetch endpoint / read boot logs
br status  <api_key|instance-id> --region us-east-1
br endpoint <api_key|instance-id> --region us-east-1
br logs     <api_key|instance-id> --region us-east-1

# Tear everything down (stops billing)
br delete  <api_key|instance-id> --region us-east-1 -y

Every command takes --profile/-p and --region/-r. deploy also supports:

Flag	Purpose
--size	small / medium / large tier from the registry
--hf-token	token for gated HF models (or set HF_TOKEN)
--cidr	restrict who can reach port 8000 (default 0.0.0.0/0)
--ttl	auto-terminate after N minutes — a cost guard for dev instances
--timeout	minutes to wait for the health check (default 30)
--no-wait	return as soon as the instance is running

Library usage

import gpuroutertest as br

dep = br.deploy("glm49b", size="small", region="us-east-1", profile="myprofile")
print(dep.endpoint_url, dep.api_key)

# ... use the OpenAI-compatible API at dep.endpoint_url with dep.api_key ...

br.destroy(dep.api_key, region="us-east-1")

Call the endpoint like any OpenAI server:

curl $ENDPOINT/chat/completions \
  -H "Authorization: Bearer $API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"zai-org/glm-4-9b-chat-hf","messages":[{"role":"user","content":"Hello"}]}'

Models

Models live in sdk/gpuroutertest/registry.json. Each entry maps a key to a Hugging Face model id and per-size serving config (instance type, disk, vLLM flags). Add a model by adding an entry — no code changes required.

Notes & limitations

• The security group opens port 8000; it defaults to the whole internet and is protected only by the API key. Use --cidr in real use.
• Traffic is plain HTTP (no TLS). Put it behind a proxy/load balancer for anything beyond dev.
• Instance boot logs are available via br logs (EC2 console output). vLLM's own container logs require SSH/SSM, which are intentionally not provisioned.

License

MIT — see LICENSE.