gpuroutertest

The easiest way to run a private, GPU-backed LLM endpoint in your own AWS account — one command up, one command down.

Prerequisites

• Python 3.9+
• AWS credentials (aws configure, AWS_PROFILE, or SSO) with permission to manage EC2
• GPU (G/P) instance quota in your region

Install

pip install gpuroutertest

Usage

gpuroutertest list-models                       # pick a <model_id>
gpuroutertest deploy <model_id> --size small --region us-east-1

Prints an endpoint URL and an API key once the model is loaded and healthy.

curl $ENDPOINT/chat/completions \
  -H "Authorization: Bearer $API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"<model_id>","messages":[{"role":"user","content":"Hello"}]}'

<model_id> is any id from gpuroutertest list-models; use that same id in deploy and in the "model" field.

Then tear it down to stop billing:

gpuroutertest delete $API_KEY --region us-east-1 -y

Spot instances

Cheaper capacity behind a stable Elastic IP, auto-relaunched if AWS reclaims it. The endpoint URL never changes.

gpuroutertest deploy <model_id> --size small --region us-east-1 --spot

Large models

Serve weights from a same-region public S3 bucket instead of HuggingFace — free, multi-GB/s, no flaky boot downloads.

gpuroutertest deploy <model_id> --size large --region us-east-1 \
  --model-uri s3://<bucket>/<model_id>

Cost guard

Auto-terminate after N minutes:

gpuroutertest deploy <model_id> --size small --region us-east-1 --ttl 120

Load testing

pip install gpuroutertest[benchmark]

gpuroutertest --test --endpoint $ENDPOINT --api-key $API_KEY \
  --model <model_id> --concurrency 10 --prompt-length 1k

Python

Full lifecycle in a script — launch, call the endpoint, tear down:

import gpuroutertest as gr
from openai import OpenAI  # pip install gpuroutertest[benchmark]

# 1. Launch (blocks until the model is loaded and healthy)
dep = gr.deploy("<model_id>", size="small", region="us-east-1")  # id from gr.list_models()
print(dep.endpoint_url, dep.api_key, "healthy:", dep.healthy)

# 2. Call it — the endpoint is OpenAI-compatible
client = OpenAI(base_url=dep.endpoint_url, api_key=dep.api_key)
resp = client.chat.completions.create(
    model=dep.model,
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

# 3. Tear down to stop billing
gr.destroy(dep.api_key, region="us-east-1")

dep exposes endpoint_url, api_key, model, instance_id, public_ip, healthy, and (for --spot) elastic_ip / recovery_armed. Any HTTP client works — swap the OpenAI SDK for requests/httpx and POST to dep.endpoint_url + "/chat/completions" with Authorization: Bearer <api_key>.

Deploy without blocking, then reconnect later by API key:

dep = gr.deploy("<model_id>", size="small", region="us-east-1", wait=False)

info = gr.get_deployment(dep.api_key, region="us-east-1")  # state, endpoint, IP
for d in gr.list_deployments(region="us-east-1"):          # everything running
    print(d.api_key, d.state, d.model, d.endpoint_url)

CLI

Every command takes --profile/-p and --region/-r.

gpuroutertest list-models                  # models you can deploy
gpuroutertest deploy   <model_id>          # launch an endpoint
gpuroutertest ps                           # your running deployments
gpuroutertest status   <api_key>           # state, capacity, health
gpuroutertest endpoint <api_key>           # just the URL
gpuroutertest logs     <api_key>           # instance boot logs
gpuroutertest delete   <api_key> -y        # terminate, stop billing
gpuroutertest version

deploy flags

--test flags

Errors and limits

• Port 8000 is open to 0.0.0.0/0 by default, protected only by the API key. Use --cidr in real use.
• Traffic is plain HTTP (no TLS). Put it behind a proxy for anything beyond dev.
• --spot recovers automatically but not instantly — there is a downtime gap while the replacement reloads the model.
• Container logs need SSH/SSM, which are intentionally not provisioned. gpuroutertest logs gives EC2 console output only.

Details

Models

Models live in sdk/gpuroutertest/registry.json, keyed by model id, each holding the per-size config (instance type, disk, server flags). That id is what you pass to deploy, what list-models prints, and what you send as "model" when calling the endpoint. Add a model by adding an entry — no code changes.

Spot failover

--spot allocates an Elastic IP up front and serves the endpoint from it, so the URL survives instance replacement. At deploy time it also provisions two more pieces, all tagged with the deployment's API key so delete removes them together:

• an EventBridge rule that fires on the EC2 Spot Instance Interruption Warning event
• a per-deployment Lambda that the rule invokes

Spot reclaim warning (~2 min notice)
        ↓
EventBridge rule  →  recovery Lambda
        ↓
Launch a replacement spot instance (same model, size, and boot config;
tries each AZ on capacity errors)
        ↓
Re-point the Elastic IP at the replacement  →  endpoint URL unchanged

The replacement is an exact clone of the original launch — same model, weights source, size, and vLLM flags. The rule isn't pinned to an instance id, so it keeps protecting each replacement in turn across any number of reclaims; the Lambda ignores warnings for instances outside this deployment.

There is a downtime gap. Loading a model into GPU memory takes several minutes — 8–15 for large models — far longer than the ~2-minute reclaim notice. Failover is automatic recovery, not zero-downtime: the endpoint is unreachable until the replacement finishes booting, then returns at the same URL. To ride through a reclaim with no gap, run two deployments behind your own load balancer. (Inference is stateless, so there is nothing to checkpoint.)

--spot needs permission for Elastic IPs, Lambda, EventBridge, and IAM (to create its recovery role). status shows Capacity: spot (failover armed). delete removes the instance, Lambda, rule, and Elastic IP in one shot; the shared IAM role is left for reuse and costs nothing.

Weights from S3

With --model-uri, the instance fetches weights from that S3 prefix with s5cmd (--no-sign-request) instead of HuggingFace. Because the bucket is in-region the transfer is free and multi-GB/s, and vLLM serves the local copy under the original model id — callers address the endpoint identically. It's opt-in per deploy; for small models the default HuggingFace path is fine.

• The bucket must be in the same region as --region (cross-region would be slow and incur egress).
• The prefix must point at the folder directly containing config.json, the safetensors shards, model.safetensors.index.json, and the tokenizer files.
• The value must start with s3:// (validated before any AWS call, so typos fail fast).

Seeding a bucket (one-time per model): scripts/seed_model_to_s3.py streams a HuggingFace repo straight into S3 without staging it on local disk, and is resumable. For an unattended overnight run use scripts/seed_overnight.sh, which re-runs the seeder until every file is confirmed uploaded.

python scripts/seed_model_to_s3.py \
  --model  <model_id> \
  --bucket <bucket> \
  --region us-east-1

Benchmark output

--test measures the same run from two vantage points: client side (what a caller experiences, including the network round-trip) and server side (read from vLLM's /metrics). The gap between them is network latency.

Model: <model_id>
Prompt Length: 1K
Concurrency: 10

                      Client        Server
Avg TTFT             1694 ms       1077 ms
P50 TTFT             1663 ms       1300 ms
P95 TTFT             2408 ms       2380 ms
TPM                     9757          9757

Network latency (client - server TTFT): 617 ms
(server observed 24 request(s) during the run)

• TTFT — time to first token; the responsiveness a caller feels.
• TPM — output tokens per minute; throughput.
• Avg / P50 / P95 — average, median, and the slow 5%. P95 far above P50 means requests are queueing.

Warmup matters. A freshly-booted server pays a one-time cold-start cost of several seconds. --warmup fires throwaway requests before the clock starts so that cost never skews results.

Server-side numbers are best-effort: if /metrics is unreachable (e.g. blocked by a security group), the test falls back to client-side numbers with a warning. Test inputs come from sdk/gpuroutertest/prompts/ — one entry per ----separated block; edit them to use your own. Inputs repeat when --requests exceeds the number of unique ones.

--matrix sweeps the full grid (concurrency 1/10/100 × input 100/1k/8k, 9 runs), prints the table, and saves it to inference_benchmark.txt. A failing cell shows as an ERROR row rather than aborting the sweep.

Benchmark Report
Model: <model_id>
(TTFT/Network Latency in ms; TPM in tokens/min; both server-side, from vLLM. Network Latency = client - server TTFT)

Length    Conc    Avg TTFT    Srv TTFT           TPM   Network Latency  Fail
---------------------------------------------------------------------------
100          1         604         106          1415               499     0
100         10         690         184          8208               506     0
100        100         734         225         15857               509     0
1K          10        1715        1034          7413               681     0
8K          10        7406        6654          4924               751     0

License

MIT — see LICENSE.