gpuroutertest 0.2.0

Launch, load-test, and tear down a GPU-backed HTTP API on your own AWS account with one command.

gpuroutertest

Launch a ready-to-call HTTP API on your own AWS GPU with a single command. gpuroutertest picks a GPU that fits, boots the instance, loads the model you choose, and hands you a URL + API key. Load-test it, then tear it down just as fast.

Built for people who want a private GPU-backed endpoint on AWS without hand-rolling EC2, security groups, drivers, and server flags. No console clicking, no infra code — one command up, one command down.

Install

pip install gpuroutertest

You need AWS credentials (aws configure, AWS_PROFILE, or SSO) with permission to manage EC2, plus GPU (G/P) instance quota in your region.

What it does

1. deploy — launch a GPU instance, load a model, and expose it at http://<host>:8000/v1 behind an API key.
2. call it — a standard chat-style REST API; point any HTTP client at the URL.
3. --test — a built-in load-tester that reports latency and throughput, measured from both the client and the server.
4. delete — terminate the instance and stop billing.

CLI quickstart

# See the available models and the GPU tiers they run on
gpuroutertest list-models

# Launch one. A model is just an id string — the same one you pass to `deploy`,
# that `list-models` prints, and that you send as "model" when calling the API.
gpuroutertest deploy zai-org/glm-4-9b-chat-hf --size small --region us-east-1 --profile myprofile

# List what's running (state lives in AWS tags, so keys are never lost)
gpuroutertest ps --region us-east-1

# Inspect / fetch the endpoint / read boot logs
gpuroutertest status   <api_key|instance-id> --region us-east-1
gpuroutertest endpoint <api_key|instance-id> --region us-east-1
gpuroutertest logs     <api_key|instance-id> --region us-east-1

# Tear everything down (stops billing)
gpuroutertest delete   <api_key|instance-id> --region us-east-1 -y

Every command takes --profile/-p and --region/-r. deploy also supports:

Flag	Purpose
--size	small / medium / large tier from the registry
--hf-token	access token for gated models (or set HF_TOKEN)
--model-uri	pull weights from a same-region public S3 prefix (s3://bucket/prefix) instead of HuggingFace — see Loading weights from S3
--cidr	restrict who can reach port 8000 (default 0.0.0.0/0)
--ttl	auto-terminate after N minutes — a cost guard for dev instances
--timeout	minutes to wait for the health check (default 30)
--no-wait	return as soon as the instance is running

Loading weights from S3 (large models)

By default deploy pulls the model straight from HuggingFace. For small models that's perfectly fine — leave it alone. But for large models (hundreds of GB), the HuggingFace download becomes the slow, flaky part of every boot. For those, host the weights once in a public S3 bucket in the same region as your GPUs and point deploy at it:

gpuroutertest deploy zai-org/GLM-5.2-FP8 \
  --size large \
  --model-uri s3://glm-5-2-fp8-weights/zai-org/GLM-5.2-FP8 \
  --region us-east-1 \
  --ttl 120

With --model-uri, the instance fetches the weights from that S3 prefix with s5cmd (--no-sign-request) instead of HuggingFace. Because the bucket is in-region, the transfer is free and multi-GB/s, and vLLM serves the local copy under the original model id — callers address the endpoint exactly the same way. Without --model-uri, nothing changes; it's opt-in per deploy.

Requirements:

• The bucket must be in the same region as --region (that's what makes it free + fast; cross-region would be slow and incur egress).
• The prefix must point at the folder that directly contains config.json, the safetensors shards, model.safetensors.index.json, and the tokenizer files.
• The value must start with s3:// (validated before any AWS call, so typos fail fast).

Seeding the bucket (one-time, per model): scripts/seed_model_to_s3.py streams a HuggingFace repo straight into a public S3 bucket without ever staging it on local disk (so it needs no free space for the model), and is resumable. For an overnight, unattended run use scripts/seed_overnight.sh, which re-runs the seeder until every file is confirmed uploaded.

python scripts/seed_model_to_s3.py \
  --model  zai-org/GLM-5.2-FP8 \
  --bucket glm-5-2-fp8-weights \
  --region us-east-1

Calling the endpoint

deploy prints an endpoint URL and API key. It's a standard chat-style REST API, so any HTTP client works:

curl $ENDPOINT/chat/completions \
  -H "Authorization: Bearer $API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"zai-org/glm-4-9b-chat-hf","messages":[{"role":"user","content":"Hello"}]}'

Load-testing (--test)

Once an endpoint is up, --test drives it under a fixed, repeatable workload and reports how it performs. It measures the same run from two vantage points:

• Client side — what a real caller experiences, including the network round-trip to AWS.
• Server side — the machine's own numbers, read from its built-in /metrics page.

Comparing the two tells you how much of the latency is the server itself versus the network in between.

gpuroutertest --test \
  --endpoint http://<host>:8000/v1 \
  --api-key <api_key> \
  --model zai-org/glm-4-9b-chat-hf \
  --concurrency 10 \
  --prompt-length 1k

Pick which side(s) to show:

• --test (bare) or --test=both — client + server side by side, plus the network gap.
• --test=client — client numbers only (skips the server scrape).
• --test=server — server numbers only.

Flags

Flag	Purpose
--test[=client|server|both]	Run a load test and pick which side(s) to report (bare = both)
--endpoint	Base URL of the server (e.g. http://host:8000/v1)
--api-key	API key for the endpoint
--model	Model id the server expects
--concurrency / -c	Requests kept in flight at once (N), default 1
--prompt-length / -l	Test-input size tier: 100, 1k, or 8k (default 100)
--requests	Total requests to send (default: one per bundled input; repeats to fill larger counts)
--warmup	Throwaway warmup requests fired before timing (default 3, 0 to disable)
--metrics-url	Override the server /metrics URL (default: derived from --endpoint)
--matrix	Run the full sweep (concurrency 1/10/100 × input 100/1k/8k) instead of a single run

What the numbers mean

• TTFT (Time To First Token) — how long until the first piece of the response comes back. This is the responsiveness a caller feels (like time-to-first-byte).
• TPM (Tokens per Minute) — output throughput. (A "token" is a chunk of the response, roughly ¾ of a word.)
• Avg / P50 / P95 — the average, the median (typical request), and the 95th percentile (your slow 5% — the tail). When P95 sits far above P50, some requests are queueing.

Warmup matters. The very first request to a freshly-booted server pays a one-time cold-start cost (loading data into GPU memory, warming caches) that can be several seconds. --warmup fires a few throwaway requests first — discarded, before the clock starts — so that cost never skews your results.

Sample output

Model: zai-org/glm-4-9b-chat-hf
Prompt Length: 1K
Concurrency: 10

                      Client        Server
Avg TTFT             1694 ms       1077 ms
P50 TTFT             1663 ms       1300 ms
P95 TTFT             2408 ms       2380 ms
TPM                     9757          9757

Network latency (client - server TTFT): 617 ms
(server observed 24 request(s) during the run)

Server-side numbers are best-effort: if /metrics can't be reached (e.g. blocked by a security group), the test falls back to client-side numbers with a warning.

Test inputs come from bundled files in sdk/gpuroutertest/prompts/ — one input per entry, separated by a lone --- line. Edit them to use your own. When you ask for more requests than there are unique inputs, they repeat — so pass enough --requests to keep high concurrency busy.

The load-tester needs two optional packages:

pip install gpuroutertest[benchmark]

Full matrix (--matrix)

Add --matrix to sweep the whole grid in one shot — concurrency 1/10/100 × input size 100/1k/8k (9 runs). It prints the table and also saves it to inference_benchmark.txt. A failing cell shows up as an ERROR row instead of aborting the sweep; Srv TTFT / TPM / Network Latency show - for any cell with no server-side numbers. TPM and Network Latency are server-side (from vLLM's /metrics).

gpuroutertest --test --matrix \
  --endpoint http://<host>:8000/v1 --api-key <api_key> \
  --model zai-org/glm-4-9b-chat-hf --requests 100

Benchmark Report
Model: zai-org/glm-4-9b-chat-hf
(TTFT/Network Latency in ms; TPM in tokens/min; both server-side, from vLLM. Network Latency = client - server TTFT)

Length    Conc    Avg TTFT    Srv TTFT           TPM   Network Latency  Fail
---------------------------------------------------------------------------
100          1         604         106          1415               499     0
100         10         690         184          8208               506     0
100        100         734         225         15857               509     0
1K          10        1715        1034          7413               681     0
8K          10        7406        6654          4924               751     0

Library usage

import gpuroutertest as gr

dep = gr.deploy("zai-org/glm-4-9b-chat-hf", size="small", region="us-east-1", profile="myprofile")
print(dep.endpoint_url, dep.api_key)

# For large models, pull weights from a same-region public S3 bucket instead of
# HuggingFace (see "Loading weights from S3" above):
# dep = gr.deploy("zai-org/GLM-5.2-FP8", size="large", region="us-east-1",
#                 model_uri="s3://glm-5-2-fp8-weights/zai-org/GLM-5.2-FP8")

# ... call the HTTP API at dep.endpoint_url with dep.api_key ...

gr.destroy(dep.api_key, region="us-east-1")

You can also run a load test from Python:

from gpuroutertest import run_benchmark

result = run_benchmark(endpoint=dep.endpoint_url, api_key=dep.api_key,
                       model="zai-org/glm-4-9b-chat-hf", concurrency=10, prompt_length="1k")
print(result.avg_ttft_ms, result.tpm, result.network_latency_ms)

Models

Models live in sdk/gpuroutertest/registry.json. Each entry is keyed by its model id and holds the per-size config (instance type, disk, server flags). That same id is what you pass to deploy, what list-models prints, and what you send as "model" when calling the endpoint. Add a model by adding an entry — no code changes required.

Notes & limitations

• The security group opens port 8000; it defaults to the whole internet and is protected only by the API key. Use --cidr in real use.
• Traffic is plain HTTP (no TLS). Put it behind a proxy or load balancer for anything beyond dev.
• Instance boot logs are available via gpuroutertest logs (EC2 console output). The server's own container logs require SSH/SSM, which are intentionally not provisioned.

License

MIT — see LICENSE.