relay-gateway 0.1.0

Smart AI Gateway — transparent LLM proxy with routing, savings, and quality evidence

Relay — Smart AI Gateway

A self-hosted gateway that sits between any LLM-using app (opencode, custom scripts, your terminal) and OpenRouter. Routes each request to the cheapest model that can handle it, logs the cost, and proves the savings against a stated baseline with a published quality delta from a benchmark of real traffic.

---

Quickstart (5 minutes)

# 1. Install + init
uv tool install git+https://github.com/pulkyeet/relay
relay init

# 2. Add your OpenRouter key
# Edit ~/.relay/relay.yaml, paste your key under openrouter.api_key (or set
# the OPENROUTER_API_KEY environment variable).

# 3. Start the server
relay run

# 4. Use it
open http://127.0.0.1:8000/health
# → {"status": "ok", "version": "0.1.0"}

The distribution is named relay-gateway on PyPI (relay was taken); the commands it installs are relay and relay-benchmark. Once a release is published, uv tool install relay-gateway works too.

---

opencode integration

Add a custom provider to ~/.config/opencode/opencode.jsonc:

{
  "provider": {
    "relay": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Relay Gateway",
      "options": {
        "baseURL": "http://127.0.0.1:8000/v1"
      },
      "models": {
        "cheap":  { "name": "Cheap (alias → ling-2.6-flash)" },
        "smart":  { "name": "Smart (alias → deepseek-v4-pro)" },
        "baseline":  { "name": "Baseline (alias → deepseek-v4-flash)" }
      }
    }
  }
}

Then inside opencode:

1. Run /connect
2. Search for "relay" — it's the custom provider you just added
3. Paste your gateway key (the key from relay.yaml)

Now when you run /models, you see the Relay models. Choose "Cheap" for the cheapest tier, "Smart" for quality-critical tasks, or "Baseline" to use the counterfactual reference model directly.

Note: Traffic through the Relay provider uses your OpenRouter API key (you pay per-token to OpenRouter). Your opencode-go subscription is a separate path — choose the "Relay" provider when you want routing + savings, or use the default opencode-go provider when you don't.

---

CLI reference

Command	What it does
relay init	Writes starter config to ~/.relay/, generates a gateway key, creates the SQLite DB, prints the opencode snippet
relay run	Starts the gateway server on 127.0.0.1:8000. Flags: --host, --port, --config
relay migrate	Runs Alembic migrations (Postgres only; SQLite auto-creates tables on startup)
relay-benchmark sample	Draws 20% of recent request_log traffic into benchmark_case rows (stratified by category)
relay-benchmark run	Executes each benchmark case through BOTH the routed path and the baseline model. Costs money. Use --dry-run first.
relay-benchmark judge	Calls an LLM judge (default: gpt-4o) to score routed vs baseline answers. Computes Cohen's κ against your hand-labels. Blocks if κ < 0.7.
relay-benchmark annotate	Walks you through each tuning-split case, shows both answers, prompts for a verdict (r/b/t/s/q). Writes to ~/.relay/judge_handlabels.yaml.
relay-benchmark ratios	Derives per-category output-length calibration ratios from a judged run. Writes calibration_ratio rows.
relay-benchmark status	Shows the latest benchmark run's quality delta, κ, and methodology.

---

The savings story

How it works

1. Route smart: the gateway checks each request against your tier rules and picks the cheapest model that can handle it. Every decision is logged with its reason (X-Relay-Route-Reason header).

2. Measure honestly: every request logs input/output tokens, actual cost (from OpenRouter prices), the chosen model, and the counterfactual cost if the baseline model had served it instead.

3. Publish the number: the dashboard shows cumulative money saved, savings this week (broken down by cheap-routing vs cache hits vs failovers), and a quality delta from the latest benchmark run — all backed by a stated methodology.

Honest accounting

Relay never hides bad news. The formula savings = counterfactual - actual produces:

• Positive → you saved money vs always using the baseline
• Negative → a failover or error cost more than the baseline would have. Shown in red on the dashboard, counted in the aggregate.
• Zero → cache hit with a known baseline → full save displayed.
• NULL → the baseline model isn't in the price cache, so we can't compute savings. Dashboard shows "—" (unknown), never "$0.00" — that would be a lie.

Naive vs calibrated

• Naive (default): assumes the baseline would have produced the same number of output tokens as the routed model did. Labeled as an estimate.
• Calibrated: applies per-category output-length ratios learned from a benchmark run. The formula becomes baseline_in_price × input_tokens + baseline_out_price × (output_tokens × ratio). Labeled as a measurement.

---

The benchmark workflow

Phase 5 turned the savings number from an estimate into a measurement. Here's the workflow to fire the benchmark and get the published quality delta:

# 1. Dogfood: run in passthrough mode for a few days to capture real traffic
#    In relay.yaml: mode: passthrough

# 2. Sample 20% of logged traffic
relay-benchmark sample --seed 0

# 3. Dual-path run (costs real money — preview with --dry-run)
relay-benchmark run --dry-run
relay-benchmark run --concurrency 4

# 4. Score with the LLM judge
relay-benchmark judge --run-id 1

# 5. Hand-label cases for κ calibration
relay-benchmark annotate --run-id 1

# 6. Re-run judge now that hand-labels exist
relay-benchmark judge --run-id 1

# 7. Compute calibration ratios
relay-benchmark ratios --run-id 1

# 8. Flip methodology in relay.yaml:
#    savings:
#      methodology: calibrated
#      calibration_run_id: 1
#    Restart: relay run

The dashboard hero now shows the real quality delta with the methodology tag, and the digest one-liner includes it.

---

Current model lineup

Tier	Primary	Sibling (failover)
cheap	inclusionai/ling-2.6-flash	nex-agi/nex-n2-mini
smart	deepseek/deepseek-v4-pro	xiaomi/mimo-v2.5-pro
baseline	deepseek/deepseek-v4-flash	xiaomi/mimo-v2.5

Baseline for the counterfactual is models[0] of the baseline tier: deepseek/deepseek-v4-flash.

---

Configuration reference

All config lives in ~/.relay/relay.yaml (or wherever RELAY_CONFIG points).

Key	Default	Description
gateway_keys	[]	List of {name, key, spend_cap_usd} pairs
openrouter.api_key	""	OpenRouter API key (or OPENROUTER_API_KEY env)
openrouter.base_url	https://openrouter.ai/api/v1	—
database.url	sqlite+aiosqlite:///~/.relay/relay.db	SQLite default; postgresql+asyncpg://... for Postgres
mode	normal	passthrough for dogfooding
savings.baseline_model	""	The model you'd always have used (budget baseline)
savings.methodology	naive	calibrated after a benchmark run
savings.calibration_run_id	null	benchmark_run.id to read ratios from
savings.digest_webhook_url	""	Optional POST-to endpoint for the weekly one-liner
benchmark.sample_fraction	1.0	Fraction of traffic drawn into the benchmark (1.0 = use all)
benchmark.min_sample_size	5	Floor on sample size at low traffic
benchmark.judge_model	openai/gpt-4o	Model used as the LLM judge
benchmark.kappa_threshold	0.7	Minimum Cohen's κ to publish a quality claim
dashboard.admin_token	""	Password-protect / and /digest with Bearer auth
log_payloads	false	Store request/response bodies (auto-forced in passthrough mode)
prices	{}	Static price overrides (keyed by model id, {input, output} in USD)

---

Architecture notes

• Single-user, self-hosted, no network calls. Everything runs on your local machine. The only outbound calls are to OpenRouter (and optionally the digest webhook).
• OpenRouter only. No Anthropic adapter, no opencode-go provider — just the OpenAI-compatible endpoint forwarded to OpenRouter.
• SQLite by default (sidecar mode). No Postgres needed. Tables auto-create on startup. For production with multiple users, the Postgres option exists.
• No Docker, no cloud deploy. uvx relay init && relay run is the only install path.
• No learned classifier router — the deterministic rules router ships and the benchmark gives you the evidence it works. A learned router trained on judge labels is the documented upgrade path (see: Maybe-Pile), but is not built.

---

Maybe-Pile (documented, not built)

• Learned router: classifier trained on judge labels. The flywheel is traffic → judge labels → better router. Not built; documented as the architecture's upgrade path once traffic volume justifies it.
• Shadow sampling: continuously re-run a slice of production traffic against the baseline. The benchmark runner contains most of this code; wiring it to production traffic is a future step.
• Semantic cache: higher hit rate than exact-match, but hot-path cost and wrong-answer risk.
• Anthropic-format endpoint (/v1/messages): not needed for single-user opencode + OpenRouter usage; explicitly out of scope.

---

Known limitations

• Single-user design. No multi-tenant, no rate limiting, no team dashboards.
• Gateway key auth only — no SSO, no OAuth, no API-key rotation.
• Dashboard auth is a shared secret (dashboard.admin_token), not per-user. Empty on localhost by default.
• The benchmark's dual-path run costs real money on OpenRouter ($5–30 for a full 200-case run). The judge model call also costs money ($2–5 for 200 cases with gpt-4o).
• Streaming response bodies are reassembled from SSE deltas with a 64KB cap. Very long responses (10K+ tokens of output) get truncated with a _truncated flag.