tierwise

A self-hosted, OpenAI-compatible LLM gateway that picks the cheapest model expected to answer each request well, across Anthropic, OpenAI, Gemini and local Ollama models, and measures whether that choice paid off. Point any OpenAI SDK client at it by changing base_url.

Benchmark: on 3,943 held-out prompts, tierwise's router keeps 95% of GPT-4's quality at 44% of its cost (random routing: 63%). Method, out-of-distribution results and limitations are in BENCHMARK.md.

Status: 0.8.0, milestones M0–M8 done: learned routing, reliability, keys and budgets, observability and a PyPI release. M9, the data flywheel, is next. See CHANGELOG.md. Design: docs/superpowers/specs/2026-11-01-tierwise-design.md

Quickstart

pip install tierwise        # or run it without installing: uvx tierwise
tierwise fetch-model        # optional: the 90 MB encoder for the learned router
export ANTHROPIC_API_KEY=…  # any of the provider keys below (or put them in .env)
tierwise                    # serves http://127.0.0.1:8000

The default model registry ships with the package. Point TIERWISE_REGISTRY at your own copy to change models or prices. From a checkout, use uv sync and then uv run tierwise; the repo's config/ and models/router take precedence over the packaged copies.

Use it from any OpenAI SDK client by changing the base URL:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused")
reply = client.chat.completions.create(
    model="auto",              # let tierwise choose; or any id from config/registry.yaml
    messages=[{"role": "user", "content": "Hello!"}],
    stream=False,              # stream=True works too
)

GET /v1/models lists the models whose provider is configured. Each response carries X-Tierwise-Model, X-Tierwise-Provider, X-Tierwise-Reason and X-Tierwise-Request-Id headers, plus X-Tierwise-Cost-USD on non-streamed replies.

Automatic routing

Send one of these as model and tierwise picks the model for you:

Put [critical] or [easy] anywhere in your last message to force the hardest or easiest routing; the tag is removed before the request is forwarded.

How a model is chosen:

1. Difficulty. src/tierwise/router/rules.py scores the request from cheap signals: length, code, math, high-stakes words ("production", "security"), reasoning words ("design", "prove"), easy words ("translate", "hi"), tools, images and conversation length. When the rules aren't sure, the learned router decides instead.
2. Quality bar. bar = 0.30 + 0.65 x difficulty, shifted by auto:cheap/auto:quality, and raised one step when the scorer is unsure (a score near 0.5).
3. Selection. Models that lack a needed capability (tools, vision, JSON) or context window are dropped. Of those that clear the bar, the one with the lowest estimated cost wins, counting prompt-cache discounts. A 422 no_eligible_model means nothing could handle it.
4. Affinity. Conversations stick to the model whose prompt cache is still warm, unless switching would save more than 10%.

Reasoning effort

Thinking tokens are billed as output, so tierwise picks a model and an effort level together: each model with an efforts block in the registry becomes one option per level, and the cheapest option that still clears the quality bar wins.

• Scale: minimal, low, medium, high, xhigh, max. Sent to Anthropic as output_config.effort, to OpenAI as reasoning_effort, and to Gemini as thinkingConfig.thinkingLevel. When the chosen level is the provider's default, nothing is sent.
• Quality: a level below the default lowers quality by its registry value at difficulty 0.5, scaled with the request's difficulty: low effort costs little on easy requests and a lot on hard ones.
• Caching: effort_in_cache_key: true (Anthropic) means a different effort can't reuse the conversation's prompt cache, so tierwise prices that switch as a cache miss.
• Your effort wins: send reasoning_effort and tierwise only considers models that support that level (422 no_eligible_model if none do).
• X-Tierwise-Effort reports the level sent, or default; scripts/costs.py breaks cost down by effort.

X-Tierwise-Reason says why: rules (scored), learned (the classifier decided), override (tag, tier or explicit model), affinity (kept on its warm model) or fallback (router error, or no model cleared the bar). Retune the scorer by editing WEIGHTS in rules.py and running uv run pytest tests/router/test_rules.py, which checks it against 30 labeled prompts.

Learned router

The rules router decides alone only when it is confident (confidence of 0.8 or more, about 1.5% of real prompts); everywhere else a trained classifier decides, and the rules can only raise its difficulty, never lower it. It embeds the last two user turns and the head of the system prompt (at most 256 tokens) with MiniLM on ONNX Runtime, and a small classifier head predicts P(a cheap model's answer won't be good enough). Inference runs in a two-thread pool, never on the event loop.

It is trained on routellm/gpt4_dataset (Apache-2.0): prompts whose Mixtral answer a GPT-4 judge graded below 4 of 5 are labeled "needs a strong model". Held-out results (models/router/metrics.json):

Trained on 32k prompts, 13% of which needed a strong model. Validation picked the MLP over logistic regression (0.745 vs 0.731 ROC-AUC) and a decision threshold of 0.15. On CPU the encoder plus classifier takes 6.3 ms at p50 (scripts/bench_router.py).

The tokenizer and trained classifier are committed and ship in the wheel; the 90 MB ONNX encoder does not. Without it the gateway logs a warning and routes with rules only. Fetch it with tierwise fetch-model, which downloads the official Hugging Face export at a pinned commit and checks its sha256 before saving it to ~/.cache/tierwise/router/, where the gateway finds it on its own. The gateway looks for model.onnx in the router directory first, then in that cache. Only use --out DIR with TIERWISE_ROUTER_DIR=DIR if DIR also holds router.json, classifier.npz and tokenizer.json. Otherwise the learned router turns itself off.

To export it yourself, or to retrain:

uv sync --group train
uv run python -m training.export_onnx             # models/router/model.onnx
uv run python -m training.prepare_data --limit 40000
uv run python -m training.train                   # classifier.npz, router.json, metrics.json
uv run python scripts/bench_router.py             # p50 latency, budget 10 ms

Set TIERWISE_LEARNED_ROUTER=false to turn it off; TIERWISE_ROUTER_DIR points elsewhere.

Keys, budgets and rate limits

Give each team or app its own key, with a monthly spending cap and a request rate:

tierwise keys create --name analytics --budget 50 --rpm 60   # prints the key once
tierwise keys list                                           # spend this month, no secrets
tierwise keys revoke 3f9c2a1b7d4e

Clients send the key the way they would an OpenAI key:

client = OpenAI(base_url="http://localhost:8000/v1", api_key="tw-...")

• Keys are stored only as SHA-256 hashes. Set TIERWISE_REQUIRE_KEY=true to reject requests without a valid key (401 invalid_api_key). Even with that off, a valid key is still used, so its spend is tracked. The gateway picks up keys created or revoked by the CLI within 30 s.
• Rate limit: a token bucket per key. A key can burst up to its --rpm, then gets --rpm requests per minute. Beyond that the response is 429 rate_limited with a Retry-After header.
• Budget: per UTC calendar month.
  • Before calling a model, the request reserves its estimated cost. A reservation that would exceed the budget gets 429 budget_exceeded. That stops a burst of concurrent requests from all passing a check before any of them is charged.
  • Reservations are estimates. Without max_tokens, the output is estimated at 512 tokens, so a key can end the month over its budget by the real cost of the requests already in flight. Set max_tokens to make the cap tight.
  • Enforcement: budgets and rate limits are enforced only with TIERWISE_REQUIRE_KEY=true. Otherwise a client can drop its key and be served anonymously.
  • After the call, the actual cost is charged and the reservation is released. Failed requests cost nothing, and cache hits cost $0.
  • Persistence: spend lives in memory and is written to the usage_monthly table without ever being dropped. A failed database write is retried, and shutdown waits for it.

Reliability

A request survives a failing model or provider:

• Retry. Up to 2 retries on the same model for rate limits, 5xx and timeouts, with exponential backoff and full jitter. Bad requests (4xx) and rejected credentials are never retried.
• Failover. Then the selector's next candidate is tried, up to 3 models per request.
  • Provider-wide failures: after a connection failure or an auth error, the rest of that provider's models are skipped.
  • Model failures: after a model fails, other providers are tried before its siblings.
  • Bad requests are never failed over.
  • X-Tierwise-Failovers and the log's failover_count say how many candidates were abandoned. A named model ("model": "claude-sonnet-5-5") is retried but never swapped.
• Circuit breakers run at two levels:
  • Per (provider, model): trips on 429, 5xx and timeouts, so a rate-limited Opus doesn't block Haiku.
  • Per provider: trips on connection failures and auth errors, or when 2 of its models are open.
  • Timing: 5 consecutive failures within 60 s open a breaker. After a 30 s cooldown, a single probe request decides whether it closes again. When every candidate is open, the gateway answers 503 upstream_unavailable.
• Streams fail over too. Nothing is sent to the client, not even the 200, until the first real content arrives (capped at 3 s or 4 KB). An upstream that dies before that is replaced without the client noticing. After that point, a failure ends the stream with an OpenAI-style error event.

Semantic cache

A repeated question is answered without calling a model (X-Tierwise-Reason: cache). The log records it with cost 0 and the full baseline cost, so it counts toward savings.

• What has to match:
  • The last user message, by meaning: MiniLM embeddings with cosine similarity of at least 0.95. A message over 256 tokens must match exactly.
  • Everything else, exactly: the system prompt, earlier turns, the routing mode and tag, and response_format.
• Skipped when:
  • temperature is above 0.3 or unset (unset means the provider default, 1.0).
  • Tools or images are present.
  • The client sends Cache-Control: no-cache.
  • The model is named explicitly.
• Lifetime: entries live for an hour in memory (10 000 at most, oldest out first). Only complete text answers are stored.
• Threshold: 0.95 is deliberately strict. "What's the capital of France?" and "What is the capital of France?" score 0.91 and miss, while two phrasings of a coding question score 0.98 and hit. Lower CACHE_SIMILARITY for more hits at more risk.
• Off by default. Turn it on with TIERWISE_CACHE=true. It also needs the learned router's ONNX encoder.
• Known limit: sentence embeddings barely register negation or direction. "Is it safe to mix bleach and ammonia?" and "Is it unsafe…" score 0.99; "Convert 100 USD to EUR" and "EUR to USD" score 0.98. Only enable the cache for traffic where such near-duplicates really do share an answer, such as FAQ bots or repeated batch prompts.
• Never matched by meaning: text the encoder can't represent faithfully is matched exactly. That covers emoji, rare symbols, words over 100 characters, and anything over 256 tokens.

Run the full stack

One command starts the gateway with Postgres, Prometheus, Grafana and Jaeger, plus a fake OpenAI-compatible model server, so it runs with no API keys and costs nothing:

docker compose -f deploy/docker-compose.yml up --build
docker compose -f deploy/docker-compose.yml exec gateway tierwise keys create --name me

The dashboard shows:

• savings against an all-flagship baseline
• spend, and requests by model
• p95 latency and stream hold time
• cache hit rate, failovers, open circuits, and errors by status

Behind it:

• Prometheus scrapes the gateway's /metrics endpoint. /metrics has no auth, by Prometheus convention, so keep port 8000 off the public internet (or put a proxy in front of it).
• Jaeger receives one trace per request, with spans for routing, the cache, selection and each provider attempt.

Real models: to use them instead of the fake server, point TIERWISE_REGISTRY at config/registry.yaml and put provider keys in .env.

scripts/compose_smoke.py checks the running stack end to end; CI runs it on every push.

Demo: uv run python scripts/demo.py sends prompts of rising difficulty through the stack and shows what each one cost against always using the flagship:

prompt                                            model   effort   reason              cost $
hi, thanks                                        small   default  rules             0.000005
What is the capital of France?                    small   default  rules             0.000005
Translate 'good morning' into Spanish.            small   default  rules             0.000005
Design a multi-tenant SaaS billing architectu...  large   default  rules             0.000385
Debug this production outage: checkout latenc...  large   default  rules             0.000365
Prove that the square root of 2 is irrational.    large   default  rules             0.000330

total $0.001095 vs $0.001995 always-large: 45% saved

Providers

Model registry

Every servable model lives in config/registry.yaml with its provider, prices (USD per 1M tokens), context window and capabilities. To add a model, add an entry:

  - id: my-model            # the name clients send
    provider: openai
    upstream_id: real-name  # optional, if the provider calls it something else
    price_in: 0.50
    price_out: 1.50
    price_cache_read: 0.05  # optional; defaults to price_in
    context: 128000
    caps: [tools, json]
    quality: 0.6
    tier: basic

The gateway refuses to start if the file is invalid. Verify prices against each provider's official pricing page; the shipped values are a starting point.

Costs and the request log

Every request is logged (SQLite by default, TIERWISE_DB_URL) with tokens, cost and the cost the same tokens would have had on the baseline model. Summarize it with:

uv run python scripts/costs.py

Check every configured provider end to end, plain and streamed:

uv run python scripts/smoke.py

uv run python scripts/smoke.py --stream

See which model auto picks for an easy, a medium and a critical prompt:

uv run python scripts/smoke.py --auto

Development

uv sync
uv run pytest
uv run ruff check .
uv run mypy src

Releasing

Releases go to PyPI from a tag through Trusted Publishing, so no token is stored.

1. Once, on pypi.org, add a pending publisher:

   • project tierwise
   • owner Suryanshu-77, repository Project-1
   • workflow release.yml, environment pypi

2. Once, in the repository settings, create the pypi environment. Add yourself as a required reviewer so every publish waits for your approval.

3. Each release: bump version in pyproject.toml, add a CHANGELOG.md entry, then:

   git tag v0.8.0
   git push origin v0.8.0
   

The workflow checks that the tag matches the version, builds, smoke-tests the installed wheel from an empty directory (scripts/wheel_smoke.py), then publishes.