switchboard-router

Pick the cheapest LLM that can handle each prompt.

switchboard-router asks Switchboard, a 0.5B open router model, which tier a prompt needs: small, mid or frontier. Easy prompts go to a cheap model, and you pay frontier prices only when the prompt needs it.

pip install switchboard-router

Quickstart

You need one way to run the model. The easiest is Ollama: install it and leave it running. If the model isn't in Ollama yet, the first call pulls it (994 MB) from Hugging Face, which takes a minute or two. To do that ahead of time:

ollama pull hf.co/switchboardllm/switchboard-router-0.5b-GGUF

from switchboard_router import Router

router = Router()  # uses Ollama when it's running

router.route("Write a haiku about rain").tier
# 'small'


router.pick_model("Unscramble the following words: eglar, ngareo, kobo")
# 'anthropic/claude-opus-5.5'

From the terminal:

switchboard "Design a rate limiter for 10M requests per second across 5 regions"
{"tier": "mid", "domain": "general", "valid": true, "model": "anthropic/claude-sonnet-5.5"}

Route, then call the model

pick_model returns a model id for the tier. By default these are Claude models, named the way OpenRouter names them, so one OpenAI-compatible key reaches all three:

from openai import OpenAI
from switchboard_router import Router

router = Router()
llm = OpenAI(base_url="https://openrouter.ai/api/v1", api_key="sk-or-...")

prompt = "Write a Python function that reverses a string"
reply = llm.chat.completions.create(
    model=router.pick_model(prompt),
    messages=[{"role": "user", "content": prompt}],
)

Use your own models per tier (give all three, or the rest fall back to the defaults with a warning):

router = Router(models={"small": "my/small-model", "mid": "my/mid-model", "frontier": "my/frontier-model"})

Put it in front of your app: switchboard serve

Run a local OpenAI-compatible proxy, point your app at it, and send model="auto". No other code changes.

switchboard serve

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="sk-or-...")  # your OpenRouter key
reply = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Write a haiku about rain"}],
)

Each auto request is routed on its last user message, then forwarded to OpenRouter with your key. Streaming works. Any other model name passes straight through, so you can mix auto with fixed models. Every response says what happened in its headers: x-switchboard-tier, x-switchboard-domain and x-switchboard-model.

• Other APIs: --upstream https://... points it at any OpenAI-compatible API: OpenAI, Together, a local vLLM or LiteLLM, or Surplus. Set --override or models= to the model names that API uses.
• Keys: your app's key is forwarded as-is. To keep the key on the server instead, use --upstream-key-env OPENROUTER_API_KEY. A key that belongs to one known provider (sk-or-, sk-surplus-) is never sent to the other: the request is refused with a 400 instead.
• Cold starts: Surplus launchpad models answer 503 while their GPU wakes up. Switchboard retries for up to 60 seconds (--cold-start-wait), then sends the request to the tier's default model instead and marks it with x-switchboard-fallback: 1.
• Costs: with --record, every routed request is written to the savings ledger, and switchboard --stats estimates what routing saved. When the upstream reports the real charge (Surplus does, in x-surplus-cost-usd), the ledger records that too.
• Safety: it listens on 127.0.0.1 only. Binding to another address needs --allow-remote, because anyone who can reach the proxy could spend through it. Prompts are never logged or stored.
• Token limits: on some APIs (Surplus, for one) Claude 5.5 models think before they answer, and the thinking counts toward max_tokens. With a small limit (say 120) the visible reply can come back empty, so leave room.

Give each agent a budget

Agents call models in loops, and a loop can run up a bill fast. switchboard serve can keep a spend limit per agent:

switchboard serve --budget trader=5/day --budget research=50/month --budget '*=1/day'

Each request names its agent in the X-Switchboard-Agent header (no header means the agent default). * gives every other agent its own meter with that limit.

client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="sk-or-...",
                default_headers={"X-Switchboard-Agent": "trader"})

Spent this period	What happens	Header
under 80%	routes as usual	x-switchboard-budget: ok
80% or more (--economy-at)	frontier is capped to mid, so the rest lasts longer	economy, plus x-switchboard-tier-capped: frontier>mid
100%	refused with 429 budget_exceeded, before anything is sent upstream

• What counts: the cost the API reports when it does, otherwise the response's token usage times list prices. For streams, Switchboard asks for usage (stream_options.include_usage). Unknown models are charged at the most expensive known price, so a budget can't be overrun by a price Switchboard doesn't know. If no usage comes back at all, it charges an estimate and marks it as one.
• Named models count too: a request that names its model is charged to the agent, and refused once the budget is used up. An agent can't get around its budget by asking for Opus directly.
• Keep going on small instead: --on-exhausted small sends a spent agent's requests to the small tier instead of refusing them. --economy-at 0 turns off the economy step.
• Restarts don't reset it: spend is written to the savings ledger (agent, model, cost; never the prompt) and read back at start. switchboard --stats --by-agent shows today, this month and the total per agent.
• Requests at the same time: each request reserves an estimate of its cost when it starts (from its size and max_tokens), and the real cost replaces the reservation when it ends. Parallel requests can't all slip under the limit; the one that crosses it can still overshoot by its own cost.
• Every request that reaches the API is charged once, including streams the client abandons. Switchboard reads the rest of the answer to find its usage, because the API bills for it anyway.
• Trust: agent names are case-insensitive and come from a header, so any caller can pick one. Budgets limit your own agents; they aren't access control. Keep the proxy on 127.0.0.1 (the default) or behind your own auth.
• Named models count against the budget and are refused at the limit, but economy mode only changes what auto picks.

Route each step of an agent loop

An agent calls a model at every step, and most steps don't need the biggest one. StepRouter reads the tools the agent has and the conversation so far, and picks the tier for the next step. It uses its own model, switchboard-agent-router-0.5b, trained on agent traces, which Ollama pulls on first use.

from switchboard_router import StepRouter

router = StepRouter(frontier_threshold=0.2)
model = router.pick_model(messages, tools)   # the same messages and tools you send to the agent's model
response = client.chat.completions.create(model=model, messages=messages, tools=tools)

• Inputs: OpenAI-style chat messages (with tool_calls and tool results) and tool schemas. The router reads a compact text made by render_step(messages, tools): the tool list, the task and the latest turns, at most 5,000 characters.
• Tiers: in training, small meant Qwen3-1.7B made the right tool call, mid meant Qwen3-8B did, and frontier meant neither did. models= maps them to your own models, as with Router.
• Safer picks: frontier_threshold sends a step to frontier whenever the model gives frontier at least that probability. It reads the model's logprobs, which the Ollama, OpenAI-compatible and Transformers backends all return. A custom backend without generate_scored is refused with a ValueError, and a step whose logprobs can't be read keeps the model's own pick.
• Per-domain models: overrides= works as with Router. Step decisions carry the domain general, so the keys are small/general, mid/general and frontier/general.
• How good: on 2,805 held-out steps from tasks it never saw, the model's own pick sent 13.5% of steps to a tier that was too small and saved 31.4% against always using frontier. With frontier_threshold=0.2, that's 5.8% and 19.2%. All the numbers are on the model card.
• Failures (backend down, unreadable output) route to frontier, unless fail_safe=False.

Use it inside vLLM Semantic Router

vLLM Semantic Router can ask a remote scorer how hard a request is. switchboard sr-backend answers those calls, so Switchboard plugs into it with configuration only:

switchboard sr-backend --backend ollama --host 0.0.0.0 --allow-remote --access-key-env SB_KEY

It serves POST /classify on port 8090. Tiers map onto vLLM SR's verdicts: small is easy, mid is medium, frontier is hard.

• Real confidence: with Ollama, vLLM or Transformers, each verdict comes with the probability the model gave it, read from the tier token's logprobs. Other backends send 1.0 for the chosen verdict.
• Contracts: label_distribution.v1 (default) returns a score for each of the three verdicts. score.v1 (--contract score.v1) returns one difficulty number from 0 to 1.
• Failures: if Switchboard can't grade a request it answers 503, never a guessed verdict. vLLM SR then treats the signal as unknown.
• Keys: with --access-key-env, every call must carry that key as a bearer token. Set the same value as access_key in vLLM SR. Listening on anything but loopback (for vLLM SR in Docker) needs both --allow-remote and a key.
• Other flags: --ollama-host if Ollama runs elsewhere, --timeout for the router backend, and --max-concurrent (default 4) for how many requests the backend handles at once.

examples/vllm-sr/config.yaml is a complete config that sends easy requests to Nemotron Nano, medium ones to Nemotron Super and hard ones to Nemotron Ultra, all through OpenRouter. It passes vllm-sr config validate. It has also been tested end to end with vLLM SR 0.4.0 in Docker: each prompt reached the expected model, and the scorer added 70–85 ms per request.

• Keep the model warm: vLLM SR waits 5 seconds (deadline_ms) for a verdict. A cold Ollama model can take longer, and then the request counts as unknown; this config sends unknown requests to the strongest model. Start Ollama with OLLAMA_KEEP_ALIVE=30m to keep the model loaded.
• OpenRouter-style responses: vLLM SR 0.4.0 rejects the extra fields these APIs add to every reply (reasoning, reasoning_details, provider, native_finish_reason) and returns a 502. Routing still works; the reply is what fails. Until vLLM SR accepts those fields, use providers that return plain OpenAI responses, or put a proxy in front that removes the fields.

Send one kind of prompt to a specific model

The router also names a domain (code, math, writing or general). Overrides send one tier and domain to a model of your choice, and everything else keeps the tier default. Failed decisions ignore overrides and always use the frontier model.

router = Router(overrides={"small/code": "my/code-model"})

If you use Surplus: models launched on its launchpad are served at a fixed $0.03 per 1M tokens. To see which ones you can call right now (run serve with --upstream https://api.surplusllm.com/api/v1 to reach them):

switchboard --launchpad

Switchboard has not measured how well any launchpad model answers a given tier, so test one on your own prompts before you rely on it.

See what routing saved you

Turn on the ledger and every pick_model call is written to a local file: the time, tier, domain and model. The prompt is never stored, and nothing is sent anywhere.

router = Router(ledger=True)   # ~/.switchboard/ledger.jsonl, or set SWITCHBOARD_LEDGER

switchboard "Summarize this ticket" --record
switchboard --stats

--stats compares what the recorded decisions would cost with sending every one to the frontier model. It's an estimate: it prices output tokens only and assumes every answer is 500 tokens long (change it with --tokens-per-answer).

To try your own prompts without installing anything, paste them into the savings calculator. It runs the router in your browser.

Backends

Backend	Use it when	How
ollama	You run models locally	Router(backend="ollama")
openai	You serve the model with llama-server or vLLM, or call it on a hosted API	Router(backend="openai", base_url="http://localhost:8080/v1", backend_model="switchboard")
transformers	You want it in-process	pip install 'switchboard-router[local]', then Router(backend="transformers")

Router() with no backend uses Ollama if it's running, then Transformers if it's installed, and otherwise tells you what to install.

The default model is the 0.5B. The same code runs the larger sizes, which were tested on the same 10,000 prompts:

Size	Runs on	backend_model
0.5B	any laptop CPU, or the browser	default
4B	Ollama on your own machine	hf.co/switchboardllm/switchboard-router-4b-GGUF
32B	vLLM on one 48 GB GPU	the name your vLLM server gives it

The 32B sends the fewest hard prompts to a cheap model; the model cards have the numbers.

Failing safe

If the backend can't be reached, errors out, or the model's output isn't valid JSON, the decision is frontier with valid=False and the reason in decision.error. A failed routing call costs you money, never quality. With fail_safe=False, those cases raise instead. Empty or non-string prompts always raise ValueError or TypeError.

Requests time out after 15 seconds by default (Router(timeout=5) to change it), and redirects are not followed, so an API key is never sent to another host.

Limits

• The router reads the first 2,000 characters of a prompt. A question at the very end of a long prompt may be missed.
• It reads the prompt it's given, so a prompt can try to talk it into a cheaper tier. Don't use it as a security boundary.

How good is the routing?

Switchboard v0.1 on 10,000 held-out prompts: 64.5% cheaper than sending every prompt to Claude Opus 5.5, and 5.5% of prompts were sent to a cheaper tier than they needed. Method, limits and the full results are in the model card. Try it in your browser: Switchboard demo.

License

Apache-2.0.