switchboard-router

Pick the cheapest LLM that can handle each prompt.

switchboard-router asks Switchboard, a 0.5B open router model, which tier a prompt needs: small, mid or frontier. Easy prompts go to a cheap model, and you pay frontier prices only when the prompt needs it.

pip install switchboard-router

Quickstart

You need one way to run the model. The easiest is Ollama: install it and leave it running. If the model isn't in Ollama yet, the first call pulls it (994 MB) from Hugging Face, which takes a minute or two. To do that ahead of time:

ollama pull hf.co/switchboardllm/switchboard-router-0.5b-GGUF

from switchboard_router import Router

router = Router()  # uses Ollama when it's running

router.route("Write a haiku about rain").tier
# 'small'


router.pick_model("Unscramble the following words: eglar, ngareo, kobo")
# 'anthropic/claude-opus-5.5'

From the terminal:

switchboard "Design a rate limiter for 10M requests per second across 5 regions"
{"tier": "mid", "domain": "general", "valid": true, "model": "anthropic/claude-sonnet-5.5"}

Route, then call the model

pick_model returns a model id for the tier. By default these are Claude models on Surplus, which serves every model behind one OpenAI-compatible key:

from openai import OpenAI
from switchboard_router import Router

router = Router()
llm = OpenAI(base_url="https://api.surplusllm.com/api/v1", api_key="sk-surplus-...")

prompt = "Write a Python function that reverses a string"
reply = llm.chat.completions.create(
    model=router.pick_model(prompt),
    messages=[{"role": "user", "content": prompt}],
)

Use your own models per tier (give all three, or the rest fall back to the defaults with a warning):

router = Router(models={"small": "my/small-model", "mid": "my/mid-model", "frontier": "my/frontier-model"})

Put it in front of your app: switchboard serve

Run a local OpenAI-compatible proxy, point your app at it, and send model="auto". No other code changes.

switchboard serve

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="sk-surplus-...")  # your Surplus key
reply = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Write a haiku about rain"}],
)

Each auto request is routed on its last user message, then forwarded to the Surplus API with your key. Streaming works. Any other model name passes straight through, so you can mix auto with fixed models. Every response says what happened in its headers: x-switchboard-tier, x-switchboard-domain and x-switchboard-model.

• Cold starts: launch models answer 503 while their GPU wakes up. Switchboard retries for up to 60 seconds (--cold-start-wait), then sends the request to the tier's default model instead and marks it with x-switchboard-fallback: 1.
• Real costs: with --record, every routed request is written to the savings ledger with what Surplus actually charged (x-surplus-cost-usd). switchboard --stats then reports real spend next to the estimate.
• Keys: your app's key is forwarded as-is. To keep the key on the server instead, use --upstream-key-env SURPLUS_API_KEY.
• Safety: it listens on 127.0.0.1 only. Binding to another address needs --allow-remote, because anyone who can reach the proxy could spend through it. Prompts are never logged or stored.
• Other APIs: --upstream https://... points it at any OpenAI-compatible API.
• Token limits: Claude 5.5 models on Surplus think before they answer, and the thinking counts toward max_tokens. With a small limit (say 120) the visible reply can come back empty, so leave room.

Use it inside vLLM Semantic Router

vLLM Semantic Router can ask a remote scorer how hard a request is. switchboard sr-backend answers those calls, so Switchboard plugs into it with configuration only:

switchboard sr-backend --backend ollama --host 0.0.0.0 --allow-remote --access-key-env SB_KEY

It serves POST /classify on port 8090. Tiers map onto vLLM SR's verdicts: small is easy, mid is medium, frontier is hard.

• Real confidence: with Ollama, vLLM or Transformers, each verdict comes with the probability the model gave it, read from the tier token's logprobs. Other backends send 1.0 for the chosen verdict.
• Contracts: label_distribution.v1 (default) returns a score for each of the three verdicts. score.v1 (--contract score.v1) returns one difficulty number from 0 to 1.
• Failures: if Switchboard can't grade a request it answers 503, never a guessed verdict. vLLM SR then treats the signal as unknown.
• Keys: with --access-key-env, every call must carry that key as a bearer token. Set the same value as access_key in vLLM SR. Listening on anything but loopback (for vLLM SR in Docker) needs both --allow-remote and a key.
• Other flags: --ollama-host if Ollama runs elsewhere, --timeout for the router backend, and --max-concurrent (default 4) for how many requests the backend handles at once.

examples/vllm-sr/config.yaml is a complete config that sends easy requests to Nemotron Nano, medium ones to Nemotron Super and hard ones to Nemotron Ultra, all on the Surplus API. It passes vllm-sr config validate. It has also been tested end to end with vLLM SR 0.4.0 in Docker: each prompt reached the expected model, and the scorer added 70–85 ms per request.

• Keep the model warm: vLLM SR waits 5 seconds (deadline_ms) for a verdict. A cold Ollama model can take longer, and then the request counts as unknown; this config sends unknown requests to the strongest model. Start Ollama with OLLAMA_KEEP_ALIVE=30m to keep the model loaded.
• Surplus and OpenRouter responses: vLLM SR 0.4.0 rejects the extra fields these APIs add to every reply (reasoning, reasoning_details, provider, native_finish_reason) and returns a 502. Routing still works; the reply is what fails. Until vLLM SR accepts those fields, use providers that return plain OpenAI responses, or put a proxy in front that removes the fields.

Send one kind of prompt to a specific model

The router also names a domain (code, math, writing or general). Overrides send one tier and domain to a model of your choice, and everything else keeps the tier default. Failed decisions ignore overrides and always use the frontier model.

router = Router(overrides={"small/code": "launch/Nock-AI/nock-coder-1.5b@18859c0"})

Models launched on the Surplus launchpad are served at a fixed $0.03 per 1M tokens. To see which ones you can call right now:

switchboard --launchpad

Switchboard has not measured how well any launchpad model answers a given tier, so test one on your own prompts before you rely on it.

See what routing saved you

Turn on the ledger and every pick_model call is written to a local file: the time, tier, domain and model. The prompt is never stored, and nothing is sent anywhere.

router = Router(ledger=True)   # ~/.switchboard/ledger.jsonl, or set SWITCHBOARD_LEDGER

switchboard "Summarize this ticket" --record
switchboard --stats

--stats compares what the recorded decisions would cost with sending every one to the frontier model. It's an estimate: it prices output tokens only and assumes every answer is 500 tokens long (change it with --tokens-per-answer).

To try your own prompts without installing anything, paste them into the savings calculator. It runs the router in your browser.

Backends

Backend	Use it when	How
ollama	You run models locally	Router(backend="ollama")
openai	You serve the model with llama-server or vLLM, or call it on the Surplus API	Router(backend="openai", base_url="http://localhost:8080/v1", backend_model="switchboard")
transformers	You want it in-process	pip install 'switchboard-router[local]', then Router(backend="transformers")

Router() with no backend uses Ollama if it's running, then Transformers if it's installed, and otherwise tells you what to install.

The default model is the 0.5B. The same code runs the larger sizes, which were tested on the same 10,000 prompts:

Size	Runs on	backend_model
0.5B	any laptop CPU, or the browser	default
4B	Ollama on your own machine	hf.co/switchboardllm/switchboard-router-4b-GGUF
32B	vLLM on one 48 GB GPU	the name your vLLM server gives it

The 32B sends the fewest hard prompts to a cheap model; the model cards have the numbers.

Failing safe

If the backend can't be reached, errors out, or the model's output isn't valid JSON, the decision is frontier with valid=False and the reason in decision.error. A failed routing call costs you money, never quality. With fail_safe=False, those cases raise instead. Empty or non-string prompts always raise ValueError or TypeError.

Requests time out after 15 seconds by default (Router(timeout=5) to change it), and redirects are not followed, so an API key is never sent to another host.

Limits

• The router reads the first 2,000 characters of a prompt. A question at the very end of a long prompt may be missed.
• It reads the prompt it's given, so a prompt can try to talk it into a cheaper tier. Don't use it as a security boundary.

How good is the routing?

Switchboard v0.1 on 10,000 held-out prompts: 64.5% cheaper than sending every prompt to Claude Opus 5.5, and 5.5% of prompts were sent to a cheaper tier than they needed. Method, limits and the full results are in the model card. Try it in your browser: Switchboard demo.

License

Apache-2.0.