switchboard-router

Pick the cheapest LLM that can handle each prompt.

switchboard-router asks Switchboard, a 0.5B open router model, which tier a prompt needs: small, mid or frontier. Easy prompts go to a cheap model, and you pay frontier prices only when the prompt needs it.

pip install switchboard-router

Quickstart

You need one way to run the model. The easiest is Ollama: install it and leave it running. If the model isn't in Ollama yet, the first call pulls it (994 MB) from Hugging Face, which takes a minute or two. To do that ahead of time:

ollama pull hf.co/switchboardllm/switchboard-router-0.5b-GGUF

from switchboard_router import Router

router = Router()  # uses Ollama when it's running

router.route("Write a haiku about rain").tier
# 'small'


router.pick_model("Unscramble the following words: eglar, ngareo, kobo")
# 'anthropic/claude-opus-5.5'

From the terminal:

switchboard "Design a rate limiter for 10M requests per second across 5 regions"
{"tier": "mid", "domain": "general", "valid": true, "model": "anthropic/claude-sonnet-5.5"}

Route, then call the model

pick_model returns a model id for the tier. By default these are Claude models on Surplus, which serves every model behind one OpenAI-compatible key:

from openai import OpenAI
from switchboard_router import Router

router = Router()
llm = OpenAI(base_url="https://api.surplusllm.com/api/v1", api_key="sk-surplus-...")

prompt = "Write a Python function that reverses a string"
reply = llm.chat.completions.create(
    model=router.pick_model(prompt),
    messages=[{"role": "user", "content": prompt}],
)

Use your own models per tier (give all three, or the rest fall back to the defaults with a warning):

router = Router(models={"small": "my/small-model", "mid": "my/mid-model", "frontier": "my/frontier-model"})

Put it in front of your app: switchboard serve

Run a local OpenAI-compatible proxy, point your app at it, and send model="auto". No other code changes.

switchboard serve

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="sk-surplus-...")  # your Surplus key
reply = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Write a haiku about rain"}],
)

Each auto request is routed on its last user message, then forwarded to the Surplus API with your key. Streaming works. Any other model name passes straight through, so you can mix auto with fixed models. Every response says what happened in its headers: x-switchboard-tier, x-switchboard-domain and x-switchboard-model.

• Cold starts: launch models answer 503 while their GPU wakes up. Switchboard retries for up to 60 seconds (--cold-start-wait), then sends the request to the tier's default model instead and marks it with x-switchboard-fallback: 1.
• Real costs: with --record, every routed request is written to the savings ledger with what Surplus actually charged (x-surplus-cost-usd). switchboard --stats then reports real spend next to the estimate.
• Keys: your app's key is forwarded as-is. To keep the key on the server instead, use --upstream-key-env SURPLUS_API_KEY.
• Safety: it listens on 127.0.0.1 only. Binding to another address needs --allow-remote, because anyone who can reach the proxy could spend through it. Prompts are never logged or stored.
• Other APIs: --upstream https://... points it at any OpenAI-compatible API.

Send one kind of prompt to a specific model

The router also names a domain (code, math, writing or general). Overrides send one tier and domain to a model of your choice, and everything else keeps the tier default. Failed decisions ignore overrides and always use the frontier model.

router = Router(overrides={"small/code": "launch/Nock-AI/nock-coder-1.5b@18859c0"})

Models launched on the Surplus launchpad are served at a fixed $0.03 per 1M tokens. To see which ones you can call right now:

switchboard --launchpad

Switchboard has not measured how well any launchpad model answers a given tier, so test one on your own prompts before you rely on it.

See what routing saved you

Turn on the ledger and every pick_model call is written to a local file: the time, tier, domain and model. The prompt is never stored, and nothing is sent anywhere.

router = Router(ledger=True)   # ~/.switchboard/ledger.jsonl, or set SWITCHBOARD_LEDGER

switchboard "Summarize this ticket" --record
switchboard --stats

--stats compares what the recorded decisions would cost with sending every one to the frontier model. It's an estimate: it prices output tokens only and assumes every answer is 500 tokens long (change it with --tokens-per-answer).

To try your own prompts without installing anything, paste them into the savings calculator. It runs the router in your browser.

Backends

Backend	Use it when	How
ollama	You run models locally	Router(backend="ollama")
openai	You serve the model with llama-server or vLLM, or call it on the Surplus API	Router(backend="openai", base_url="http://localhost:8080/v1", backend_model="switchboard")
transformers	You want it in-process	pip install 'switchboard-router[local]', then Router(backend="transformers")

Router() with no backend uses Ollama if it's running, then Transformers if it's installed, and otherwise tells you what to install.

Failing safe

If the backend can't be reached, errors out, or the model's output isn't valid JSON, the decision is frontier with valid=False and the reason in decision.error. A failed routing call costs you money, never quality. With fail_safe=False, those cases raise instead. Empty or non-string prompts always raise ValueError or TypeError.

Requests time out after 15 seconds by default (Router(timeout=5) to change it), and redirects are not followed, so an API key is never sent to another host.

Limits

• The router reads the first 2,000 characters of a prompt. A question at the very end of a long prompt may be missed.
• It reads the prompt it's given, so a prompt can try to talk it into a cheaper tier. Don't use it as a security boundary.

How good is the routing?

Switchboard v0.1 on 10,000 held-out prompts: 64.5% cheaper than sending every prompt to Claude Opus 5.5, and 5.5% of prompts were sent to a cheaper tier than they needed. Method, limits and the full results are in the model card. Try it in your browser: Switchboard demo.

License

Apache-2.0.