smart-llm-router

Provider-agnostic LLM router. Pick the cheapest capable model per prompt with rule-based scoring. Wraps LiteLLM for format conversion, streaming, tool calls, and 100+ provider integrations.

Why

Every LLM proxy today routes based on a model name you pick. This one picks the model for you — locally, in <1ms, with zero ML — by scoring the prompt across 14 dimensions (code presence, reasoning markers, multi-step patterns, multilingual keywords, etc.) and mapping to one of four tiers (SIMPLE / MEDIUM / COMPLEX / REASONING).

You bring an upstream (OpenRouter, Together, Fireworks, Groq, Anthropic direct, vLLM, Ollama — anything OpenAI-compatible). It does the rest.

Install

pip install smart-llm-router

Two console scripts ship with the package: smart-llm-router (full name) and slr (short alias).

Quick start with OpenRouter (default upstream)

# 1. Get an OpenRouter key at https://openrouter.ai/keys
export OPENROUTER_API_KEY=sk-or-v1-...
export LITELLM_MASTER_KEY=sk-anything    # gates the proxy itself

# 2. Start the proxy on :4000 (uses bundled OpenRouter config by default)
smart-llm-router start

In another terminal — any OpenAI-compatible client works:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:4000/v1", api_key="sk-anything")

# Smart routing — rule-based scorer picks the cheapest capable model
resp = client.chat.completions.create(
    model="smart/auto",
    messages=[{"role": "user", "content": "prove that sqrt(2) is irrational step by step"}],
)
# → routed to REASONING tier (e.g. deepseek/deepseek-r1)

Or curl:

curl http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-anything" \
  -H "Content-Type: application/json" \
  -d '{"model":"smart/auto","messages":[{"role":"user","content":"hi"}]}'

Inspect routing without dispatching

slr test "what is the capital of france"
# → SIMPLE / google/gemini-2.5-flash-lite / 100% savings vs claude-sonnet-4.6

slr test "Prove that sqrt(2) is irrational step by step"
# → REASONING / deepseek/deepseek-r1 / 90% savings

slr test "design a high-availability microservices architecture" --profile premium
# → COMPLEX / anthropic/claude-opus-4.7

slr models --profile auto    # show the tier→model table

Pointing at a different upstream

The bundled config targets OpenRouter, but anything OpenAI-compatible works (Together, Fireworks, Groq, DeepInfra, vLLM, Ollama, OpenAI direct). Copy the bundled YAML and edit api_base / api_key:

# Copy the bundled config to your working directory
python -c "from importlib.resources import files; import shutil; shutil.copy(files('smart_llm_router') / 'default_config.yaml', './smart-llm-router.yaml')"

# Edit smart-llm-router.yaml — swap api_base / api_key per model_list entry
# Then start with --config
smart-llm-router start --config smart-llm-router.yaml

Available routing profiles

Pin a specific model (no routing)

Pass a concrete model ID and the router leaves it alone:

client.chat.completions.create(
    model="anthropic/claude-sonnet-4.6",   # always Sonnet
    messages=[...]
)

client.chat.completions.create(
    model="anthropic/claude-opus-4.7",     # always Opus
    messages=[...]
)

client.chat.completions.create(
    model="openai/gpt-4o",                 # always GPT-4o
    messages=[...]
)

Models pre-wired in the bundled config: anthropic/claude-haiku-4.5, anthropic/claude-sonnet-4.6, anthropic/claude-opus-4.6, anthropic/claude-opus-4.7, openai/gpt-4o, openai/gpt-4o-mini, openai/o1, openai/o3, openai/o3-mini, openai/o4-mini, google/gemini-2.5-flash-lite, google/gemini-2.5-flash, google/gemini-2.5-pro, google/gemini-2.0-flash-lite-001, deepseek/deepseek-chat, deepseek/deepseek-r1, meta-llama/llama-3.3-70b-instruct. Add more by editing the model_list in your config YAML.

Use with Claude Code

Claude Code respects ANTHROPIC_BASE_URL. Point it at the proxy:

export ANTHROPIC_BASE_URL=http://127.0.0.1:4000
export ANTHROPIC_AUTH_TOKEN=sk-anything   # the proxy's master key
claude

Then inside Claude Code: /model anthropic/claude-opus-4.7 to pin Opus, or /model smart/premium to let the router pick the best Claude per request.

How it works

1. Client sends OpenAI/Anthropic/Gemini-format request to localhost:4000.
2. LiteLLM Proxy parses; SmartRouterHook.async_pre_call_hook intercepts.
3. If model is a smart/* profile, the rule-based router scores the prompt and picks a concrete upstream model ID.
4. LiteLLM dispatches to the configured upstream — handling format conversion, streaming, tool calls, retries, etc.

Routing internals

The classifier (smart_llm_router/router/rules.py) scores each prompt across 14 weighted dimensions:

Keyword sets are multilingual — EN + ZH + JA + RU + DE + ES + PT + KO + AR — so the same scorer works across 9 languages without translation.

The score maps to a tier through three boundaries:

< 0.0   → SIMPLE        0.3-0.5 → COMPLEX
0.0-0.3 → MEDIUM        > 0.5   → REASONING

Plus three hard overrides: 2+ reasoning keywords ⇒ force REASONING; >100k tokens ⇒ force COMPLEX; system prompt mentioning json/schema ⇒ floor at MEDIUM.

Attribution

The 14-dimension rule-based router in smart_llm_router/router/ is ported from ClawRouter (MIT). Format conversion and streaming come from LiteLLM (MIT).

License

MIT