coding-router

A lightweight LLM router that classifies coding requests and selects the best-fit model using semantic similarity and cost-aware scoring.

Based on the LLMRouter paper (Feng et al., 2026):

reward_m = α · perf_normalized(m) − β · cost_normalized(m)

Features

• Local classifier — runs the adapted-arch-router-1.5B classifier locally via a quantised GGUF model (~600 MB, auto-downloaded). No external server needed.
• Semantic routing — embeds your query and every model description with Qwen3-Embedding-0.6B, then ranks by cosine similarity + cost penalty.
• Cost-aware selection — five built-in routing modes from quality-only to cost-dominant, plus arbitrary (α, β) overrides.
• Cross-platform — Metal acceleration on macOS, CPU/CUDA on Windows and Linux.
• Customisable catalog — edit a JSONC file to enable only the models you actually use.

Install

pip install coding-router

Platform notes

Quick start

1. Initialise the model catalog

coding-router init

This copies the bundled model catalog to ~/.config/coding-router/coding_llm_models.jsonc.

2. Enable the models you use

Open ~/.config/coding-router/coding_llm_models.jsonc and uncomment the models you have API keys for (cloud) or running locally (self-hosted):

// Before (disabled):
// "claude-sonnet-5": {
//   "size": "Undisclosed (mid-tier)",
//   ...
// },

// After (enabled):
"claude-sonnet-5": {
  "size": "Undisclosed (mid-tier)",
  ...
},

3. Set API keys

export OPENAI_API_KEY=...
export ANTHROPIC_API_KEY=...
export GOOGLE_API_KEY=...
# etc.

4. Route a query

# Route without calling a provider (dry run):
coding-router --route-only "Fix an intermittent race condition in our Python worker"

# Route and invoke the selected model:
coding-router "Design a caching layer for this API"

Python API

from coding_router import CodingRouter, RouterConfig

router = CodingRouter()
result = router.route("Fix this race condition", route_only=True)

print(result["selected_model"]["catalog_key"])  # e.g. "claude-sonnet-5"
print(result["classifier_category"])            # e.g. "bug_fixing"
print(result["selected_model"]["reward"])        # cost-adjusted score

Custom configuration

from pathlib import Path
from coding_router import CodingRouter, RouterConfig

router = CodingRouter(RouterConfig(
    routing_mode="cost_efficient",       # favour cheaper models
    classifier_backend="local",          # default: local GGUF
    classifier_n_threads=4,              # limit CPU threads
    target_max_tokens=2048,
))

Routing modes

Override with --alpha and --beta for arbitrary sweep points.

Add custom models

# Add a local Ollama model:
coding-router add-model \
  --key my-local-coder \
  --model my-coder-model \
  --endpoint http://localhost:11434/v1 \
  --feature "Fast local coding model for Python and TypeScript" \
  --size "14B" \
  --input-price 0 --output-price 0

# Add a cloud model:
coding-router add-model \
  --key team-cloud-coder \
  --model provider-coder-v1 \
  --endpoint https://provider.example/v1 \
  --feature "Advanced agentic model for refactors and migrations" \
  --size "Unknown" --tier advanced \
  --input-price 0.8 --output-price 3.2 \
  --api-key-env TEAM_CLOUD_CODER_API_KEY

Use --pool user to route only among your added models.

Classifier backends

Local GGUF (default)

The classifier model is automatically downloaded on first use (~600 MB) and cached at ~/.cache/coding-router/. No separate server required.

# Override the default GGUF path:
coding-router --classifier-model-path /path/to/custom.gguf "your query"

Remote vLLM (advanced)

If you already have a vLLM server running the classifier:

coding-router --classifier-backend vllm --classifier-base-url http://localhost:8000/v1 "your query"

Environment variables

License

MIT