xrouter-llm

Stop sending every prompt to your most expensive LLM.

xrouter-llm is a prompt-aware LLM routing-decision service: it predicts which models can complete a prompt, then chooses the cheapest model that clears the bar. On our tested dataset, it cuts realized cost by 53.2% while improving completion by +1.9 pts.

It answers "which model should serve this prompt?" and records the choice — it does NOT call the underlying LLMs.

Install

pip install xrouter-llm        # ships a trained router + model registry
# or, for development:
pip install -e ".[dev]"

The wheel bundles a trained router artifact, the model-profile registry, and the router configs, so a fresh install can serve immediately with no extra files.

Serve

The bundled router, registry, and configs are the defaults, so a bare invocation works out of the box:

xrouter-llm serve --port 8080

Override any of them to use your own trained model or registry:

xrouter-llm serve \
  --model artifacts/models/irt_router_350k.joblib \
  --models-dir path/to/models --routers-dir path/to/routers \
  --db artifacts/calls.db --port 8080

• GET / — single-page UI (prompt box, config picker, decision table, history)
• GET /api/configs, POST /api/route ({prompt, config, task?, preferred_input_modalities?}), GET /api/history?limit=N
• Route candidates include their input_modalities; responses also report whether the requested input-modality preference narrowed the candidate pool.
• Every decision is logged to SQLite (*.db/*.sqlite are gitignored — the log holds user prompts).

Xinference embeddings

IRTRouter can use a Xinference embedding model through its OpenAI-compatible /v1/embeddings endpoint. For best calibration, train the router with the same embedding backend you will serve:

PYTHONPATH=src python3 -m xrouter_llm.cli train-irt \
  --embedding-backend xinference \
  --embedding-model bge-m3 \
  --xinference-base-url http://127.0.0.1:9997/v1 \
  --dataset llmrouterbench:data/raw/llmrouterbench_stream_sample_350k \
  --benchmark-profiles artifacts/profiles/llmrouterbench_350k_profiles.json,src/xrouter_llm/resources/config/models \
  --output artifacts/models/irt_router_xinference.joblib

If the loaded artifact was trained with the same embedding model/dimension, the serve command can replace the serialized backend at startup:

xrouter-llm serve \
  --model artifacts/models/irt_router_xinference.joblib \
  --override-embedding-backend \
  --embedding-backend xinference \
  --embedding-model bge-m3 \
  --xinference-base-url http://127.0.0.1:9997/v1

Model registry

One YAML per supported model, bundled under src/xrouter_llm/resources/config/models/ (capability profile: provider, costs, context, published benchmarks as 0-100 percentages). model_id is the model's canonical OpenRouter slug (e.g. anthropic/claude-opus-5). The bundled registry is the default for --benchmark-profiles; point it at your own directory or file to extend it. Add a model = add a file.

Profiles with time-varying provider prices can define utc_price_overrides. Serving resolves the active input/output rate once per request using the current UTC time; the scalar input_cost_per_1k / output_cost_per_1k values remain the fallback outside those windows. Windows are start-inclusive and end-exclusive, and cannot use the same start and end time. Windows without utc_days may wrap across midnight. Optional utc_days entries use full weekday names; omitted means every day. A weekday-scoped window cannot wrap across midnight; split it into separate windows on the adjacent UTC days. Windows whose day sets intersect must not overlap. Quote YAML clock values (for example, "01:00") for portability; the loader also accepts PyYAML's unquoted sexagesimal representation as minutes since midnight.

Offline evaluation deliberately uses the scalar fallback price so repeated runs remain deterministic. For profiles whose scalar is an off-peak rate, offline decision cost therefore understates peak-hour serving cost and may produce a different routing distribution from production during those windows.

from xrouter_llm import IRTRouter, default_model_path, default_models_dir, load_benchmark_profiles

router = IRTRouter.load(default_model_path())
for profile in load_benchmark_profiles(default_models_dir()).profiles():
    router.add_benchmark_profile(profile)

preds = router.predict(
    "Design a distributed consensus algorithm",
    model_ids=["anthropic/claude-opus-5", "deepseek/deepseek-v4.1-flash"],
)
print({p.model_id: round(p.mu, 3) for p in preds})

How it works

Do not train:  prompt -> selected model
Train:         prompt + model -> probability the model completes the prompt
Decide:        predicted completion + cost -> cheapest model that can complete

Completion is factored into two decoupled axes (an IRT-style model):

P(complete) = sigmoid(a * capability(model) + b * difficulty(prompt) + c)

• capability(model) = the mean of the model's published gpqa_diamond and livecodebench (both full-coverage on the training side). Going wider doesn't help at this data scale — a flat mean dilutes and learned weights overfit at 37 profiled models; see AGENTS.md "Capability benchmarks". Used directly, so a brand-new model's benchmarks drive its ranking.
• difficulty(prompt) = a Ridge regressor on a multilingual embedding (Qwen/Qwen3-Embedding-0.6B), trained on each prompt's empirical pass-rate. Multilingual (Chinese transfers from English training data). Picked over bge-m3 by a controlled probe (scripts/probe_qwen_difficulty.py): higher held-out Pearson and it no longer rates trivial prompts ("1+1=?") as maximally hard.

This factoring is the key lesson: a single joint classifier could not rank unseen models by their benchmarks (on this data, model capability barely explains completion marginally — but it does once difficulty is controlled, which is exactly what the factored model exploits).

Datasets

The production difficulty model is trained on multiple datasets combined (all feed the difficulty axis; only profiled models feed the capability axis):

The current artifact trains on LLMRouterBench 350k + Terminal-Bench + SWE-bench Verified + xagent labels (378,397 rows / ~14,463 prompts / 287 subjects). The agentic matrices come from agent-psychometrics (MIT) via agentic.py. In IRTRouter, only the 37 profiled llmrouterbench models feed the capability axis and agentic subjects feed difficulty only. RouterBench (withmartian/routerbench) remains a smaller legacy baseline. Local datasets and trained artifacts are not committed (data/, artifacts/ are gitignored).

Adding more agentic prompt types (e.g. your own traffic) is the only way to make difficulty accurate for task mixes outside coding/terminal — see AGENTS.md.

Train

xrouter-llm train-irt \
  --dataset llmrouterbench:data/raw/llmrouterbench_stream_sample_350k \
  --dataset agentic:agentic/terminalbench \
  --dataset agentic:agentic/swebench_verified \
  --dataset xagent-labels:Xorbits/xagent-xrouter-labels:full \
  --benchmark-profiles artifacts/profiles/llmrouterbench_350k_profiles_priority_collected.json,src/xrouter_llm/resources/config/models \
  --output artifacts/models/irt_router_350k.joblib

Diagnostics: sweep-thresholds (cost/completion frontier + calibration) and eval-model-holdout (leave-one-model-out generalization).

Components

• IRTRouter (irt_router.py): conservative production baseline (difficulty x capability).
• RoutingPolicy (policy.py): "cheapest model whose predicted completion clears completion_threshold; else the cheapest within fallback_quality_margin of the best predicted completion".
• serving.py / server.py: HTTP routing-decision API + single-page web UI.
• resources/config/models/: a per-model YAML registry of capability profiles and supported input modalities. Routing can prefer compatible models while falling back to the full candidate set when none match (bundled in the package; resolve with default_models_dir()).
• resources/config/routers/: named "auto configs" — a candidate model set + policy (bundled; default_routers_dir()).
• resources/models/irt_router_350k.joblib: the trained router shipped with the package (default_model_path()).

License

xrouter-llm is released under the Xagent Source License (© Xorbits Inc.) — see LICENSE. It is source-available, not an OSI-approved open source license.

The license text is shared verbatim with Xagent; for this project the licensed "Software" is xrouter-llm, and the "Restricted Functionality" / hosted-service and competitive-use clauses apply to its routing-decision and model-selection capabilities. In short: use, modification, and internal/single-tenant deployment are permitted; offering it as a multi-tenant hosted/managed service, or a directly competing service, is not. See LICENSE for the controlling terms.