EvalRouter CLI

Evaluate models in a full-screen terminal workspace or from scripts. Browse benchmarks, choose models, optionally set a spending cap, follow runs, compare or export their results, and author your own benchmarks. The CLI is built on the evalrouter-sdk Python SDK, which it installs at exactly the same version.

Requires Python 3.12 or 3.13. Distributed under the proprietary Kimpton EvalRouter SDK License, included as LICENSE in the package.

Install and create your account

We recommend uv to install the CLI in its own Python environment:

uv tool install evalrouter
evalrouter

With pip, use python -m pip install evalrouter in a Python 3.12 or 3.13 environment. uv selects a compatible Python for the CLI and downloads one if needed. If your shell cannot find evalrouter, run uv tool update-shell and restart your terminal. Upgrade an existing installation with evalrouter upgrade (or evalrouter --upgrade). Stable installations stay on stable unless you pass --pre; installed prereleases include prereleases automatically. The command uses uv, pipx or pip in the current virtual environment. Source installs, system Python and installer-pinned versions need their original installer. Check the installed version with evalrouter --version. Use evalrouter --help for explicit commands, or see the CLI guide.

The interactive workspace checks for newer releases in the background and shows an update notice when one is available. The check never installs an update or delays opening the workspace. Set EVALROUTER_NO_UPDATE_CHECK=1 to disable it. Scripts, editable installs and explicitly pinned installations skip update checks.

Complete command names with Tab

For zsh, add this line to ~/.zshrc after your compinit setup, then start a new terminal:

eval "$(evalrouter completion zsh)"

For bash, add this line to ~/.bashrc:

eval "$(evalrouter completion bash)"

For fish, run evalrouter completion fish > ~/.config/fish/completions/evalrouter.fish.

Tab completes commands, subcommands, flags and their choices, plus benchmark and model names, ACP Registry agents, connection names, profiles and recent run IDs. Names come from a local cache that a background process refreshes with read-only list requests; pressing Tab never waits for the API.

This package was previously published as kimpton-evalrouter-sdk. The command stays evalrouter. Use from evalrouter import EvalRouter for new integrations; legacy kimpton_evalrouter imports remain compatible and refer to the same public classes. To switch, run uv tool uninstall kimpton-evalrouter-sdk (or pip uninstall -y kimpton-evalrouter-sdk evalrouter) and install evalrouter.

The package supplies the evalrouter command; no repository checkout is needed. Create an EvalRouter account, verify your email, and follow the browser sign-in below. Package installation does not grant account access or evaluation credit.

Sign in with your account:

evalrouter login
evalrouter whoami

Follow the browser sign-in; on a local computer the browser returns to the waiting CLI. For a remote terminal, use evalrouter login --no-browser and approve the code on a device with a browser. New accounts finish setup there. The CLI selects your only workspace automatically, or lets you choose one. Production is the default API; evalrouter options lists --base-url and the other settings that apply to every command. No API key or credential-storage setup is needed for a normal sign-in. The optional evalrouter[keyring] extra enables a compatible OS keychain backend; otherwise credentials use a private file readable only by your user. In the workspace, Account and workspace → Sign in starts the same flow. See the sign-in guide for workspace switching, stored API keys and signing out.

CI and services can use EVALROUTER_API_KEY and EVALROUTER_WORKSPACE_ID supplied privately by a secret manager. Never put credentials in arguments, URLs, request files or logs.

Interactive workspace

Run evalrouter without arguments in a terminal to open the full-screen workspace. Choose Account and workspace → Sign in to connect your account. Local sign-in returns from the browser to the waiting CLI; for a remote terminal, use evalrouter login --no-browser and approve the code on a device with a browser.

Run a benchmark guides you through the benchmark, your selected compatible models, each model's reasoning settings and coverage. Type to filter lists; Space or Enter toggles a model and Tab continues. A one-model evaluation shows its quote and starts when you confirm. A spending cap (--max-cost) is optional; without one the run is bounded by your wallet credit. Each run is one model on one benchmark; to compare models, run each one and compare the results.

Choose a suggested sample, Custom sample size, or Full benchmark for coverage. A custom sample accepts 1 to 1,000 tasks, limited to the benchmark's task count when known. Full benchmark includes every task.

Home includes Run history and Results, plus model and benchmark browsing, billing and account settings. Results offers Single run, Saved result group and Compare runs. While watching, choose Open live run in browser with Enter, B or a click. Live status continues in the terminal.

Escape goes back; Left also goes back when you are not editing text. Backspace edits text and filters. Back controls stay on Home at the root, and returning to Home preserves its selection. Ctrl+Q or the Exit action exits. Leaving a live run stops watching only: submitted server work continues. Use PgUp/PgDn for long reviews and results, and F1 for keyboard help. Select output with the mouse and press Ctrl+C to copy; with no selection, Ctrl+C goes back. Password fields cannot be copied. EVALROUTER_TUI=off selects the older interactive shell; explicit commands, plain output and JSON remain available.

Run a model benchmark

For a shared native benchmark, the single-benchmark model picker lists the routes accepted by that exact version's catalog checks. Its explanation names fixed sampling or native continuation scoring requirements where applicable; the listed models are alternatives. A named excluded route reports the API's reason. Unavailable or unknown discovery remains explicit, and launch checks current eligibility again. Private benchmarks and registry evals without this scoped catalog check retain their qualification step before launch.

Sign in, choose a ready benchmark and compatible model, then run:

evalrouter login
evalrouter benchmarks list
evalrouter models list
evalrouter run BENCHMARK_SLUG --model MODEL_ID --sample 3
evalrouter run BENCHMARK_SLUG --model MODEL_ID --task TASK_ID --task OTHER_TASK_ID

A spending cap is optional: with --max-cost USD a run starts only when its quote's maximum charge fits under the cap; without it the run is bounded by your available wallet credit. Without a terminal, a paid start needs --yes.

For scripts, approve paid work explicitly and retain the operation identity:

evalrouter run BENCHMARK_SLUG --model MODEL_ID --sample 3 --yes --json
evalrouter runs list
evalrouter runs wait RUN_ID
evalrouter runs results RUN_ID
evalrouter runs export RUN_ID --file-format json --output result.json

A sample is not a full-benchmark score. A local timeout or interrupted watcher does not stop server work. Reuse the saved request identity when recovering an uncertain submission; a new identity can start another evaluation.

Use repeatable --task ID flags to run exact task IDs. In an interactive run, choose Specific tasks to browse the selected source. Required dependencies are added automatically and included in the displayed task count. The service checks direct task IDs during preflight before starting work.

Use evalrouter --help and each command's --help for supported options. Account commands include login, whoami, workspaces and logout; credits reads balance and holds. Payments take place on the website.

Author your own benchmark

benchmarks init, benchmarks validate and benchmarks bundle run entirely on your machine: no account, no network request, and nothing from your package is imported or executed. The normative validator ships inside evalrouter-sdk, so the base installation includes it:

evalrouter benchmarks init ./my-benchmark --namespace example --name my-benchmark
evalrouter benchmarks validate ./my-benchmark
evalrouter benchmarks bundle ./my-benchmark   # evalrouter-bundle-example-my-benchmark-<version>.zip

init writes an explicitly synthetic draft and claims no license: replace the tasks, rights, references and limits before submitting. Local validation is not platform admission — EvalRouter revalidates every submission on its own servers.

Where private benchmarks are enabled, benchmarks uploads, benchmarks imports and benchmarks private use workspace-private data with the service's built-in graders. These are account-backed operations, separate from local package validation. See the private benchmark guide for manifests, supported formats, source management and permissions.

Python SDK

Applications and scripts use the SDK directly: python -m pip install evalrouter-sdk installs the typed client, evalrouter.agents, trace capture and the LangSmith connector without the terminal application. See evalrouter-sdk.