EvalRouter CLI

Evaluate models or qualified repository agents from your terminal: discover supported targets, review a spending cap, run an evaluation, and export results. Requires Python 3.12 or 3.13. Distributed under the proprietary Kimpton EvalRouter SDK License.

Install and create your account

We recommend uv to install the CLI in its own Python environment:

uv tool install --python 3.12 evalrouter
evalrouter --help

With pip, use pip install evalrouter. uv supplies Python 3.12 if needed. If your shell cannot find evalrouter, run uv tool update-shell and restart your terminal. Upgrade an existing installation with uv tool upgrade evalrouter.

This package was previously published as kimpton-evalrouter-sdk. The command, the kimpton_evalrouter import and every API are unchanged. To switch, run uv tool uninstall kimpton-evalrouter-sdk (or pip uninstall -y kimpton-evalrouter-sdk evalrouter) and install evalrouter.

The package supplies the evalrouter command; no repository checkout is needed. Create an EvalRouter account, verify your email, and follow the browser sign-in below. Package installation does not grant account access or evaluation credit.

Sign in with your account:

evalrouter login
evalrouter whoami

Approve the browser code. New accounts finish setup on the website; the CLI continues when that is complete. It selects your only workspace automatically. Production is the default API; evalrouter options lists --base-url and the other settings that apply to every command. No API key or credential-storage setup is needed for a normal sign-in.

CI and services can use EVALROUTER_API_KEY and EVALROUTER_WORKSPACE_ID supplied privately by a secret manager. Never put credentials in arguments, URLs, request files or logs.

Import LangSmith traces

The optional langsmith extra provides a local connector for completed agent runs. Create a LangSmith connection in Traces → Connect agent, then supply LANGSMITH_API_KEY and the connection's EvalRouter environment variables through your secret manager. The source key is never sent to EvalRouter.

pip install 'evalrouter[langsmith]'
python -m kimpton_evalrouter.langsmith projects
python -m kimpton_evalrouter.langsmith sync --project 'YOUR_PROJECT_ID' \
  --state './evalrouter-langsmith-state.json' --watch

The process must stay running for automatic imports. Its private checkpoint resumes progress across restarts. Set --since to an ISO timestamp to choose the initial import window. Inputs and outputs are bounded and field-name redacted; only activity recorded by LangSmith can be imported. This connector requires a package build that includes the langsmith extra.

Run an evaluation in one command

evalrouter run gpqa-diamond --model gpt-4o-mini

run matches the benchmark and model by slug, name or ID, suggesting the closest names after a typo. It creates a free quote, shows the coverage, conservative estimate, spending cap and concurrency, and asks Run it? [Y/n]. Only a yes starts paid work. It then follows progress and prints the scores, the change from your previous completed run of the same benchmark and model, and the report link. In a terminal, missing choices are asked for; run evalrouter run alone to pick everything. --sample N or --full sets coverage, --max-cost USD the cap and --dry-run stops after the quote.

Without a terminal (CI, agents, --json or --no-input) run never prompts and refuses unless both --yes and --max-cost are given:

evalrouter run gpqa-diamond --model gpt-4o-mini --max-cost 5 --yes --json

The idempotency key of each accepted quote is saved locally before the run is submitted (~/.local/state/evalrouter, or EVALROUTER_STATE_DIR; identifiers only, never credentials). Repeating the same command after a dropped connection resumes that run instead of starting another; --new starts a separate one. Afterwards, results, export --format html,json, wait and open default to your last run and accept a unique run ID prefix. Watching a run survives brief API outages such as a 503 during a deployment.

Lists

catalog, catalog --models, status, connections list and workspace list print a compact table sized to your terminal with one suggested next command; catalog hides benchmarks in review or unavailable unless you pass --all, and catalog -i browses them interactively. Every list accepts --json, --jq '.data[].id' (a small jq subset), -w/--wide, --columns a,b, --sort COL (--sort=-COL descending), --filter KEY=VALUE and --limit N. Piped output stays JSON. Colour follows NO_COLOR, CI and TERM=dumb.

run shows the quote (expected cost from your previous run when known, worst case, and the smallest workable cap) before asking for the cap, refuses a cap that cannot fit one task, and asks before repeating a run that just finished (scripts pass --new). --all-metrics lists every metric and comparison.

Discover, quote, then run

evalrouter catalog --status active --json
evalrouter catalog --models --json
evalrouter catalog --slug BENCHMARK_SLUG --json

Choose a profile whose quote_availability.status is ready_for_quote and a compatible managed model route from those responses. An active catalog entry alone does not mean it is ready to run. Save this as quote.json, replacing both uppercase identifiers:

{
  "model": {"kind": "managed", "route_id": "MODEL_ROUTE_ID"},
  "selection": {"profile_ids": ["BENCHMARK_PROFILE_ID"]},
  "coverage": {"mode": "sample", "sample_count": 3, "seed": 42},
  "max_charge_microusd": "1000000"
}

evalrouter quote --config quote.json --json

Review compatibility, coverage, cost components, warnings and expiry. A quote starts no paid work. The example's $1 platform cap is not a price guarantee or promise that a particular evaluation fits. Paid runs require available credit.

Persist the returned quote ID and an operation key before submitting:

evalrouter run --quote REVIEWED_QUOTE_ID --idempotency-key SAVED_OPERATION_KEY --json
evalrouter wait RUN_ID --wait-timeout 3600 --json
evalrouter results RUN_ID --json
evalrouter export RUN_ID --format json --output result.json
evalrouter export RUN_ID --format csv --output result.csv
evalrouter export RUN_ID --format html --output result.html

Replace RUN_ID with the returned ID. A lost submission response is recovered with the same quote and operation key; a new key may start separate work. Inspect terminal status, coverage, errors and billing alongside scores. A sample is not a full-benchmark score. Exports refuse overwrite by default; use the result's integer --version for repeatable reports.

Supported commands

catalog (--models, --slug or --environment), connections list/create/check/update/disable, quote, run, status, wait, cancel, results, export and open. Repository qualification uses agent-builds options/preview/create/list/status/cancel. Authoring your own benchmark package uses benchmark init/validate/bundle. Run each command with --help for exact flags. A connected model's credential is never a command argument: connections create asks with hidden input in a terminal, and scripts use --key-stdin or --key-env NAME (connections update --rotate-key replaces it). Connection checks can send a small model request, and your model provider may bill separately.

For wait and run --wait, --progress auto shows elapsed time and status in an interactive terminal, with a percentage when validated processed and planned counts are available. Use --progress plain for structured progress or --progress off to suppress it. Redirected stderr and --json retain structured progress. A progress percentage counts processed samples; it is not a score.

Use --config - for JSON on stdin. --json writes one result on stdout and progress on stderr. Exit codes: 0 success; 1 API/transport or failed run; 2 invalid input or rejected request (including a refused unattended run); 4 partial/cancelled waited run; 5 confirmation declined, nothing started. Interrupting a local wait does not cancel server work. Use evalrouter cancel RUN_ID explicitly when cancellation is intended.

See the CLI documentation for account setup, model selection, spending semantics, recovery and versioned exports. The CLI uses the model API, which is also documented for direct HTTP integrations.

Author your own benchmark

benchmark init, benchmark validate and benchmark bundle run entirely on your machine: no account, no network request, and nothing from your package is imported or executed. The normative validator ships inside evalrouter, and its one dependency (pydantic) is an optional extra so the base installation stays thin (httpx and rich only). With pip, use pip install 'evalrouter[benchmark]':

uv tool install --python 3.12 "evalrouter[benchmark]"
evalrouter benchmark init ./my-benchmark --namespace example --name my-benchmark
evalrouter benchmark validate ./my-benchmark
evalrouter benchmark bundle ./my-benchmark --output my-benchmark.zip

Without the extra these three commands stop before reading your package and report benchmark_extra_required with the exact install string; --help and every network command keep working. init writes an explicitly synthetic draft and claims no license: replace the tasks, rights, references and limits before submitting. Local validation is not platform admission — EvalRouter revalidates every submission on its own servers.

Repository agents

Repository commands require version 0.2.0 or later and a service that has enabled repository qualification for the workspace. Installing the package does not enable that service. Check its current availability and supported targets first:

evalrouter agent-builds options --json
evalrouter catalog --environment ENVIRONMENT_FAMILY_PATH --json

The catalog lookup takes the family path before @; qualification inputs still require the exact versioned reference returned by options. Stop if qualification is disabled or the desired exact target/model is absent. The supported runtime is locked Node/npm with an evalrouter-agent.json manifest; custom images and Python agent runtimes are not supported. Save build.json:

{
  "repository_url": "YOUR_PUBLIC_GITHUB_REPOSITORY_URL",
  "ref": "main",
  "manifest_path": "evalrouter-agent.json",
  "qualification": {
    "environment_ref": "EXACT_ENVIRONMENT_FROM_OPTIONS",
    "model": {"kind": "managed", "route_id": "MODEL_FROM_OPTIONS"}
  },
  "max_cost_microusd": "1000000"
}

For a private repository, connect GitHub in the web application and explicitly select that repository for this workspace. Add source: {"github_connection_id": "CONNECTION_ID", "github_repository_id": 123} using the two actual IDs from that connection. These IDs select a grant; they are not credentials. The CLI does not accept GitHub tokens or authorize access.

evalrouter agent-builds preview --config build.json --json

Preview resolves an immutable commit and returns the qualification scope, costs, cap and expiry. It creates no job or credit hold and starts no sandbox/model work. Review it, then save the quote ID and a durable operation key before:

evalrouter agent-builds create --quote REVIEWED_BUILD_QUOTE --idempotency-key SAVED_BUILD_KEY --json
evalrouter agent-builds status BUILD_JOB_ID --json
evalrouter agent-builds list --quote REVIEWED_BUILD_QUOTE --json

Creation reserves the quoted cap and may incur its stated costs. Recover an unknown submission with the same quote and key or the quote-filtered list; a new key may create new work. Cancel explicitly with evalrouter agent-builds cancel BUILD_JOB_ID. A cancelling or reconciling status remains unresolved. Wait for terminal cleanup and zero held funds.

Ready requires status: ready, cleanup_confirmed: true, zero held funds and an agent.ref. It establishes only the returned scope, not benchmark quality. Use that record's exact qualification.environment_ref, qualification.split and qualification.evaluation_coverage in a separate evaluation quote:

{
  "agent": "READY_AGENT_REF",
  "selection": {"environment": "RETURNED_ENVIRONMENT_REF", "split": "RETURNED_SPLIT"},
  "coverage": "REPLACE_WITH_RETURNED_EVALUATION_COVERAGE_OBJECT",
  "max_charge_microusd": "1000000"
}

Replace the coverage placeholder with the entire returned object; do not guess tasks, counts or seeds. Use the existing quote, run, wait, results and export commands above. Qualification and evaluation have separate caps and charges. A qualification pass does not submit an evaluation automatically.

Live agent capture

Install evalrouter[tracing] and use from kimpton_evalrouter.tracing import TraceCapture. Create a connection in Traces and configure its restricted ingest key with EVALROUTER_TRACE_KEY and EVALROUTER_TRACE_CONNECTION_ID, alongside the API origin and workspace ID. Wrap each run with capture.agent(...), and calls with capture.model(...) or capture.tool(...). Use span.input(value) and span.output(value) to capture content. Call capture.flush() before a serverless invocation exits and capture.close() on shutdown. The optional OpenTelemetry dependency is isolated from the base HTTP client.

See the capture guide for complete examples, redaction settings, coverage, retention, and delivery limits.

Use client.traces for connection, trace, evaluator and finding resources, and client.datasets for curation, review, immutable versions and exports. These operations require the corresponding service release and workspace permissions.