EvalRouter Python SDK

The programmatic interface to EvalRouter: a typed client for discovery, quotes, evaluations and results, local waiting and exports, evalrouter.agents for evaluating an agent you run, trace capture and the LangSmith connector.

Requires Python 3.12 or 3.13. Distributed under the proprietary Kimpton EvalRouter SDK License, included as LICENSE in the package.

The SDK has no terminal interface. For the evalrouter command and its interactive workspace, install evalrouter (uv tool install --prerelease allow evalrouter), which depends on this package at the same version.

Python client

Install this alpha with python -m pip install --pre evalrouter-sdk in your application's environment. The import name is evalrouter; the client supports context-manager cleanup:

Canonical imports require a release containing this namespace. Older registry releases through 0.3.24 use kimpton_evalrouter. Select alpha releases explicitly to use the canonical evalrouter import and current tracing APIs.

import os
from evalrouter import Client

with Client(
    base_url="https://api.evalrouter.ai",
    api_key=os.environ["EVALROUTER_API_KEY"],
    workspace_id=os.environ["EVALROUTER_WORKSPACE_ID"],
) as client:
    print(client.catalog.models())

Use client.catalog, client.evals, client.quotes and client.runs for typed discovery, submission and result resources. Creating a quote starts no evaluation; submitting its accepted ID can start paid work. Keep the quote ID and idempotency key before submission. The model API guide describes request fields, limits and result contracts.

client.catalog.resolve_benchmark(name) and resolve_model(name, benchmark=...) resolve the names the command line accepts, without prompting. client.runs.follow(run_id) yields a run's events until it ends; client.evaluation_jobs.wait(job_id) waits for a multi-model job; report_url on runs and evaluation jobs returns the web report, where the charts are drawn.

Resources and imports are stable or preview. Preview ones (including evalrouter.agents and evalrouter.tracing) are supported but may change in a minor release. The Python SDK guide lists each tier and the deprecation notes for entries that left stable.

Evaluate an agent you run

evalrouter.agents evaluates an agent on EvalRouter tasks while the agent runs on your machine. EvalRouter delivers each task, meters every model call through the run's gateway (managed models only), grades the submission with the benchmark's own grader and records the trajectory. It is available where your workspace has agent evaluation enabled; otherwise quoting refuses with client_run_unavailable.

The agent can be an ACP Registry agent (registry_agent("opencode")), any local ACP command (local_agent("my-agent --acp"), which needs evalrouter-sdk[agents]), or an agent written in Python:

from evalrouter import Client, agents


class Solver(agents.Agent):
    name = "my solver"

    async def run(self, context):
        task = await context.task.task()
        reply = await context.gateway.chat(messages=task["messages"])
        await context.task.submit(reply["choices"][0]["message"]["content"])
        await context.task.done()


client = Client()
evaluation = agents.quote(
    client,
    agent=Solver(),
    model="ADMITTED_ROUTE_ID",
    tasks=agents.benchmark("eval://PROVIDER/NAME/VERSION"),
    coverage={"mode": "sample", "sample_count": 3, "seed": 42},
    budget_usd="2",
)
# Review evaluation.quote before starting; nothing is charged until start().
run = evaluation.start()
for event in run.events():  # or: async for event in run.events_async()
    print(event.type, event.task_ref, event.failure_code or "")
print(run.wait()["status"], run.results())

Tasks can also come from an environment version (agents.environment(REF, split="test")), your private benchmarks (agents.private_benchmark(ID)) or a suite. context.task offers the attempt's task tools (call, list, instructions, task, submit, done); file tasks are written in context.workspace. context.gateway is OpenAI chat-compatible (gateway.openai() returns an AsyncOpenAI client when openai is installed). Leaving the event loop early finishes the attempt in flight as an interrupted controller; agents.AgentRun.resume(client, RUN_ID, agent) continues the run with the same agent, and run.cancel() stops it. run.costs(), run.export("csv"), run.trajectories() and agents.compare(client, [run_a, run_b]) read the finalized results. agents.evaluate(...) does all of this in one call. Its result's status and succeeded report the run's outcome: a failed, partial or abandoned run is returned with succeeded false, as runs.wait returns it, not raised. The CLI's evalrouter run --agent uses the same functions and exits 0 only for a completed run (1 when it failed, 4 otherwise).

Import LangSmith traces

The optional langsmith extra provides a local connector for completed agent runs. Where your workspace has Traces, create an SDK destination connection in Traces → Connect agent, then supply LANGSMITH_API_KEY and the connection's EvalRouter environment variables through your secret manager. The source key is never sent to EvalRouter.

python -m pip install --pre 'evalrouter-sdk[langsmith]'
python -m evalrouter.langsmith projects
python -m evalrouter.langsmith sync --project 'YOUR_PROJECT_ID' \
  --state './.private/evalrouter-langsmith-state.json' --metadata-only --watch

The process must stay running for automatic imports. Its private checkpoint resumes progress across restarts. Set --since to an ISO timestamp to choose the initial import window. This explicit metadata-only mode omits source names, inputs, outputs, error text, tags, events and arbitrary metadata. Hierarchy, timing, error status and reported numeric usage remain. The compatibility default captures bounded field-name-redacted content if --metadata-only is omitted.

Checkpoint version 2 binds the content policy and destination. Unknown acknowledgments replay a private adjacent .delivery.json payload unchanged; keep it with the checkpoint. A policy change on an existing checkpoint is refused. Version 1 upgrades only in its original content-capture mode and cannot restore payload bytes lost by the older connector. Finish unresolved deliveries before using a new checkpoint and connection for a different policy. Only activity recorded by LangSmith can be imported; two stable closed reads are observed coverage, not proof that every upstream child exists.

Live agent capture

Install the alpha with python -m pip install --pre 'evalrouter-sdk[tracing]' in your agent's own environment. Configure EVALROUTER_BASE_URL, EVALROUTER_WORKSPACE_ID, EVALROUTER_TRACE_CONNECTION_ID and the restricted EVALROUTER_TRACE_KEY through an authorized secret manager. Install alpha releases explicitly; an ordinary install selects the stable release.

import json
import os
from anthropic import Anthropic
from evalrouter.tracing import TraceCapture

# Also install anthropic, and supply ANTHROPIC_API_KEY privately.
model = Anthropic(max_retries=0)
capture = TraceCapture(capture_content=True)
with capture.agent("Support assistant") as root:
    root.input({"message": "Explain what an evaluation benchmark is."})
    with capture.model(os.environ["ANTHROPIC_MODEL"]) as span:
        answer = model.messages.create(
            model=os.environ["ANTHROPIC_MODEL"], max_tokens=256,
            messages=[{"role": "user", "content": "Explain what an evaluation benchmark is."}],
        )
        text = "".join(block.text for block in answer.content if block.type == "text")
        span.output({"answer": text})
    root.output({"answer": text})
receipt = capture.seal(root.trace_id)
print(json.dumps(receipt))
if receipt["state"] != "accepted" or not receipt["complete"] or not capture.close():
    raise SystemExit(1)

This example makes a real paid model request. Content capture is explicitly enabled; use capture_content=False to omit input, output and error payloads. Caller attributes still require your metadata policy. Common secret field names are redacted, but secrets in arbitrary text cannot be detected. Content capture defaults to true; select metadata-only explicitly.

Root exit and flush() do not seal. Join owned children, then seal(root.trace_id) to acknowledge closure. receipt(root.trace_id) reports local delivery state, counts, accepted server UUID and server completion, independently of evaluation completion. Unknown acknowledgments retain identical batches for later flush/seal/close on the same instance. Zero newly inserted spans on replay is success. close() returns false while active, failed or unknown, and true after acknowledged closure and shutdown. A context manager calls close, but does not raise on a false result; inspect receipts explicitly. There is no background HTTP export. Buffers are process-local; retry before exit. Each instance retains at most 64 local trace receipts, 512 active spans and 16 MiB of pending records. When another root needs room, the oldest successfully sealed idle receipt is evicted. Save the receipt/server UUID if you need it later; receipt(id) raises ValueError after eviction. Pending, unknown, active or dropped traces are never evicted. If all 64 are unsettled, a new root is not recorded (its handle has no trace ID and failed_exports increases); resolve delivery or use another instance. Late activity inherited from an evicted trace is rejected, never made a new root. Close attempts every retained trace within one aggregate deadline and leaves the provider alive if any closure fails. Capture loss prevents successful seal. The optional OpenTelemetry dependency stays isolated from the base HTTP client.

Create a connection under Traces → Connect agent where Traces is enabled for your workspace. Detailed delivery receipts are described in the trace connection setup in your workspace.

Attach existing OpenTelemetry instrumentation

Install the alpha evalrouter-sdk[tracing] extra into your existing Python 3.12/3.13 agent environment. Pass the OTel SDK provider the framework actually uses; do not replace its global provider. No agent inheritance or per-call EvalRouter wrappers are required for supported spans:

from evalrouter.tracing import TraceCapture

capture = TraceCapture(capture_content=False)
attachment = capture.attach(existing_provider, scope_names=("your.instrumentation",))
# Run the already-instrumented agent, then join all owned child work.
for trace_id in attachment.trace_ids:
    receipt = capture.seal(trace_id)
    print(receipt)  # IDs/counts/delivery state only; inspect accepted + complete.
# At application shutdown, check capture.close(); it never stops existing_provider.

Attachment exports metadata only, even when wrapper content capture is enabled. It removes source names, resource attributes, events, prompts, tool arguments, results and exception text. It retains original trace/span/parent IDs, closed operation/kind/status enums and bounded numeric token counters. Source counters are observations, not native usage charges. evalrouter.otel.status=unset means unknown source success: wire ok only means no explicit source ERROR, never a native grade or proof that a tool succeeded. Unknown/conflicting kinds are counted in attachment.unsupported_spans and prevent successful sealing. Provider flush queues records; it does not prove accepted ingestion or closure.

The optional [strands] extra pins Strands for applications that need it; the base client has no Strands dependency. Attachment only sees selected spans that the supplied provider actually records. It cannot recover unsampled or missing work, or capture uninstrumented frameworks automatically. LangSmith project imports and ACP evaluation execution are separate integrations.

Native multi-model client requests

Preview the matrix before starting paid work. Save the request and idempotency key so an uncertain reply can be recovered without starting another job.

from evalrouter import Client

with Client() as client:
    request = {
        "name": "Model comparison",
        "candidates": [
            {"model": {"kind": "managed", "route_id": "YOUR_FIRST_ROUTE"}},
            {"model": {"kind": "managed", "route_id": "YOUR_SECOND_ROUTE"}},
        ],
        "evaluations": [{
            "selection": {"profile_ids": ["YOUR_EXACT_NATIVE_PROFILE"]},
            "coverage": {"mode": "sample", "sample_count": 1, "seed": 42},
        }],
    }
    preview = client.evaluation_jobs.preflight(request)
    if preview["supported"]:
        job = client.evaluation_jobs.start(
            {**request, "preflight_digest": preview["digest"]},
            idempotency_key="persist-this-job-request-key",
        )
        print(job["id"])

The job's selected models or agents and benchmarks stay together in the results board. Each runnable pair has its own run and receipt. Charges draw from the workspace wallet as work proceeds. Existing capped quotes keep their terms.