EvalRouter Python SDK

Evaluate a model or an agent on a benchmark in one call. Requires Python 3.12 or 3.13. Distributed under the proprietary Kimpton EvalRouter SDK License, included as LICENSE in the package.

python -m pip install evalrouter-sdk

The command line is a separate package: uv tool install evalrouter (it depends on this SDK at the same version).

Run a benchmark

from evalrouter import EvalRouter

client = EvalRouter()  # EVALROUTER_API_KEY and EVALROUTER_BASE_URL from the environment
run = client.runs.create(benchmark="gsm8k", model="gpt-5", max_cost=5, wait=True)
print(run.score)

The same run from a terminal:

evalrouter run gsm8k --model gpt-5 --max-cost 5
evalrouter run gsm8k --model gpt-5 --max-cost 5 --dry-run   # the quote; nothing is charged

• Names, not IDs. benchmark and model take a slug or an unambiguous name ("gsm8k", "gsm8k@2" to pin a version, "gpt-5"). Exact IDs work too.
• Money is USD. max_cost=5 or max_cost="0.25" is the most the run may charge; the run is refused before it starts if its quote is higher. max_cost is optional: without it the run is bounded by your wallet balance. Your API key's spend ceiling bounds everything.
• Waiting is one flag. wait=True returns the finished run; without it, run.wait() later. run.results() reads the results, run.score is the headline metric, run.cancel() stops it and run.export(format="csv") exports it.
• Look before you spend. dry_run=True returns the quote (maximum charge and plan) and starts nothing; client.runs.create(quote_id=quote.id) starts it. Repeating a create never starts a second run.

Advanced quote fields are optional keywords: coverage={"mode": "sample", "sample_count": 50, "seed": 42}, concurrency, name, metadata. AsyncEvalRouter has the same calls, awaited.

Evaluate an agent

The agent is one argument:

client.runs.create(benchmark="gsm8k", model="gpt-5", agent="opencode")  # ACP Registry agent
client.runs.create(benchmark="gsm8k", model="gpt-5", agent="local:my-agent --acp")
client.runs.create(benchmark="gsm8k", agent="remote:my-a2a-agent")  # a connection's name
client.runs.create(
    benchmark="gsm8k", model="gpt-5", agent="opencode", hosted=True
)  # run hosted by EvalRouter

Registry and local agents run on your machine: EvalRouter delivers each task, meters every model call through the run's gateway and grades the submission with the benchmark's own grader. evalrouter agents list shows the registry agents with their integrity: pinned (a binary with a published sha256), package-registry (an exact npm or PyPI package) or unverified-upstream, which runs only when you allow it (agents.registry_agent("cursor", allow_unverified=True), or --allow-unverified on the command line). registry_agent("opencode@1.18.34") pins a version; without one the pinned snapshot's version runs.

An agent written in Python is an agents.Agent:

from evalrouter import EvalRouter, agents


class Solver(agents.Agent):
    name = "my solver"

    async def run(self, context):
        task = await context.task.get()
        reply = await context.gateway.chat(messages=task["inputs"]["messages"])
        await context.task.submit(reply["choices"][0]["message"]["content"])
        await context.task.done()


run = EvalRouter().runs.create(
    benchmark="gsm8k", model="gpt-5", agent=Solver(), max_cost=2, wait=True
)

For more control, evalrouter.agents has the step-by-step form: agents.quote(...) returns an evaluation whose start() gives an AgentRun with events(), wait() and resume().

Import LangSmith traces

The optional langsmith extra provides a local connector for completed agent runs. Where your workspace has Traces, create an SDK destination connection in Traces → Connect agent, then supply LANGSMITH_API_KEY and the connection's EvalRouter environment variables through your secret manager. The source key is never sent to EvalRouter.

python -m pip install 'evalrouter-sdk[langsmith]'
python -m evalrouter.langsmith projects
python -m evalrouter.langsmith sync --project 'YOUR_PROJECT_ID' \
  --state './.private/evalrouter-langsmith-state.json' --metadata-only --watch

The process must stay running for automatic imports. Its private checkpoint resumes progress across restarts. Set --since to an ISO timestamp to choose the initial import window. This explicit metadata-only mode omits source names, inputs, outputs, error text, tags, events and arbitrary metadata. Hierarchy, timing, error status and reported numeric usage remain. The compatibility default captures bounded field-name-redacted content if --metadata-only is omitted.

Checkpoint version 2 binds the content policy and destination. Unknown acknowledgments replay a private adjacent .delivery.json payload unchanged; keep it with the checkpoint. A policy change on an existing checkpoint is refused. Version 1 upgrades only in its original content-capture mode and cannot restore payload bytes lost by the older connector. Finish unresolved deliveries before using a new checkpoint and connection for a different policy. Only activity recorded by LangSmith can be imported; two stable closed reads are observed coverage, not proof that every upstream child exists.

Live agent capture

Install with python -m pip install 'evalrouter-sdk[tracing]' in your agent's own environment. Configure EVALROUTER_BASE_URL, EVALROUTER_WORKSPACE_ID, EVALROUTER_TRACE_CONNECTION_ID and the restricted EVALROUTER_TRACE_KEY through an authorized secret manager. Package installation and connection authorization are separate.

import json
import os
from anthropic import Anthropic
from evalrouter.tracing import TraceCapture

# Also install anthropic, and supply ANTHROPIC_API_KEY privately.
model = Anthropic(max_retries=0)
capture = TraceCapture(capture_content=True)
with capture.agent("Support assistant") as root:
    root.input({"message": "Explain what an evaluation benchmark is."})
    with capture.model(os.environ["ANTHROPIC_MODEL"]) as span:
        answer = model.messages.create(
            model=os.environ["ANTHROPIC_MODEL"],
            max_tokens=256,
            messages=[{"role": "user", "content": "Explain what an evaluation benchmark is."}],
        )
        text = "".join(block.text for block in answer.content if block.type == "text")
        span.output({"answer": text})
    root.output({"answer": text})
receipt = capture.seal(root.trace_id)
print(json.dumps(receipt))
if receipt["state"] != "accepted" or not receipt["complete"] or not capture.close():
    raise SystemExit(1)

This example makes a real paid model request. Content capture is explicitly enabled; use capture_content=False to omit input, output and error payloads. Caller attributes still require your metadata policy. Common secret field names are redacted, but secrets in arbitrary text cannot be detected. Content capture defaults to true; select metadata-only explicitly.

Root exit and flush() do not seal. Join owned children, then seal(root.trace_id) to acknowledge closure. receipt(root.trace_id) reports local delivery state, counts, accepted server UUID and server completion, independently of evaluation completion. Unknown acknowledgments retain identical batches for later flush/seal/close on the same instance. Zero newly inserted spans on replay is success. close() returns false while active, failed or unknown, and true after acknowledged closure and shutdown. A context manager calls close, but does not raise on a false result; inspect receipts explicitly. There is no background HTTP export. Buffers are process-local; retry before exit. Each instance retains at most 64 local trace receipts, 512 active spans and 16 MiB of pending records. When another root needs room, the oldest successfully sealed idle receipt is evicted. Save the receipt/server UUID if you need it later; receipt(id) raises ValueError after eviction. Pending, unknown, active or dropped traces are never evicted. If all 64 are unsettled, a new root is not recorded (its handle has no trace ID and failed_exports increases); resolve delivery or use another instance. Late activity inherited from an evicted trace is rejected, never made a new root. Close attempts every retained trace within one aggregate deadline and leaves the provider alive if any closure fails. Capture loss prevents successful seal. The optional OpenTelemetry dependency stays isolated from the base HTTP client.

Create a connection under Traces → Connect agent where Traces is enabled for your workspace. Detailed delivery receipts are described in the trace connection setup in your workspace.

Attach existing OpenTelemetry instrumentation

Install the evalrouter-sdk[tracing] extra into your existing Python 3.12/3.13 agent environment. Pass the OTel SDK provider the framework actually uses; do not replace its global provider. No agent inheritance or per-call EvalRouter wrappers are required for supported spans:

from evalrouter.tracing import TraceCapture

capture = TraceCapture(capture_content=False)
attachment = capture.attach(existing_provider, scope_names=("your.instrumentation",))
# Run the already-instrumented agent, then join all owned child work.
for trace_id in attachment.trace_ids:
    receipt = capture.seal(trace_id)
    print(receipt)  # IDs/counts/delivery state only; inspect accepted + complete.
# At application shutdown, check capture.close(); it never stops existing_provider.

Attachment exports metadata only, even when wrapper content capture is enabled. It removes source names, resource attributes, events, prompts, tool arguments, results and exception text. It retains original trace/span/parent IDs, closed operation/kind/status enums and bounded numeric token counters. Source counters are observations, not native usage charges. evalrouter.otel.status=unset means unknown source success: wire ok only means no explicit source ERROR, never a native grade or proof that a tool succeeded. Unknown/conflicting kinds are counted in attachment.unsupported_spans and prevent successful sealing. Provider flush queues records; it does not prove accepted ingestion or closure.

The optional [strands] extra pins Strands for applications that need it; the base client has no Strands dependency. Attachment only sees selected spans that the supplied provider actually records. It cannot recover unsampled or missing work, or capture uninstrumented frameworks automatically. LangSmith project imports and ACP evaluation execution are separate integrations.