evalpilot 0.1.1

A general framework for online-driving, recording, and scoring AI agents against task suites.

evalpilot

A general framework for evaluating agents. It online-drives any agent through a task suite, records its trajectory, scores it with four pluggable scorers (objective checkpoints, LLM-as-judge, trajectory/process, efficiency), and emits a JSON + static HTML report.

Install

Published on PyPI as evalpilot; the CLI command is evalpilot.

Run with no install (recommended):

uvx evalpilot run

Install as a tool:

uv tool install evalpilot      # or: pipx install evalpilot

For browser-UI (webui) agents, install the extra and the browser once:

uv tool install "evalpilot[webui]"
playwright install chromium

From source (development):

python -m venv .venv
.venv/bin/pip install -e ".[dev]"

Quick start

.venv/bin/evalpilot run --suite suites/example.yaml --agent configs/function-echo.yaml --run-id demo --no-llm-judge
open runs/demo/report.html

Or run it interactively — omit --suite/--agent and a wizard walks you through defining the agent and tasks (no YAML needed):

evalpilot run

Browse all runs at once:

open runs/index.html

Concepts

• Task / TaskSuite — declarative units of evaluation (prompt, setup, checkpoints, rubric, budget).
• AgentAdapter — narrow interface that drives the agent under test. Built-in: function, cli, http, webui. Add a new agent by writing one subclass.
• Trajectory — the recorded run (steps, output, usage, final state). Sole data source for all scorers.
• Scorers — checkpoint, llm_judge, trajectory, efficiency. Read-only, pluggable, weighted. Checkpoints include file/command checks plus output_contains / output_matches for grading text answers objectively.
• Runner / Reporter — orchestrate and report.

Adding an agent

Write a YAML config selecting an adapter type. For a CLI agent:

type: cli
name: my-agent
command: my-agent --prompt {prompt} --json-stream

For an in-process Python callable:

type: function
name: my-agent
entrypoint: my_module:run

For an OpenAI-style HTTP endpoint:

type: http
name: my-agent
url: http://localhost:8000/v1/chat/completions

For a browser-based (web UI) agent:

type: webui
name: my-chat-agent
url: https://my-agent.example.com/chat
input_selector: "textarea"
output_selector: ".message.assistant:last-child"

The webui adapter needs Playwright (an optional extra):

pip install -e ".[webui]"
playwright install chromium

It types the prompt into input_selector, waits until the reply text in output_selector stops changing (default 1.5s), then scrapes it. Reuse a logged-in session with storage_state: ./auth.json.

Scoring weights

Defaults: checkpoint=3, llm_judge=2, trajectory=1, efficiency=0. Override per suite (default_weights) or per task (weights). Each scorer's sub-score and its reason are always preserved in the report, not just the aggregate.

Commands

evalpilot run --suite SUITE.yaml --agent AGENT.yaml [--run-id ID] [--out DIR] [--no-llm-judge]
evalpilot run                            # interactive wizard (builds agent + tasks)
evalpilot report --run runs/ID           # re-render report.html from result.json
evalpilot compare runs/a runs/b          # combined leaderboard across runs
evalpilot index [--out runs]             # overview page listing all runs -> runs/index.html

Testing

.venv/bin/python -m pytest

All scorer/adapter tests inject fakes (fake CLI scripts, fake HTTP post, fake LLM completion), so the suite runs fully offline with no network or API key.