pytest-xharness-eval 🧪🤖

pytest plugin for cross AI agent harness evaluation.

Write the eval once. Run it against every harness and every model.

• pytest-xharness-eval 🧪🤖
  • What it does
  • Quickstart
  • Narrow a run
  • Configuration
  • How it works
  • What it does not do
  • Development
  • Read next

What it does

A pytest plugin that runs the claude and codex CLIs headlessly against a fixture workspace, captures each run's own session log, prices it, and grades what the agent left behind. A skill opts in by adding an evals/ directory; pytest does the rest.

Every eval cell is live and costs money. There is no replay mode. Preview the spend with --dry-run before a sweep. The design rationale lives in ARCHITECTURE.md and the decision log in docs/adrs/; agents start at AGENTS.md.

Quickstart

1. Install the plugin into the repository that holds your skills. The pytest11 entry point registers it; no conftest.py wiring is needed:

   uv add --dev pytest-xharness-eval
   

2. Pin pytest's rootdir to the repository root, so the plugin finds skills/ from any argument path. An empty [tool.pytest.ini_options] table is enough:

   [tool.pytest.ini_options]
   

3. Add an eval beside the skill. Files and functions both carry the eval_ prefix, as test_ does for pytest. Fixtures are seed workspaces copied fresh for every cell; captured/ is written by each run and git-ignored; its history.jsonl gains one metrics line per live cell:

   skills/<skill>/
     SKILL.md
     evals/
       eval_<suite>.py
       fixtures/<name>/
       captured/<case>/
       captured/history.jsonl
   

   from pytest_xharness_eval import evalcase
   
   @evalcase(prompt="...", skill="<skill>", fixture="<name>")
   def eval_<case>(run, workspace):
       assert run.exit_code == 0
       assert (workspace / "OUTPUT.md").exists()
   

4. Preview the matrix. Nothing is invoked:

   uv run pytest skills/<skill>/evals --dry-run
   

   xharness-eval: skills root = /repo/skills, workdir = /repo/tmp/evals
   xharness-eval: matrix = plugin default (2 entries); a case's models= overrides it
   collected 2 items
   skills/<skill>/evals/eval_<case>.py ss
   
   ============================ agent eval report ============================
     dry-run          -  skills/<skill>/evals/eval_<case>.py::eval_<case>[claude/claude-opus-5]
     dry-run          -  skills/<skill>/evals/eval_<case>.py::eval_<case>[codex/gpt-5.6-sol]
     total spend: $0.0000 across 2 cell(s)
     report: /repo/tmp/evals/report.json
   

5. Run it live, with -v so every cell reports its verdict, USD, context, wall clock, turns, and tool calls as it lands. Add -n 2 to run cells in parallel. This spends money:

   uv run pytest skills/<skill>/evals -v
   

   skills/<skill>/evals/eval_<case>.py::eval_<case>[claude/claude-opus-5] PASSED  est $0.5762 (harness $0.5773)  352,451 accumulative_billed_tokens  23,898 baseline_tokens  76.0s  9 turns  8 tools
   

   Read the status word as: this plugin's estimate from its price table (and the harness CLI's own figure, where it reports one), every billed token summed over all turns (the cached prefix is re-read each turn), the harness's own prompt on turn 1, wall clock, model calls, tool calls. Every estimate records the rates it used and where they came from (rates_applied).

   Each cell leaves its verbatim session log and a normalised .result.json with a per-turn ledger under captured/<case>/, appends one metrics line to captured/history.jsonl, and the sweep writes tmp/evals/report.json with USD per cell plus a browsable captured/report.html with its glossary (XHARNESS-REPORT-GLOSSARY.md) beside it. Serve it with python3 -m http.server --directory <captured dir>; it fetches the JSON beside it.

run is a RunResult: session id, log path, token usage by tier, tool calls, files written, and USD cost. The reference case, with its assertions written as a tutorial, is eval_palette_mandate.py, the reference case kept beside the skill it grades in the consuming repository.

Narrow a run

The matrix is the spend dial. These options are the plugin's own; everything else is stock pytest (-k, -x, -m eval, node ids).

Do not run pytest skills from the root: it walks into every skill's scripts/ directory and collects their unit tests too. skills/*/evals is the full matrix.

A case overrides the matrix with @evalcase(..., models=["codex/gpt-5.6-sol"]).

Configuration

The matrix has three scopes, highest precedence first: a case's models=, the project's xharness_matrix ini key, and the plugin's bundled default (claude/claude-opus-5, codex/gpt-5.6-sol). The report header names which one applied.

Four ini keys, paths relative to pytest's rootdir:

[tool.pytest.ini_options]
xharness_matrix = [
    "claude/claude-opus-5",
    "claude/claude-sonnet-5",
    "claude/claude-haiku-4-5-20251001",
    "codex/gpt-5.6-luna",
    "codex/gpt-5.6-terra",
    "codex/gpt-5.6-sol",
]

An unpriced model stops the sweep at collection, before any spend. Add a row to a prices.toml at the rootdir:

["gpt-5.6-luna"]
input       = 1.25e-6
output      = 1.0e-5
cache_read  = 1.25e-7
cache_write = 1.25e-6

How it works

flowchart LR
    CASE["eval_*.py case"]
    PLUG["plugin.py<br/>collect, expand matrix"]
    WS["workspace.py<br/>pristine copy"]
    RUN["runner.py<br/>claude | codex"]
    LOG["session log<br/>this run's own"]
    NORM["normalise.py<br/>RunResult"]
    PRICE["pricing.py<br/>USD"]
    GRADE["case assertions"]
    REP["report.json"]

    CASE --> PLUG --> WS --> RUN --> LOG --> NORM --> PRICE --> GRADE --> REP

    classDef new fill:#7c3aed,color:#fff
    classDef data fill:#0f766e,color:#fff
    classDef good fill:#047857,color:#fff
    class CASE,PLUG,WS,RUN new
    class LOG,NORM data
    class PRICE,GRADE,REP good

One cell flows left to right: a case is expanded into cells, each cell gets a fresh workspace, the CLI runs, its own log is located and normalised, priced, graded, and reported. The hard part is the middle: the two CLIs need different contracts to tie a verdict to the right log. ARCHITECTURE.md explains both.

What it does not do

• It does not replay recorded runs. Every cell invokes the real CLI (ADR 0002).
• It does not give the agent a git repository. The workspace is a plain copy of the fixture, so skills that read git history are out of scope (ADR 0004).
• It does not mock either CLI, in tests or in evals.
• It does not price an unknown model as zero. It refuses to run (ADR 0007).
• It does not throttle providers. -n N runs N cells at once; each cell is isolated (own workspace, own CODEX_HOME, own Claude session). If a provider rate-limits you, -n 2 --dist loadgroup keeps each harness's cells on one worker (parallel across harnesses, serial within one).

Development

make format   # ruff format + isort
make check    # ruff check + isort --check-only + mypy --strict
make test     # pytester-based suite, no mocks, coverage badge refresh
make build    # wheel into dist/

The functions that spawn a CLI (runner.run_claude, runner.run_codex and their helpers, plugin.EvalItem._run_live) are excluded from coverage with a stated reason rather than faked. They are exercised by the paid evals in a consuming repository.

Publishing happens from GitHub Releases via .github/workflows/publish.yml (PyPI trusted publishing).

The report page

captured/report.html is built from report-ui/, a bun workspace (Vite, React, TypeScript, Tailwind, shadcn/ui, Vitest) that emits one self-contained HTML file (ADR 0028). bun is needed only to change the page; the plugin ships the built file.

make ui-dev CAPTURED=path/to/<skill>/evals/captured    # hot reload against real captured data
make ui-check                                          # tsc + eslint + prettier
make ui-test                                           # vitest component tests
make ui-smoke CAPTURED=path/to/<skill>/evals/captured  # build, populate inline, render headlessly
make ui-promote                                        # ship the build as assets/report.html (CI checks it is current)

Read next

• docs/token-accounting.md: how accumulative_billed_tokens (billed across turns) and peak_context_tokens (the largest prompt) are derived from what each provider reports, with a worked session
• ARCHITECTURE.md: why the two CLIs need different capture contracts, how pricing works, and the vocabulary the code uses.
• AGENTS.md: operating instructions and hard boundaries for agents.
• docs/adrs/README.md: the decision index.