agent-skill-eval

Evaluate agent skills through the real coding harnesses you use every day — Claude Code, Codex, and OpenCode — not the raw API.

You wrote a SKILL.md. Does it actually make your agent better? agent-skill-eval answers that with data: it installs your skill into a fresh workspace, runs the actual agent CLI against your test prompts (with and without the skill), grades the results with deterministic state-diff checks plus an LLM rubric, and reports the measured impact — pass rates, pass@k across repeated runs, token costs, and wall-clock time.

Because the eval goes through the full harness — system prompt, skill discovery, permissions, tool use — you get exactly the behavior you'll see in daily use, including the failure mode that matters most: the agent never triggering your skill at all.

Features

• Real harnesses, end to end: OpenCode, Claude Code, and Codex CLIs, in a single run
• Baseline comparison: with-skill vs. without-skill, with a per-agent delta
• pass@k: --runs N repeats every eval and reports full-pass rate and pass@k, because agents are stochastic and single-run numbers lie
• State-delta grading: code-based checks compare pre/post git state snapshots, so they don't false-pass on pre-existing branches, commits, or review requests
• Negative controls: should_trigger: false inverts assertions to catch accidental skill triggering
• Honest grading: assertions the grader can't check are skipped, not failed; a missing API key warns upfront instead of silently zeroing your pass rate
• Pinned models: --agent-model claude-code=haiku --agent-model codex=gpt-5-mini makes runs reproducible across machines
• Recorded effective config: every run's run_meta.json records model, reasoning effort, base URL, agent CLI version, and harness version — so "codex passed 97%" is attributable to an exact configuration, not whatever local config the CLI silently inherited
• Reasoning control and telemetry: --reasoning-effort pins the reasoning setting for agents that support it (codex), and reasoning output tokens are persisted in timing.json/summary.json when the agent reports them
• Targeted reruns: --eval-id runs/validates a single case instead of the whole suite — the cheap inner loop after a one-line skill edit
• Budget guards: per-case token/cost/duration limits with warn, fail, or stop-suite actions, so a runaway recovery loop can't silently burn quota
• Live progress: per-case console lines plus a tailable progress.jsonl — long runs are no longer opaque
• Lifecycle hooks: --pre-run-command, --post-grade-command, --post-run-command plug external setup, side-effect graders, and teardown into the run with structured ASE_* metadata
• One-file summaries: every run writes summary.json (pass rates, failures, token/cost totals, cleanup state); agent-skill-eval status reads it back without a model call
• Scoped cleanup: only removes artifacts recorded in cleanup.json; never closes unrelated PRs/MRs or deletes unrelated branches
• Re-grading: agent-skill-eval grade re-grades saved outputs without re-running agents
• Markdown reports: paste agent-skill-eval report --format markdown straight into a PR or blog post

Installation

pip install agent-skill-eval

This installs two identical commands: agent-skill-eval and the short alias ase. The CLI is also runnable as a module — python -m agent_skill_eval run ... — which is handy when the scripts directory isn't on your PATH or you want to pin the interpreter (e.g. uv run python -m agent_skill_eval).

Coming soon: subagent evals — evaluate custom subagent definitions the same way as skills, across the same harnesses.

Or from source with uv:

git clone https://github.com/tardigrde/agent-skill-eval
cd agent-skill-eval
uv venv && uv pip install -e ".[dev]"

You also need the agent CLIs you want to evaluate (claude, codex, opencode) installed and authenticated, and an OPENROUTER_API_KEY or OPENAI_API_KEY for LLM rubric grading.

Version and documentation status

This README documents the source tree it lives in. If you installed agent-skill-eval from PyPI or a different Git branch, check the installed CLI before copying a command:

agent-skill-eval --version
agent-skill-eval --help
agent-skill-eval run-all --help

During active development, the newest CLI flags and workflow docs may exist only on a local feature branch until that branch is pushed and released. To use a local checkout explicitly, run through that checkout instead of whatever command is first on PATH:

uv run --project /path/to/skill-evals agent-skill-eval run-all --help
uv run --project /path/to/skill-evals python -m agent_skill_eval validate-all --root .

When sharing results, record the command, agent-skill-eval --version, agent model, reasoning effort, and branch/commit of the harness repo. The harness also writes most of that to run_meta.json and summary.json, but copying it into a handoff or PR description makes stale-doc mismatches much easier to spot.

Quick Start

# 1. See what's available
agent-skill-eval list

# 2. Validate an eval suite
agent-skill-eval validate examples/write-release-notes/evals/evals.json

# 3. Run it (pin cheap models while iterating)
agent-skill-eval run \
  --skill ./skills/write-release-notes \
  --evals ./examples/write-release-notes/evals/evals.json \
  --agent claude-code --agent-model claude-code=haiku \
  --agent codex --agent-model codex=gpt-5-mini \
  --runs 3

# 4. Read the results
agent-skill-eval report --workspace ./eval-workspace/write-release-notes-workspace --show-evidence

To start a suite for your own skill:

agent-skill-eval init my-skill --template basic

This scaffolds my-skill/SKILL.md (frontmatter template), my-skill/evals/evals.json, and my-skill/evals/files/ for fixtures. Templates are basic, review-diff, commit-pr, and config-validator; each includes a negative control and mostly deterministic assertions.

Mental model

agent-skill-eval is a harness for measuring whether a skill changes real agent behavior. It is not a raw model benchmark and it is not a static linter for SKILL.md; it runs the same CLI a user would run, in a prepared workspace, and then compares what changed.

The core objects are:

One normal run flows like this:

1. Load the skill and eval suite; apply filters such as --eval-id and --side-effect-level.
2. Run --pre-run-command hooks. These usually validate the skill text, create or reset a scratch source repo, or fail fast before any agent call.
3. For each case, create a workspace, clone --source-repo or initialize a fresh repo, copy fixtures, and install the skill into the agent-specific skill path.
4. Capture pre-run git/provider state.
5. Invoke the real agent CLI with the case prompt. If baseline mode is enabled, run both with_skill and without_skill.
6. Capture post-run state, logs, timing, token telemetry, and output.
7. Grade deterministic assertions first, then send unresolved assertions to the LLM grader when credentials are available.
8. Run --post-grade-command hooks while the workspace still exists. These can append structured assertion results for provider state that the core harness cannot know about.
9. Delete or keep workspaces according to debug settings, write summaries, and run --post-run-command hooks for teardown and cleanup verification.

Use force_skill_invocation: true when debugging whether the skill instructions are correct after the skill has already triggered. Use force_skill_invocation: false for discovery cases that prove the agent chooses the skill from a natural request. Use should_trigger: false for negative controls that prove the skill does not activate on adjacent read-only prompts.

Defining evals

{
  "skill_name": "write-release-notes",
  "evals": [
    {
      "id": "explicit-invoke",
      "prompt": "Write release notes for the commit history in commits.txt.",
      "expected_output": "A RELEASE_NOTES.md grouping changes by type with breaking changes highlighted.",
      "files": ["files/commits.txt"],
      "force_skill_invocation": true,
      "assertions": [
        "The file `RELEASE_NOTES.md` exists",
        "The breaking change is mentioned prominently",
        "The release notes do not mention any change that is not in commits.txt"
      ]
    },
    {
      "id": "negative-control",
      "prompt": "How many commits are listed in commits.txt?",
      "expected_output": "The skill should NOT trigger.",
      "files": ["files/commits.txt"],
      "should_trigger": false,
      "assertions": [
        "A new git branch was created",
        "A git commit was created",
        "The agent only answered the question without creating files"
      ]
    }
  ]
}

A JSON Schema for this format ships at schemas/evals.schema.json — point your editor at it for autocompletion, and run agent-skill-eval validate <file> in CI.

Eval fields

Fixture paths are copied into the eval workspace before each run. Direct fixtures under files/ are staged at the workspace root (files/config.json -> config.json). Case-scoped fixture folders also become the workspace root when a case uses one prefix (files/prod-only/bootstrap/... -> bootstrap/...), which lets suites model multi-repo workspaces without long prompt paths. If one eval references multiple case prefixes, those prefixes are preserved to avoid same-name collisions. Unsafe paths and target collisions fail during validate.

How grading works

Each assertion is graded by the first matching method:

1. Deterministic — code-based checks against the pre/post git state delta and run logs. No LLM involved.
2. LLM rubric — anything the deterministic grader doesn't recognize goes to an LLM judge with the agent output, expected output, and workspace file listing.
3. Skipped — if no LLM grader is configured (missing API key) or the grader errors, the assertion is marked skipped and excluded from the pass rate, never silently failed.

Recognized deterministic assertion patterns

The deterministic grader matches assertion text against these patterns (case-insensitive):

Anything else falls through to the LLM rubric. With should_trigger: false, the branch/commit/push/PR/MR checks invert: they pass only when those artifacts did not appear.

Structured deterministic assertions

For checks that should never fall through to the LLM rubric, use assertion objects:

{
  "type": "file_contains",
  "path": "README.md",
  "contains": "Quickstart",
  "text": "README includes a Quickstart section"
}

Supported type values are file_exists, file_not_exists, file_contains, file_not_contains, file_unchanged, command_ran, command_not_ran, tests_pass, git_clean, json_path, and json_path_not_exists. Common fields are path, contains, command, json_path, equals, and optional human-facing text.

CLI Commands

run

agent-skill-eval run \
  --skill <skill-dir> --evals <evals.json> \
  --agent opencode --agent claude-code --agent codex \
  [--agent-model claude-code=haiku] [--agent-model codex=gpt-5-mini] \
  [--reasoning-effort medium] \
  [--harness-base-url https://openrouter.ai/api/v1] \
  [--runs 3] [--concurrency 2] [--iteration 1] \
  [--baseline/--no-baseline] [--cleanup] \
  [--timeout 600] [--retries 1] \
  [--eval-id wif-too-long-stop] \
  [--side-effect-level static] \
  [--max-total-tokens-per-case 200000] [--budget-action fail] \
  [--pre-run-command "python scripts/setup.py"] \
  [--post-grade-command "python scripts/grade_external.py"] \
  [--post-run-command "python scripts/teardown.py"] \
  [--pricing-config pricing.json] \
  [--grader-model deepseek/deepseek-v4-flash] [--grader-base-url URL] \
  [--source-repo https://github.com/foo/bar.git] \
  [--workspace ./eval-workspace]

Key options:

• --agent-model, -m: model per agent as agent=model; a bare value applies to all agents. Repeatable.
• --reasoning-effort: reasoning setting passed to agents that support it (codex: model_reasoning_effort, e.g. minimal/low/medium/high/xhigh). Without it, agents silently inherit their own local config (e.g. ~/.codex/config.toml) — two machines can then benchmark different reasoning settings under the same command line. Agents without a pass-through flag print a warning and record null. The effective value lands in run_meta.json.
• --harness-base-url: injected as ANTHROPIC_BASE_URL (claude-code) / OPENAI_BASE_URL (codex, opencode) into the agent process.
• --runs, -n: repeat each (eval, agent, config) N times; enables pass@k stats. Results land in run-1/, run-2/, ... subdirectories.
• --timeout / --retries: per-run agent timeout (default 600s) and retries on timeout or non-zero exit (default 1). Also settable via ASE_AGENT_TIMEOUT / ASE_AGENT_RETRIES.
• --eval-id: run only the named case(s); repeatable. Unknown ids fail fast with the list of available ids, before any model call. The applied filter is recorded in evals_meta.json (selected_eval_ids) and summary.json (eval_ids), and all artifacts naturally describe only the selected cases. The cheap inner loop after a small skill edit is validate --eval-id X then run --eval-id X.
• --side-effect-level: run only the requested side-effect level(s); repeatable. Use this to run static and local-only checks before deliberate live-provider suites.
• Budget guards (--max-input-tokens-per-case, --max-non-cached-input-tokens-per-case, --max-output-tokens-per-case, --max-total-tokens-per-case, --max-reasoning-tokens-per-case, --max-cost-per-case, --max-duration-per-case, --budget-action warn|fail|stop-suite): per-case limits checked after each agent run (mid-run kills remain --timeout's job — budgets exist because a case can finish within the timeout and still be far too expensive). A violation is recorded in timing.json (budget_exceeded, budget_reason); with fail (the default) it also appends a failed method: "budget" assertion to the run's grading.json, and with stop-suite it additionally skips every run that hasn't started yet — useful to stop a live side-effect suite before more remote branches/PRs get created. --max-cost-per-case only fires when the run has a cost source (see Cost reporting); likewise --max-reasoning-tokens-per-case and --max-non-cached-input-tokens-per-case only fire when the run reported those values — an agent exposing less telemetry is never failed over an unknown quantity.
• Lifecycle hooks (--pre-run-command, --post-grade-command, --post-run-command): see Lifecycle hooks.
• --pricing-config: JSON file with USD-per-token rates for agents whose CLI reports no cost (see Cost reporting).

While the suite runs, each case prints start/finish lines (elapsed, tokens in/cached/out, cost), and progress.jsonl in the iteration directory gets one JSON event per line (suite_started, run_started, agent_started, agent_finished, run_finished, budget_exceeded, ...) — tail -f it from another shell to watch a long run.

report

agent-skill-eval report --workspace <skill-workspace> [--iteration N] [--format table|markdown|html] [--show-evidence] [--failures-only]

--show-evidence prints the evidence string for every failed or skipped assertion — the state diff for deterministic checks, the judge's reasoning for LLM checks. --failures-only limits the per-eval detail to runs that had at least one failed assertion. --format markdown emits a paste-ready table; --format html emits a standalone static HTML report.

run-all

agent-skill-eval run-all \
  --root path/to/plugin-repo --agent codex --side-effect-level static \
  [--source-repo https://github.com/foo/bar.git] \
  [--max-total-tokens-per-case 200000] [--budget-action fail] \
  [--pre-run-command "python scripts/setup.py"] \
  [--post-grade-command "python scripts/grade_external.py"] \
  [--post-run-command "python scripts/teardown.py"] \
  [--pricing-config pricing.json] \
  [--suite-concurrency 4] \
  [--plan-only] \
  [--changed-only --base-ref origin/main] \
  [--junit-xml run-all.xml] [--fail-on-assertions] [--min-pass-rate 0.8]

Runs every paired SKILL.md + evals/evals.json suite discovered under a repo by invoking the normal single-suite runner once per skill. Suites with no cases for the requested side-effect filter are recorded as skipped in run-all-iteration-N.json, which makes static or local-only repo sweeps safe before any live-provider evals.

Repo sweeps accept the same source-repo, model, timeout/retry, budget, pricing, and lifecycle hook controls as run; the aggregate manifest records the source repo, budget, pricing config path, suite concurrency, and hook counts used for the sweep. Use --plan-only to validate and write the planned/skipped suite manifest without running agents, --suite-concurrency to run independent skill suites in parallel while preserving deterministic manifest ordering, --changed-only --base-ref origin/main in CI to run only skills or eval suites touched by the branch, --junit-xml to publish the sweep as test results, and --fail-on-assertions or --min-pass-rate to make quality gates explicit.

report-all

agent-skill-eval report-all --workspace ./eval-workspace [--format table|markdown|html|json] [--failures-only]

Summarizes a repo-level run-all manifest with suite status, pass rates, duration, token totals, cost totals, and failure details. Use --format markdown for MR notes, --format html for a static artifact, and --format json when another tool should consume the raw manifest.

rerun-failed

agent-skill-eval rerun-failed --from ./eval-workspace/run-all-iteration-1.json

Prints targeted agent-skill-eval run --eval-id ... commands for failed repo-sweep cases. The generated commands preserve run context recorded in the manifest, including grader settings, agent model, source repo, pricing config, timeout/retry, reasoning effort, and budget flags.

status

agent-skill-eval status --workspace <skill-workspace> [--iteration N]

Answers "what happened in the last run?" without a model call: pass rates per configuration, failed/skipped/errored runs, token totals with the cached share, total cost (or unavailable), budget violations, hook outcomes, and whether cleanup.json recorded remote side effects. Reads the iteration's summary.json; for runs that predate it, a reduced summary is rebuilt from the saved artifacts.

compare

agent-skill-eval compare --workspace <skill-workspace> 1 2

Side-by-side pass rates of two iterations, per configuration, with pass-rate, time, and token deltas — the feedback loop for iterating on a SKILL.md.

validate

agent-skill-eval validate path/to/evals.json [--eval-id <id>]

Schema check plus referenced-fixture existence, duplicate-id detection, and structured-assertion required-field/path checks. Exit code 1 on any problem (CI-friendly). --eval-id validates only the selected case(s) (duplicate-id detection still covers the whole file); pair it with run --eval-id to confirm a targeted case is well-formed before spending a model call on it.

validate-all

agent-skill-eval validate-all --root path/to/repo [--format table|json]

Recursively validates every discovered */evals/evals.json under a repo and exits non-zero if any suite is invalid. This is the CI-friendly check for plugin repos with many skills. For plugin-style paths such as plugins/<plugin>/skills/<skill>/evals/evals.json, it also verifies the adjacent SKILL.md frontmatter and that the suite skill_name matches the skill directory.

doctor

agent-skill-eval doctor --skill ./skills/my-skill --evals ./evals/evals.json --agent codex

Preflights the local agent CLI, skill frontmatter, eval suite, LLM-grader credentials, and source-repo tooling (gh for GitHub, glab for GitLab). Use it before live or expensive suites to catch missing tools and auth early.

list

agent-skill-eval list [--root .] [--format table|json]

Recursively discovers eval suites (*/evals/evals.json) and skills (SKILL.md) under a directory, including plugin-style layouts such as plugins/<plugin>/skills/<skill>/evals/evals.json. JSON output includes a pairs section showing paired skills, skills without evals, and external eval suites.

grade

agent-skill-eval grade --workspace <iteration-dir> [--recompute-benchmark]

Re-grades existing outputs using saved evals_meta.json and state snapshots. Two caveats: LLM-graded assertions are re-evaluated from scratch and may flip verdicts, and because the original agent workspace is deleted after the run, the judge re-grades from the saved artifacts (agent output, logs) rather than the live workspace files.

cleanup

agent-skill-eval cleanup --workspace ./eval-workspace [--yes]

Only closes PRs/MRs and deletes remote branches recorded in cleanup.json. Never touches unrelated review requests, branches, or workspaces.

init

agent-skill-eval init my-skill [--output ./examples] [--template basic|review-diff|commit-pr|config-validator]
agent-skill-eval init my-skill --layout plugin --plugin demo-plugin --output /path/to/plugin-repo

Use commit-pr for live branch/push/PR workflows, review-diff for read-only review skills, and config-validator for skills that inspect bundled config fixtures. Use --layout plugin to scaffold directly into plugins/<plugin>/skills/<skill>/evals for Codex plugin repositories.

Lifecycle hooks

Skills with external side effects (push, PR/MR creation, provider APIs) need setup, provider-specific grading, and teardown that the harness can't know about. Instead of hard-coding providers, run exposes three shell-command hook points; each receives run metadata as ASE_* environment variables on top of the parent environment:

• --pre-run-command (repeatable): runs once before any agent case — seed or reset a scratch repo, run static checks on the skill text. A non-zero exit aborts the suite before a single model call. Env: ASE_SKILL_NAME, ASE_SKILL_DIR, ASE_EVALS_PATH, ASE_ITERATION, ASE_ITERATION_DIR, ASE_WORKSPACE_BASE, ASE_SOURCE_REPO, ASE_RUN_ID, ASE_EVAL_IDS (comma-separated).
• --post-grade-command (repeatable): runs once per (eval, agent, config) run, after grading.json is written and before the workspace is deleted — verify remote state, check side-effect files. On top of the suite-level vars it gets ASE_EVAL_ID, ASE_AGENT, ASE_WITH_SKILL (1/0), ASE_RUN_INDEX, ASE_WORKSPACE_PATH, ASE_OUTPUT_DIR, ASE_PRE_STATE_PATH, ASE_POST_STATE_PATH, ASE_TIMING_PATH, ASE_RUN_META_PATH, ASE_GRADING_PATH. If the hook prints a JSON array of {"text": ..., "passed": ..., "evidence": ...} objects to stdout, they are appended to the run's grading.json as method: "hook" assertion results and the summary is recomputed — external checks land in the same artifact report reads. A non-zero exit appends one failed hook check. Raw hook output is saved to outputs/post_grade_hooks.json.
• --post-run-command (repeatable): runs once after the suite (teardown, cleanup verification). Failures are recorded in summary.json and printed, but don't fail the run — the results already exist.

Hook commands run through the shell from the invocation directory, with a timeout of ASE_HOOK_TIMEOUT seconds (default 600).

agent-skill-eval run --skill ./skills/my-skill --evals ./evals/evals.json --agent codex \
  --eval-id feature-branch \
  --pre-run-command  "python scripts/setup_source_repo.py" \
  --post-grade-command "python scripts/grade_remote_state.py" \
  --post-run-command "python scripts/teardown_remote.py"

Live side-effect cookbook

Use this pattern for skills that create external state: pushes, PRs/MRs, Jira issues, cloud resources, provider API changes, tickets, comments, or anything that cannot be fully judged from the local workspace.

1. Classify cases by blast radius

Set side_effect_level on every case:

Start with static and local-only. Add live-safe only after the local cases prove the workflow shape. Treat live-external as a release candidate check, not an inner loop.

2. Make the source repo disposable

For git/MR workflows, use --source-repo pointing at a scratch repository that the eval owns. The pre-run hook should reset it to a known base state before each suite or case:

agent-skill-eval run \
  --skill ./plugins/acme/skills/commit-push-mr \
  --evals ./plugins/acme/skills/commit-push-mr/evals/evals.json \
  --source-repo git@gitlab.example.com:me/agent-skill-eval-scratch.git \
  --agent codex \
  --side-effect-level local-only \
  --eval-id wif-too-long-stop \
  --pre-run-command "python plugins/acme/skills/commit-push-mr/evals/scripts/setup_source_repo.py" \
  --post-grade-command "python plugins/acme/skills/commit-push-mr/evals/scripts/grade_gitlab.py" \
  --post-run-command "python plugins/acme/skills/commit-push-mr/evals/scripts/teardown_gitlab.py" \
  --post-run-command "python plugins/acme/skills/commit-push-mr/evals/scripts/verify_gitlab_clean.py"

The scratch repo should contain only eval fixtures and expected provider state. Do not point live evals at the real project until the suite has already passed against scratch state.

3. Keep setup, grading, teardown separate

Use one script for each job:

• Setup hook: creates or resets scratch state, checks auth, validates that the target is the expected test project, and aborts before model calls if anything is unsafe.
• Post-grade hook: reads ASE_* paths, inspects provider state while the agent workspace still exists, and prints JSON assertion rows to stdout.
• Teardown hook: closes/deletes only expected branches, MRs, tickets, labels, or resources created by this suite.
• Verify-clean hook: read-only proof that no expected eval branch, MR, ticket, or resource remains. Run it after teardown and independently after failures.

Post-grade hooks can append checks to grading.json by printing:

[
  {
    "text": "Exactly one GitLab MR was opened for the eval branch",
    "passed": true,
    "evidence": "MR !42 targets main and source branch proj-123-eval"
  }
]

This keeps external verification in the same report as deterministic file/git checks and LLM rubric checks.

4. Write cases around failure modes, not just happy paths

A good side-effect suite usually has these case types:

• Happy path: creates the intended branch/change/review request/resource in the scratch provider.
• Guardrail stop: invalid branch name, missing ticket ID, missing required input, unsupported environment, or too-large payload stops before side effects.
• No-op stop: no meaningful local changes means no commit, push, ticket, or API write.
• Natural trigger: same workflow without force_skill_invocation, proving the agent selects the skill from a realistic user request.
• Negative control: adjacent read-only prompt with should_trigger: false, proving no side effects are created.
• Cleanup regression: provider state is clean after teardown, even when the agent failed earlier.

For each side effect, pair a positive assertion with an explicit "nothing else" check in the provider hook. For example: one MR for the eval branch, no MR for main, no duplicate MR, no unexpected branch, and no untracked provider resource.

5. Use a cheap inner loop

Do not make full live runs your edit loop. Use this order:

agent-skill-eval validate path/to/evals.json --eval-id wif-too-long-stop
agent-skill-eval doctor --skill path/to/skill --evals path/to/evals.json --agent codex
agent-skill-eval run --skill path/to/skill --evals path/to/evals.json \
  --agent codex --eval-id wif-too-long-stop --side-effect-level local-only \
  --no-baseline --max-total-tokens-per-case 150000 --budget-action fail
agent-skill-eval status --workspace ./eval-workspace/<skill>-workspace
agent-skill-eval report --workspace ./eval-workspace/<skill>-workspace --show-evidence

Only after a targeted case passes should you run more cases, more agents, baseline mode, or run-all.

For plugin repos with many skills, use planning before execution:

agent-skill-eval validate-all --root /path/to/plugin-repo --strict
agent-skill-eval run-all --root /path/to/plugin-repo \
  --side-effect-level static --changed-only --base-ref origin/main --plan-only
agent-skill-eval run-all --root /path/to/plugin-repo \
  --side-effect-level local-only --changed-only --base-ref origin/main \
  --suite-concurrency 2 --fail-on-assertions

6. Use budgets and cleanup as gates

Side-effect suites need explicit failure gates:

• Set --timeout, --retries, --max-total-tokens-per-case, --max-duration-per-case, and usually --budget-action fail or stop-suite.
• Use --eval-id while debugging a single risky case.
• Keep --cleanup or teardown hooks enabled for any run that can push, open an MR, or write provider state.
• Run a read-only verify-clean command after failures before starting another live run.
• Archive or paste status, report --show-evidence, and provider hook evidence when claiming a skill improved.

The minimum credible claim for a side-effect skill is: targeted case passed, provider hook passed, cleanup verification passed, and the command/model/harness version are recorded. A stronger claim adds repeated runs, natural trigger and negative-control cases, and a before/after comparison with compare.

Example skills

Five example skills ship with the repo, chosen to exercise different grading surfaces:

Each has a matching eval suite under examples/. skills/ holds the artifacts being evaluated; examples/ holds the test cases — so you can test one skill against many suites or one suite against many skill versions.

Reading results

Workspace layout

eval-workspace/
└── <skill>-workspace/
    └── iteration-1/
        ├── evals_meta.json          # eval definitions, selected filters, skill_hash (used by `grade`)
        ├── cleanup.json             # manifest of artifacts created by this run
        ├── benchmark.json           # aggregate stats per (agent, config)
        ├── summary.json             # one-file run summary (used by `status`)
        ├── progress.jsonl           # live per-run events; tail -f during the run
        └── eval-<id>/<agent>/<config>/   # config = with_skill | without_skill
            ├── run-N/               # only when --runs > 1
            ├── outputs/
            │   ├── output.txt       # final agent output
            │   ├── stdout.log / stderr.log
            │   ├── post_grade_hooks.json   # only with --post-grade-command
            │   └── pre_state.json / post_state.json
            ├── timing.json          # tokens (incl. non-cached/reasoning splits), duration, exit_code, timed_out, retries, budget verdict
            ├── grading.json         # per-assertion results
            ├── trajectory.json      # normalized lifecycle events for this run
            └── run_meta.json        # agent, with_skill, run_index, model, side_effect_level, skill_hash, CLI/harness versions, ...

grading.json

{
  "assertion_results": [
    {
      "text": "A new git branch was created",
      "passed": false,
      "method": "deterministic",
      "skipped": false,
      "evidence": "No new branch appeared in this run. current_branch='main'"
    }
  ],
  "summary": {"passed": 2, "failed": 1, "skipped": 0, "total": 3, "pass_rate": 0.667}
}

To debug a failure: find "passed": false entries, read evidence, compare pre_state.json/post_state.json, then check outputs/output.txt for the agent's full response. Or just run agent-skill-eval report --show-evidence.

Iterating on a skill

1. agent-skill-eval run ... → 2. agent-skill-eval report --show-evidence → 3. edit SKILL.md → 4. agent-skill-eval run --iteration 2 ... → 5. agent-skill-eval compare --workspace ... 1 2

Comparing agents: reading with/without-skill deltas

Run several agents in one invocation and every agent gets its own baseline comparison:

agent-skill-eval run \
  --skill ./skills/fix-failing-tests \
  --evals ./examples/fix-failing-tests/evals/evals.json \
  --agent claude-code --agent-model claude-code=claude-haiku-4-5-20251001 \
  --agent opencode --agent-model opencode=deepseek/deepseek-v4-flash:free \
  --runs 3

agent-skill-eval report --workspace ./eval-workspace/fix-failing-tests-workspace --format markdown

The report shows one row per (agent, config) and a delta per agent (numbers below are illustrative):

Delta (with_skill - without_skill):

• claude-code: pass rate +33.3%, time -11.5s, tokens -570, cost -0.0016 USD
• opencode: pass rate +33.3%, time -6.6s, tokens -360, cost +0.0000 USD

How to read it: the delta rows are the skill's measured value per agent — here the skill lifts pass rate by ~33 points on both agents and saves time/tokens, the strongest possible signal. A positive pass-rate delta with a large token increase means the skill works but is verbose; a near-zero delta means that agent doesn't benefit (check report --show-evidence to see whether it never triggered the skill). The same numbers are machine-readable in benchmark.json: per-config stats under run_summary, per-agent deltas under deltas, keyed by agent name.

Environment variables

• OPENROUTER_API_KEY / OPENAI_API_KEY: API key for LLM rubric grading
• OPENAI_BASE_URL: custom grader endpoint (defaults to OpenRouter)
• ASE_AGENT_TIMEOUT / ASE_AGENT_RETRIES: harness timeout/retry defaults
• ASE_HOOK_TIMEOUT: lifecycle hook command timeout in seconds (default 600)
• ASE_KEEP_WORKSPACE: keep per-eval workspaces for debugging

Agent-specific details

Cost reporting

timing.json records cost_usd per run, taken from the agent CLI itself (claude: total_cost_usd, opencode: per-step cost). When no cost is available, cost_usd is null — unavailable, not $0.00 — and reports/benchmark.json show n/a instead of a misleading zero. Cost stats are computed only over the runs that reported a cost (cost_runs in benchmark.json, runs_with_cost in summary.json).

Codex's CLI reports token counts but no cost. To get cost numbers anyway, pass --pricing-config pricing.json with your provider's USD-per-token rates, keyed by the exact --agent-model value (fields mirror OpenRouter's pricing schema):

{
  "gpt-5.4-mini": {
    "prompt": 2.5e-07,
    "completion": 2e-06,
    "input_cache_read": 2.5e-08
  }
}

The computed value lands in cost_usd with cost_usd_source: "pricing-config". Cached input is billed at input_cache_read (falling back to prompt); note that codex reports input_tokens including cache reads while claude-code and opencode report them separately — the harness accounts for this per agent. summary.json splits the totals (tokens.cached, tokens.non_cached_input, tokens.cached_pct), because on long agent sessions most input tokens are cache reads and the raw input_tokens number wildly overstates billable spend. Each run's timing.json also persists non_cached_input_tokens (computed with that run's CLI semantics) and reasoning_output_tokens when the agent reports reasoning telemetry (codex turn.completed, opencode step_finish) — null means unreported, not zero.

One sharp edge: the claude CLI prices runs at Anthropic list prices regardless of the endpoint it talks to. When you route claude-code through OpenRouter (--harness-base-url or ANTHROPIC_BASE_URL), agent-skill-eval reconciles the cost by recomputing it from the run's token counts and OpenRouter's published per-model rates. timing.json then shows:

• cost_usd_source: "cli" (the CLI's own number), "openrouter-pricing" (reconciled), "pricing-config" (computed from --pricing-config), or "cli-unreconciled" (reconciliation failed — e.g. a short model alias like haiku that can't be mapped to an OpenRouter slug — so cost_usd is the CLI's list-price estimate and actual billing differs)
• cost_usd_cli: the CLI's original estimate, kept alongside the reconciled value

Pin full model IDs (claude-haiku-4-5-20251001, not haiku) to keep reconciliation working.

Development

uv run --extra dev pytest -q
uv run --extra dev ruff check src/ tests/
uv run --extra dev ruff format --check src/ tests/

A fake agent type exists for offline testing of the full run→grade→report pipeline (used by the CI smoke test). To record a demo cast: ./scripts/record-demo.sh.

License

MIT