caliper-eval 0.6.0

CLI for evaluating Claude Code skills and AI agents

Caliper — Reliability testing for agent skills

Run your skill k times, get a pass@k score you can track and compare, and prove the skill beats the base agent. Works with the agent you already use — Claude Code, Codex, or Pi.

npx skills@latest add edonadei/caliper

Run each task with and without the skill, and Caliper shows you the difference:

ID      Task                              k (3)   pass@k
task-1  Writes a conventional commit msg  3/3     100%     PASS
task-2  Generates a valid config file     2/3      96%     PASS

With skill     98%    ###################-
No skill       55%    ###########---------
Delta          +43%   up

---

Agent skills are hard to test. A skill that works on your machine, on this prompt, today, might fail tomorrow after a model update or a one-line prompt edit. Caliper makes reliability measurable: define what success looks like, run the skill repeatedly, and get a pass@k score you can track over time.

Use Caliper to answer questions like:

• Did my prompt edit actually improve the skill?
• Is the skill doing the work, or would the base agent pass without it?
• Does it still pass the workflows it passed last week?
• Which agent — Claude Code, Codex, or Pi — runs this skill more reliably?

---

Quick start

Path A — Agentic (let your agent drive)

1. Install the skills

npx skills@latest add edonadei/caliper

2. Generate a spec interactively

In your agent (Claude Code or Codex):

/grill-skill ./my-skill/SKILL.md

grill-skill reads your SKILL.md, interviews you, and writes a 3-task .eval.yaml (happy path, edge case, adversarial).

3. Run and measure

/evaluate-skill run my-skill.eval.yaml --k 3 --baseline

Browse past runs:

/evaluate-skill list
/evaluate-skill report my-skill

Path B — CLI (run it yourself)

1. Install the CLI

pipx install caliper-eval   # requires Python 3.10+

2. Write a spec

# my-skill.eval.yaml
skill:
  path: ./SKILL.md

tasks:
  # Autorater — the LLM judge reads the transcript and decides
  - name: Writes a conventional commit message
    prompt: "Summarize the staged git diff as a commit message."
    expect: >
      The response is a conventional-commit message: a concise subject
      line under 72 characters, followed by a body explaining why the
      change was made, not just what changed.

  # Script execution — a deterministic Python assertion
  - name: Generates a valid config file
    cleanup: rm -f /tmp/app.config.json
    prompt: "Generate a config at /tmp/app.config.json with a 'port' of 8080."
    assert: |
      import json
      from pathlib import Path
      data = json.loads(Path("/tmp/app.config.json").read_text())
      assert data["port"] == 8080

expect: is graded by the judge LLM; assert: runs locally as Python. Use either or both.

The spec never names an engine — the skill and judge default to claude-code, and you pick a different agent/model at run time with --model / --judge-model (see Choosing an engine).

3. Run it

caliper run my-skill.eval.yaml --k 3 --baseline

4. Read the output

CALIPER  -  my-skill  -  k=3  -  claude-code

ID      Task                              k (3)   pass@k
task-1  Writes a conventional commit msg  3/3     100%     PASS
task-2  Generates a valid config file     2/3      96%     PASS

With skill     98%    ###################-
No skill       55%    ###########---------
Delta          +43%   up

Results saved to .caliper/results/my-skill/2026-06-19T14-23-01Z.json

Not sure what to put in a spec?

The Eval Starter Pack has four copy-paste templates, each catching a real agent failure (false success, tool misuse, runaway loops, prompt regressions). Every template runs green as-is against a bundled example, then points at your own skill by editing two or three commented lines.

---

How it works

.eval.yaml spec
      │
      ▼
  Harness  ──── runs your skill against the agent (Claude Code / Codex / Pi)
      │
      ▼
   Judge   ──── LLM autorater and/or deterministic Python assertions
      │
      ▼
  pass@k score + saved transcript

Each attempt runs in an isolated temporary home with no session history. Results are saved as JSON you can inspect and diff later.

---

Agent skills

The repo ships two agent skills. Install both with:

npx skills@latest add edonadei/caliper

evaluate-skill — run and manage evals

Create, validate, run, and summarize evals from inside your normal workflow — no separate terminal needed. The skill installs Caliper automatically if it's missing.

Then use it in Claude Code:

/evaluate-skill run my-skill.eval.yaml --k 3
/evaluate-skill validate my-skill.eval.yaml

Or in Codex:

Use the evaluate-skill skill to run my-skill.eval.yaml with k=3 and summarize the result.

grill-skill — create evals interactively

Don't have evals yet? grill-skill guides you through creating them. It reads your SKILL.md, interviews you about what good behavior looks like, and generates a 3-task spec (happy path, edge case, adversarial). Then it runs the eval and loops — k=1 to validate, k=3 to measure, baseline before you commit.

/grill-skill ./my-skill/SKILL.md

No path needed if you're already in the skill's directory:

/grill-skill

If an .eval.yaml already exists next to your skill, grill-skill reads the existing tasks and interviews you about gaps instead of starting from scratch.

---

Core concepts

Term	What it is
Spec	A .eval.yaml file that describes the skill, judge, and tasks to run
Backend	The CLI agent that executes the skill (claude-code, codex, pi)
Judge	What decides pass/fail — an LLM reading the transcript (expect:), Python assertions (assert:), or both
pass@k	Reliability score: run k times, measure how often the skill succeeds
Baseline	Re-run the same tasks without the skill to prove the skill is doing the work
Attempt	One isolated run of a single task — fresh temporary home, no session history

---

Choosing an engine

The engine (backend + model) is a runtime axis, not a spec field — the spec describes what is tested and how success is judged, and you pick the agent that runs and grades it at invocation. Both default to claude-code; select a different one with --model / --judge-model:

caliper run my-skill.eval.yaml                          # claude-code (default)
caliper run my-skill.eval.yaml --model codex            # codex, its default model
caliper run my-skill.eval.yaml --model codex:gpt-5-codex
caliper run my-skill.eval.yaml --model pi --judge-model claude-code

Backend	Requires	Best for
claude-code	Claude Code CLI installed and authenticated	Testing Claude Code slash-command skills
codex	Codex CLI installed (npm install -g @openai/codex)	Testing Codex skills
pi	pi CLI installed (npm install -g @earendil-works/pi-coding-agent) and authenticated	Testing pi skills (agentskills.io)

Caliper runs skills only through CLI agents — every backend can actually load and run a skill. There is no direct-API backend: to run against API-priced billing, configure one of these CLIs with an API key (e.g. ANTHROPIC_API_KEY / OPENAI_API_KEY) rather than selecting a separate backend.

The skill engine and judge engine are independent — you can test a Codex skill with a Claude judge, or any other combination, by pairing --model with --judge-model.

Claude Code setup

Install and authenticate the claude CLI. --model claude-code uses your existing Claude Code auth — no extra configuration needed.

Codex setup

npm install -g @openai/codex
codex login

--model codex calls codex exec. If the Codex desktop app is installed, Caliper prefers the app-bundled binary over codex on PATH. Set CODEX_CLI_PATH to force a specific binary.

pi setup

npm install -g @earendil-works/pi-coding-agent
pi   # then authenticate (e.g. /login for a subscription provider, or set the provider API key)

--model pi runs pi --print --mode json and loads the skill natively via pi's --skill flag (the agentskills.io standard). It reuses your ~/.pi/agent auth and settings — the :model half of --model pi:<model> overrides pi's configured default when set. Set PI_CLI_PATH to force a specific binary. Note: pi's built-in default provider is google, so running --model pi with no model relies on your pi config to resolve a provider you are authenticated for.

Check installed CLI versions:

caliper update-cli --check

---

Recommended workflow

1. Create a spec for one behavior you care about.
2. Run with --k 1 while iterating on the spec.
3. Add assert: for facts an LLM judge might guess wrong (files, JSON, command output).
4. Move to --k 3 or higher once the task is stable.
5. Add --baseline to prove the skill is making a difference.
6. Commit the spec alongside the skill so contributors can run the same eval.

/evaluate-skill run my-skill.eval.yaml --k 3 --baseline --verbose

---

Spec format

To scaffold a spec, use the evaluate-skill or grill-skill skill, or hand-write the YAML below.

skill:
  path: ./SKILL.md              # path to the skill file (optional for baseline-only runs)

# Note: there is no `backend`/`model` or `judge:` block. The engine is a runtime
# axis — pass `--model` / `--judge-model` at run time (default: claude-code).

sandbox:
  extra_path:
    - ./bin                     # prepended to PATH inside each attempt
  forbidden_files:
    - ".*\\.eval\\.yaml$"       # prevents agent from reading the spec
    - "./.caliper/.*"           # prevents agent from reading saved results

tasks:
  - name: Short task name
    setup: <shell command>      # optional, runs before each attempt
    cleanup: <shell command>    # optional, always runs after each attempt
    prompt: <prompt sent to the agent>
    expect: <natural-language success condition>
    assert: |
      # optional inline Python assertion
      assert True

  - name: Task with external assertion script
    prompt: "Generate a report"
    assert: ./assertions/check_report.py

Each task needs at least one of expect or assert. Task IDs are assigned automatically as task-001, task-002, and so on.

---

Judging

LLM autorater (expect:)

The judge engine reads the full attempt transcript and decides whether the expect condition was met. When the backend captures tool-call traces (Claude Code, Codex, pi), those traces are included — the judge can verify things like "the agent used tool X" without relying on the final text alone.

The judge engine is chosen at run time and defaults to claude-code; point it at a different agent with --judge-model (e.g. --judge-model codex), independently of the skill's --model.

Deterministic assertions (assert:)

Python assertions run locally. Use these for facts the LLM judge might guess:

• file exists / exact file contents
• JSON / schema validity
• command output
• images or screenshots
• repository state

tasks:
  - name: Writes an output file
    cleanup: rm -f /tmp/out.txt
    prompt: "Write hello world to /tmp/out.txt"
    assert: |
      from pathlib import Path
      path = Path("/tmp/out.txt")
      assert path.exists(), "Output file was not created"
      assert path.read_text().strip() == "hello world"

When both expect and assert are present, both must pass.

---

CLI reference

Command	Description
caliper run <spec>	Run an evaluation spec
caliper validate <spec>	Validate a spec file
caliper list [spec]	List specs and saved runs
caliper report <spec-or-result>	Re-render saved results
caliper compare <A> <B>	Diff two saved runs of the same eval, task by task
caliper update-cli [backend]	Check or update installed agent CLI versions

caliper run flags

Flag	Default	Description
--k INT	3	Attempts per task
--baseline	off	Also run each task without the skill
--workers INT	4	Parallel task workers
--timeout INT	120	Seconds per attempt
--fail-fast INT	0	Stop a task after N consecutive infra_error/timeout attempts (0 disables)
--model TARGET	claude-code	Skill engine — backend and/or model (see below)
--judge-model TARGET	claude-code	Judge engine — backend and/or model (see below)
--verbose	off	Show per-attempt judge reasoning
--output PATH	—	Also save results JSON to a specific path

--model and --judge-model syntax

The engine is not stored in the spec — these flags select it, defaulting to claude-code when omitted. Both accept a backend:model compound value, a bare backend name, or a bare model name:

# Backend and model together
caliper run my-skill.eval.yaml --model codex:gpt-5-codex

# Backend only (that backend's default model)
caliper run my-skill.eval.yaml --model codex

# Model only (backend stays claude-code)
caliper run my-skill.eval.yaml --model claude-sonnet-4-6

# Select the judge engine independently
caliper run my-skill.eval.yaml --model codex --judge-model claude-code:claude-haiku-4-5-20251001

Accepted backends: claude-code, codex, pi (alias: claude → claude-code). The actual engine used is recorded in each saved run's RunMeta, so results stay traceable even though the spec doesn't pin it.

---

Comparing two runs (caliper compare)

An ablation compares two runs of the same eval — a full skill vs. a shortened variant, or the same skill over time. caliper compare <A> <B> diffs two already-saved runs task by task, so you don't hand-write a JSON script to answer "did this change regress?".

# Latest run of each spec (a bare spec name resolves to its latest run)
caliper compare commit-simple-full commit-simple-short

# Pin specific runs by pointing at their results JSON
caliper compare .caliper/results/demo/2026-07-01T10-00-00Z.json \
                .caliper/results/demo/2026-07-02T09-00-00Z.json

# Machine-readable diff for a ship / no-ship decision
caliper compare A B --format json

Each positional (A, B) is addressed exactly like report's argument: a spec name (→ its latest run) or a path to a results JSON. There are no --run-a/-b flags — pin a historical run by naming its JSON path.

──────────────────── CALIPER  —  compare  —  commit-simple ─────────────────────
    A 2026-07-01T10-00-00Z (claude-code)   ·   B 2026-07-02T09-00-00Z (claude-code)   ·   k=5

╭──────────────────┬──────────┬──────────┬────────┬─────────┬─────────╮
│ Task             │ A pass@k │ B pass@k │      Δ │ A strip │ B strip │
├──────────────────┼──────────┼──────────┼────────┼─────────┼─────────┤
│ commits cleanly  │   100.0% │   100.0% │      — │ ✓✓✓✓✓   │ ✓✓✓✓✓   │
│ handles conflict │    80.0% │    40.0% │ -40.0% │ ✓✓✓✓✗   │ ✓✗✓✗✗   │
│ pushes upstream  │    80.0% │        — │      — │ ✓✓✓✓✗   │ ⊘⊘⊘⊘⊘   │
╰──────────────────┴──────────┴──────────┴────────┴─────────┴─────────╯

 A 90.0%   B 70.0%   Δ (matched) -20.0% ↓
 ⚠ 1 regression: handles conflict
 ⊘ 1 unmeasured (excluded from Δ): pushes upstream
 unmatched — only in A: flaky task   only in B: new task

How the diff reads:

• Tasks are matched by name. task_id is only positional, so name is the stable identity — reordered tasks still line up. A task present in only one run is listed as unmatched and left out of the delta.
• Δ is b − a. A negative Δ renders red and flags the task as a regression (any-below rule: B below A by any amount).
• pass@k excludes unusable attempts. The strips reuse the run report's glyphs; ⊘ marks an unusable attempt (rate-limit / timeout / judge error). A task with no usable attempts on a side shows — (unmeasured) and is never counted as a regression — infra noise can't fake a loss.
• The headline Δ (matched) averages each side over only the tasks measured on both sides, so it is strictly like-for-like.
• Guards for a k mismatch or different spec names print as warnings in the header and in --format json (k_mismatch, spec_mismatch, warnings), so an agent driving compare sees them too.

The --format json output serializes the full comparison (per-task scores, deltas, regression/has_regression flags, unmatched task lists, and the warnings) for scripting.

---

Scoring

Every attempt carries a typed outcome, so infrastructure and judge noise are not scored as task failure:

Outcome	Meaning	Counts toward pass@k?
pass	satisfied the task's judge(s)	✅ success
task_fail	the skill genuinely failed the task	✅ attempt
cheat	a forbidden-file read was detected	✅ attempt
infra_error	harness failure — nonzero exit, or a detected rate-limit / spending-cap	❌ unusable
timeout	exceeded the time budget with no result	❌ unusable
judge_error	the judge produced no verdict (unparseable / errored autorater)	❌ unusable

passed is retained in the JSON as a convenience, equal to outcome == pass.

For each task, pass@k is computed over the usable attempts (those that got a fair shot); unusable attempts leave the denominator and are reported as a separate "N unusable" count:

usable  = pass + task_fail + cheat
pass@k  = 1 - (1 - successes / usable) ^ usable      # None if usable == 0

The aggregate score is the average task pass@k, skipping tasks with no usable attempts. With --baseline, Caliper runs the same tasks without the skill and reports the delta. A throttled or judge-flaked run therefore no longer masquerades as a regression.

For persistent infrastructure failures, --fail-fast N can stop scheduling new attempts for a task after N consecutive infra_error or timeout outcomes. The default 0 keeps the historical behavior and runs all k attempts. An early-stopped task is shown as ABORTED in the report; if every completed attempt was unusable, its pass_at_k remains null and it is skipped in the aggregate score.

---

Project layout

caliper/
  commands/       CLI command implementations
  harness/        Agent execution backends (Claude Code, Codex, API)
  judge/          LLM and script judging implementations
  schema/         Eval spec and result models
  runner.py       Evaluation orchestration
skills/
  evaluate-skill/ Agent skill for running Caliper from Claude Code or Codex
  grill-skill/    Agent skill for creating and iterating on evals interactively
tests/            Pytest coverage for harnesses, judges, and runner behavior

---

Contributing

Contributions are welcome. See CONTRIBUTING.md for good first areas, the pre-PR checklist, the ruff formatting convention and pinned version, and the one-time pre-commit install step.

---

Troubleshooting

codex judge failed: model ... is not supported The model name is not available to your Codex account. Use a model that codex exec --model <name> accepts.

codex CLI not found

npm install -g @openai/codex

claude command not found Install and authenticate Claude Code, or switch the backend to codex or pi.

A task passes only because of assert: When a task has only assert:, no LLM judge runs. Add expect: if you also want an LLM to evaluate the transcript.