Skip to main content

skillbench

Test your Claude Code skills and agent workflows, the way you test code.

You write a small YAML file that says "send this to Claude, then check these things". skillbench runs Claude Code for real on a copy of your repo, records everything it did, and tells you pass or fail.


1. Install

You need Python 3.11+, git, and Claude Code installed and logged in (check with claude --version). Then install with whichever tool you use:

pip install skillbench-cli
pipx install skillbench-cli
uv tool install skillbench-cli

The command is skillbench either way. To try it without installing: uvx --from skillbench-cli skillbench init.

2. Try it (2 minutes)

In your repo:

skillbench init      # creates a skillbench/ folder with two example tests
skillbench skill     # runs the skill test
skillbench agent     # runs the agent test

You'll see something like:

[1/1] hello-greets-by-name  default model
hello-greets-by-name: pass in 2.3s
dashboard: skillbench/runs/index.html

Open skillbench/runs/index.html in your browser to see exactly what Claude did: every message, tool call and check.

That's it. The rest of this page shows how to write your own tests.


3. Two kinds of test

Skill test Agent test
Question it answers "When my skill runs, does it do the right thing?" "Given a normal request, does Claude pick the right skills and get the job done?"
Lives in skillbench/skills/ skillbench/agent/
Run with skillbench skill skillbench agent
You write skill: and input: prompt:

A skill test

skillbench/skills/grilling.yaml

skill: grilling                 # your skill's name
input: an idea for a todo app   # what to send it
fixture: .                      # run it on a copy of this repo

assert:
  - final_text_contains: "Q1"
  - tool_not_called: Write

skillbench finds the skill wherever it lives in your repo (.claude/skills/, skills/, .github/skills/ or a plugin) and runs it as /grilling an idea for a todo app.

An agent test

skillbench/agent/csv-export.yaml

prompt: add CSV export to the report page
fixture: .

answers:                       # replies, sent one at a time when Claude asks something
  - Yes to all your recommendations.
  - approved

assert:
  - skill_invoked: planner
  - tool_order: [Read, Edit, Bash]
  - command_succeeds: python -m pytest -q

Nothing is forced here. Claude sees every skill in your repo and decides for itself, and the test checks what it chose.


4. Checks you can make

Put any of these under assert:.

Check Passes when
skill_invoked: grilling Claude used that skill
skill_not_invoked: grilling Claude didn't use that skill
tool_order: [Read, Edit] Claude used these tools in this order (other tools in between are fine)
tool_not_called: Write Claude never used that tool
final_text_contains: "Done" Claude's last message contains the text
file_contains: {path: notes.md, text: "approved"} That file exists afterwards and contains the text
command_succeeds: python -m pytest -q The command runs successfully in the workspace afterwards

A test passes when every check passes.


5. Which repo Claude works in (fixture:)

Claude always works on a copy, so your real files are never touched.

fixture: Claude gets
(leave it out) An empty folder
. A copy of your repo, including work you haven't committed
some/folder A copy of that folder
https://github.com/you/app.git A fresh clone
{git: https://github.com/you/app.git, ref: v1.2} A clone at that branch, tag or commit

Paths are relative to your repo root. The copy is kept after the run so you can look inside, and its path is in the dashboard.


6. Running tests

skillbench skill                 # all skill tests
skillbench skill grilling        # just the tests for one skill (or one test by name)
skillbench agent csv-export      # one agent test
skillbench agent --repeat 5      # run each test 5 times
skillbench agent --model claude-sonnet-5-5

While it runs you'll see the current test, what Claude is doing, and roughly how long is left.


7. More options

You won't need these to start.

Check a skill gets picked on its own. List prompts that should and shouldn't trigger it. Each runs briefly and stops as soon as Claude decides:

skill: grilling
triggers:
  should: ["stress-test my plan"]
  should_not: ["fix this typo"]

Answer Claude yourself. Add interactive: true and skillbench asks you in the terminal whenever Claude is waiting for a reply. Press Enter on an empty line to finish.

Check for flakiness. Claude doesn't do exactly the same thing every time. --repeat 5 (or repeat: 5 in the test) runs it 5 times and shows how often each check passed. Add min_pass_rate: 0.8 to pass when 4 out of 5 runs pass.

Set defaults for every test in skillbench/skillbench.yaml:

model: claude-sonnet-5-5
timeout_s: 600        # seconds before a turn is stopped
fixture: .
repeat: 1
min_pass_rate: 1.0

A test's own settings win over these, and command-line flags win over both.

All test fields:

Field Meaning
skill / input Skill tests: which skill, and what to send it
prompt Agent tests: the first message
answers Replies sent one per turn
interactive Ask you in the terminal for replies
triggers Prompts that should / shouldn't pick the skill
fixture Which repo Claude works in
assert The checks
name A display name (defaults to the file name)
timeout_s, repeat, min_pass_rate As above

Good to know

  • It costs real usage. Every run is a real Claude session. Start with skill tests; long agent tests take longer and cost more.
  • Your personal setup is ignored. Only the skills and plugins in the repo being tested are loaded, so results don't depend on whose machine runs them.
  • Only test repos you trust. Claude has full tool access inside the copy, and command_succeeds runs commands there.
  • command_succeeds uses your system shell (sh on Mac/Linux, cmd on Windows), so keep commands simple if you share tests across systems.
  • Results are in skillbench/runs/, which is git-ignored. Each run has a run.json and Claude's raw output.
Status Meaning
pass Every check passed
fail Claude finished, but a check failed
error Claude crashed or timed out, or the skill or fixture couldn't be found

Metadata

Release files for skillbench-cli 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for skillbench-cli 0.1.0
File Size Uploaded Provenance Actions
skillbench_cli-0.1.0.tar.gz 45.8 kB Without provenance Details for skillbench_cli-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for skillbench-cli 0.1.0
File Interpreter ABI Platform Actions
skillbench_cli-0.1.0-py3-none-any.whl Python 3 none any Details for skillbench_cli-0.1.0-py3-none-any.whl

Total release size: 88.2 kB

Release files / skillbench_cli-0.1.0.tar.gz

Download URL skillbench_cli-0.1.0.tar.gz
Size 45.8 kB
Tags Source
SHA-256 checksum
How to use checksums
56261128063cd08afc9219a42b07de49d6abf14b8e0598ffe35ae39f64297f27
BLAKE2b-256 checksum
How to use checksums
fe4640f7ee0a1e1f5c77f91d3c5ead18421fc0774cb577a6923f0624a0b946fa
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.13 {"installer":{"name":"uv","version":"0.11.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / skillbench_cli-0.1.0-py3-none-any.whl

Download URL skillbench_cli-0.1.0-py3-none-any.whl
Size 42.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
6ce6e274c8278dadc12c36277b290068aa4af03c65e3c5acbf35117d66e36069
BLAKE2b-256 checksum
How to use checksums
57c718f1446564d6381b40e8d3b76f5e76e78de211ced1fd9082ff4509251be9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.13 {"installer":{"name":"uv","version":"0.11.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for skillbench-cli 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page