skillbench
Test your Claude Code skills and agent workflows, the way you test code.
You write a small YAML file that says "send this to Claude, then check these things". skillbench runs Claude Code for real on a copy of your repo, records everything it did, and tells you pass or fail.
1. Install
You need Python 3.11+, git, and Claude Code installed and logged in (check with claude --version). Then install with whichever tool you use:
pip install skillbench-cli
pipx install skillbench-cli
uv tool install skillbench-cli
The command is skillbench either way. To try it without installing: uvx --from skillbench-cli skillbench init.
2. Try it (2 minutes)
In your repo:
skillbench init # creates a skillbench/ folder with two example tests
skillbench skill # runs the skill test
skillbench agent # runs the agent test
You'll see something like:
[1/1] hello-greets-by-name default model
hello-greets-by-name: pass in 2.3s
dashboard: skillbench/runs/index.html
Open skillbench/runs/index.html in your browser to see exactly what Claude did: every message, tool call and check.
That's it. The rest of this page shows how to write your own tests.
3. Two kinds of test
| Skill test | Agent test | |
|---|---|---|
| Question it answers | "When my skill runs, does it do the right thing?" | "Given a normal request, does Claude pick the right skills and get the job done?" |
| Lives in | skillbench/skills/ |
skillbench/agent/ |
| Run with | skillbench skill |
skillbench agent |
| You write | skill: and input: |
prompt: |
A skill test
skillbench/skills/grilling.yaml
skill: grilling # your skill's name
input: an idea for a todo app # what to send it
fixture: . # run it on a copy of this repo
assert:
- final_text_contains: "Q1"
- tool_not_called: Write
skillbench finds the skill wherever it lives in your repo (.claude/skills/, skills/, .github/skills/ or a plugin) and runs it as /grilling an idea for a todo app.
An agent test
skillbench/agent/csv-export.yaml
prompt: add CSV export to the report page
fixture: .
answers: # replies, sent one at a time when Claude asks something
- Yes to all your recommendations.
- approved
assert:
- skill_invoked: planner
- tool_order: [Read, Edit, Bash]
- command_succeeds: python -m pytest -q
Nothing is forced here. Claude sees every skill in your repo and decides for itself, and the test checks what it chose.
4. Checks you can make
Put any of these under assert:.
| Check | Passes when |
|---|---|
skill_invoked: grilling |
Claude used that skill |
skill_not_invoked: grilling |
Claude didn't use that skill |
tool_order: [Read, Edit] |
Claude used these tools in this order (other tools in between are fine) |
tool_not_called: Write |
Claude never used that tool |
final_text_contains: "Done" |
Claude's last message contains the text |
file_contains: {path: notes.md, text: "approved"} |
That file exists afterwards and contains the text |
command_succeeds: python -m pytest -q |
The command runs successfully in the workspace afterwards |
A test passes when every check passes.
5. Which repo Claude works in (fixture:)
Claude always works on a copy, so your real files are never touched.
fixture: |
Claude gets |
|---|---|
| (leave it out) | An empty folder |
. |
A copy of your repo, including work you haven't committed |
some/folder |
A copy of that folder |
https://github.com/you/app.git |
A fresh clone |
{git: https://github.com/you/app.git, ref: v1.2} |
A clone at that branch, tag or commit |
Paths are relative to your repo root. The copy is kept after the run so you can look inside, and its path is in the dashboard.
6. Running tests
skillbench skill # all skill tests
skillbench skill grilling # just the tests for one skill (or one test by name)
skillbench agent csv-export # one agent test
skillbench agent --repeat 5 # run each test 5 times
skillbench agent --model claude-sonnet-5-5
While it runs you'll see the current test, what Claude is doing, and roughly how long is left.
7. More options
You won't need these to start.
Check a skill gets picked on its own. List prompts that should and shouldn't trigger it. Each runs briefly and stops as soon as Claude decides:
skill: grilling
triggers:
should: ["stress-test my plan"]
should_not: ["fix this typo"]
Answer Claude yourself. Add interactive: true and skillbench asks you in the terminal whenever Claude is waiting for a reply. Press Enter on an empty line to finish.
Check for flakiness. Claude doesn't do exactly the same thing every time. --repeat 5 (or repeat: 5 in the test) runs it 5 times and shows how often each check passed. Add min_pass_rate: 0.8 to pass when 4 out of 5 runs pass.
Set defaults for every test in skillbench/skillbench.yaml:
model: claude-sonnet-5-5
timeout_s: 600 # seconds before a turn is stopped
fixture: .
repeat: 1
min_pass_rate: 1.0
A test's own settings win over these, and command-line flags win over both.
All test fields:
| Field | Meaning |
|---|---|
skill / input |
Skill tests: which skill, and what to send it |
prompt |
Agent tests: the first message |
answers |
Replies sent one per turn |
interactive |
Ask you in the terminal for replies |
triggers |
Prompts that should / shouldn't pick the skill |
fixture |
Which repo Claude works in |
assert |
The checks |
name |
A display name (defaults to the file name) |
timeout_s, repeat, min_pass_rate |
As above |
Good to know
- It costs real usage. Every run is a real Claude session. Start with skill tests; long agent tests take longer and cost more.
- Your personal setup is ignored. Only the skills and plugins in the repo being tested are loaded, so results don't depend on whose machine runs them.
- Only test repos you trust. Claude has full tool access inside the copy, and
command_succeedsruns commands there. command_succeedsuses your system shell (shon Mac/Linux,cmdon Windows), so keep commands simple if you share tests across systems.- Results are in
skillbench/runs/, which is git-ignored. Each run has arun.jsonand Claude's raw output.
| Status | Meaning |
|---|---|
pass |
Every check passed |
fail |
Claude finished, but a check failed |
error |
Claude crashed or timed out, or the skill or fixture couldn't be found |
Metadata
Release files for skillbench-cli 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| skillbench_cli-0.1.0.tar.gz | 45.8 kB | Without provenance | Details for skillbench_cli-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| skillbench_cli-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for skillbench_cli-0.1.0-py3-none-any.whl |
Total release size: 88.2 kB
Release files / skillbench_cli-0.1.0.tar.gz
| Download URL | skillbench_cli-0.1.0.tar.gz |
|---|---|
| Size | 45.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
56261128063cd08afc9219a42b07de49d6abf14b8e0598ffe35ae39f64297f27
|
|
BLAKE2b-256 checksum How to use checksums |
fe4640f7ee0a1e1f5c77f91d3c5ead18421fc0774cb577a6923f0624a0b946fa
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.13 {"installer":{"name":"uv","version":"0.11.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / skillbench_cli-0.1.0-py3-none-any.whl
| Download URL | skillbench_cli-0.1.0-py3-none-any.whl |
|---|---|
| Size | 42.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6ce6e274c8278dadc12c36277b290068aa4af03c65e3c5acbf35117d66e36069
|
|
BLAKE2b-256 checksum How to use checksums |
57c718f1446564d6381b40e8d3b76f5e76e78de211ced1fd9082ff4509251be9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.13 {"installer":{"name":"uv","version":"0.11.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for skillbench-cli 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.