skillbench
Test your Claude Code skills and agent workflows, the way you test code.
You write a short YAML file: "send Claude this, then check these things". skillbench runs Claude Code for real on a copy of a repo, records everything it did, and tells you pass or fail.
Jump to: Get started · Write a test · Run tests · Reference
Get started
You need Python 3.11+, git, and Claude Code installed and logged in (claude --version should work).
1. Install it. Use whichever tool you normally use:
pip install skillbench-cli
pipx install skillbench-cli
uv tool install skillbench-cli
The package is skillbench-cli; the command is skillbench.
2. Set up your repo. In the repo you want to test:
skillbench init
This creates one folder, skillbench/, with two example tests:
skillbench/
skillbench.yaml settings for every test
skills/ skill tests
agent/ agent tests
fixtures/ small sample repos for tests to run on
runs/ results (git-ignored)
3. Run the examples.
skillbench skill
skillbench agent
4. Look at the results. Open skillbench/runs/index.html in your browser. The list on the left groups runs by what they test (a skill, a plugin, or this repo), newest first. Expand a group to see its tests. Click a test to see its run.
Now write your own tests: next section.
Write a test
Find what you want to check in this list, then follow that recipe.
| I want to check that… | Recipe |
|---|---|
| my skill does the right thing when it runs | A. Test a skill |
| Claude picks my skill when it should, and not when it shouldn't | B. Test a skill gets picked |
| a normal request gets done, with the right skills | C. Test a request end to end |
| my multi-step workflow works on a real repo | D. Test a workflow on a repo |
| the plugin I'm building works on other projects | E. Test a plugin you're building |
Or let Claude write it. Install the /eval-creator skill once:
skillbench install-skill
Then in Claude Code, in your repo, say what you want to test:
/eval-creator test that my grilling skill asks questions before writing anything
It reads your skills, then proposes the tests in plain words: what each one runs, what it checks, when it passes, and a rough cost. Nothing is written until you reply approve. Ask for changes and it proposes again. Once approved, it writes the files and checks them. It never runs them, so you decide when to spend tokens.
Run skillbench install-skill again after you upgrade skillbench.
Whichever way you write tests, check them for free before running:
skillbench check
A. Test a skill
Runs your skill directly (/grilling an idea for a todo app) and checks what it did.
Create skillbench/skills/grilling.yaml:
description: Grilling asks questions before it writes anything.
skill: grilling # the skill's name
input: an idea for a todo app # the text after /grilling
fixture: . # run on a copy of this repo
assert:
- final_text_contains: "?"
- tool_not_called: Write
Run it: skillbench skill grilling
skillbench finds the skill wherever it is: .claude/skills/, skills/, .github/skills/, or inside a plugin.
B. Test a skill gets picked
Checks your skill's description is good enough for Claude to choose it on its own. Each prompt is a short run that stops as soon as Claude decides, so these are cheap.
Add triggers: to a skill test:
skill: grilling
triggers:
should: ["stress-test my plan before I build it"]
should_not: ["fix the typo in README.md"]
Run it: skillbench skill grilling
C. Test a request end to end
Sends a normal message, with nothing forced. Claude sees every skill in the repo and decides for itself; the test checks what it chose and whether the job got done.
Create skillbench/agent/csv-export.yaml:
description: Adding CSV export uses the planner and leaves the tests passing.
prompt: add CSV export to the report page
fixture: .
answers: # sent one at a time, each time Claude stops and waits
- Yes to all your recommendations.
- approved
assert:
- skill_invoked: planner
- command_succeeds: python -m pytest -q
Run it: skillbench agent csv-export
Not sure what Claude will ask? Leave out answers: and add interactive: true. skillbench then asks you in the terminal each time Claude waits for a reply (press Enter on an empty line to stop). Copy your replies into answers: afterwards.
D. Test a workflow on a repo
For workflows with several phases, where in Claude Code you'd type things between messages: commit, /clear, start the next skill. Put those in answers: exactly as you'd type them:
In answers: |
What happens |
|---|---|
approved |
Sent to Claude as a message |
/clear |
The next message starts a fresh session |
!git add -A && git commit -m "plan" |
Runs in the repo as a shell command, not sent to Claude |
Point fixture: at the repo to run on. A git URL is cloned fresh for each run, and pushing from the clone is turned off.
Create skillbench/agent/full-loop.yaml:
description: The plan, build and review loop adds a dark mode toggle to test-app.
prompt: /my-plugin:plan add a dark mode toggle. Use the slug dark-mode.
fixture: https://github.com/you/test-app.git
answers:
- Yes to all your recommendations.
- approved
- '!git add -A && git commit -m "plan: dark-mode"'
- /clear
- /my-plugin:build dark-mode
- '!git add -A && git commit -m "dark-mode: build"'
- /clear
- /my-plugin:final-review dark-mode
assert:
- subagent_invoked: reviewer
- file_contains: {path: docs/features/dark-mode/plan.md, text: "approved"}
- command_succeeds: npm test
Run it on a cheap model first: skillbench agent full-loop --model claude-haiku-5-5
If a ! command fails, the run stops there with error, since later steps depend on it.
E. Test a plugin you're building
If your repo is a Claude Code plugin (it has .claude-plugin/plugin.json), add this line to skillbench/skillbench.yaml:
plugin: .
Now your plugin's skills and subagents are loaded into every test, whatever repo the test runs on. Combine it with recipe D to run your workflow on any project. Each run gets its own copy of the plugin, so the dashboard shows exactly which version was tested. skillbench check warns you if you forget this line.
Run tests
skillbench skill # every skill test
skillbench agent # every agent test
skillbench agent full-loop # one test, by name (or every test for one skill)
skillbench agent --model claude-haiku-5-5
skillbench agent --repeat 5 # run each test 5 times
The terminal shows one line per test:
✓ hello-greets-by-name passed 12s $0.03
✗ full-loop error 41s $0.02
stopped at step 7: '!git commit -m "dark-mode: build"' exited with code 1
1 passed, 1 failed · 53s · $0.05
details: skillbench/runs/index.html
| Result | Meaning |
|---|---|
pass |
Every check passed |
fail |
Claude finished, but a check failed |
error |
The run stopped: Claude crashed or timed out, a ! command failed, or the skill or repo couldn't be found |
For the details, open skillbench/runs/index.html. Each run page starts with the answer: whether it passed, and if not, the step it stopped at or the checks that failed. Below that are the steps in plain words, the checks as sentences, Claude's last message and the numbers. The full timeline of messages and tool calls, and the raw output, are collapsed at the end.
Keep costs down. Every run is a real Claude session. Use --model claude-haiku-5-5 while you're getting a test right, start with skill tests, and run long workflows only when you need to.
Flaky results? Claude doesn't do exactly the same thing every time. --repeat 5 shows how often each check passed. Add min_pass_rate: 0.8 to a test to pass it when 4 of 5 runs pass.
Reference
Checks (assert:)
A test passes when every check passes.
| Check | Passes when |
|---|---|
skill_invoked: grilling |
Claude used that skill |
skill_not_invoked: grilling |
Claude didn't use that skill |
subagent_invoked: reviewer |
Claude handed work to that subagent |
tool_order: [Read, Edit] |
Claude used these tools in this order (others in between are fine) |
tool_not_called: Write |
Claude never used that tool |
final_text_contains: "Done" |
Claude's last message contains the text |
file_contains: {path: notes.md, text: "approved"} |
The file exists afterwards and contains the text |
command_succeeds: npm test |
The command succeeds in the repo afterwards |
Repos (fixture:)
Claude always works on a copy, so your real files are never touched.
fixture: |
Claude works in |
|---|---|
| (left out) | An empty folder |
. |
A copy of your repo, including uncommitted work |
some/folder |
A copy of that folder (paths are from your repo root) |
https://github.com/you/app.git |
A fresh clone |
{git: https://github.com/you/app.git, ref: v1.2} |
A clone at that branch, tag or commit |
The copy is kept after the run so you can look inside. Its path is on the run's dashboard page.
Test fields
| Field | Meaning |
|---|---|
description |
One sentence saying what the test checks, shown on the dashboard |
skill, input |
Skill tests: which skill, and the text sent with it |
triggers |
Skill tests: prompts that should and should_not pick the skill |
prompt |
Agent tests: the first message |
answers |
Replies, one per turn; also /clear and !command |
interactive |
true to answer in the terminal once answers run out |
fixture |
The repo Claude works in |
assert |
The checks |
name |
Display name (defaults to the file name) |
timeout_s |
Seconds before one turn is stopped |
repeat, min_pass_rate |
Run it N times; the share of runs that must pass |
Settings for every test (skillbench/skillbench.yaml)
plugin: . # load this repo's plugin into every test
model: claude-haiku-5-5
fixture: .
timeout_s: 600
repeat: 1
min_pass_rate: 1.0
A test's own fields win over these, and --model / --repeat win over both.
Commands
| Command | Does |
|---|---|
skillbench init |
Creates skillbench/ with example tests |
skillbench skill [name] |
Runs skill tests |
skillbench agent [name] |
Runs agent tests |
skillbench check |
Checks every test file, without running anything |
skillbench install-skill |
Adds /eval-creator to Claude Code |
Good to know
- Your personal Claude setup is ignored. Only the skills and plugins in the repo being tested (plus
plugin:) are loaded, so results don't depend on whose machine runs them. - Only test repos you trust. Claude has full tool access inside the copy, and
command_succeedsand!commands run there. - Commands use your system shell:
shon Mac/Linux,cmdon Windows. Keep them simple if tests are shared across systems. - Each run is saved in
skillbench/runs/, with arun.jsonand Claude's raw output. - Upgrading:
pip install -U skillbench-cli(or your tool's upgrade command), thenskillbench install-skill.
Releasing a new version (maintainers)
From an up-to-date main with no uncommitted changes:
scripts/release.sh 0.2.0 --dry-run # bump, lint, build and test the install, then undo
scripts/release.sh 0.2.0 # the same, then commit, tag, publish to PyPI and push
Publishing needs a PyPI API token: export UV_PUBLISH_TOKEN=pypi-... first, or uv asks for it.
Metadata
Release files for skillbench-cli 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| skillbench_cli-0.2.0.tar.gz | 59.6 kB | Without provenance | Details for skillbench_cli-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| skillbench_cli-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for skillbench_cli-0.2.0-py3-none-any.whl |
Total release size: 115.7 kB
Release files / skillbench_cli-0.2.0.tar.gz
| Download URL | skillbench_cli-0.2.0.tar.gz |
|---|---|
| Size | 59.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6be5242966aa345e6d8a508b2ff8daa5cdf0f44648668486780583f991923023
|
|
BLAKE2b-256 checksum How to use checksums |
88b618d20afd2df9bc36584c504ae47ae47a2779edffda819053807517e9c0d7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.13 {"installer":{"name":"uv","version":"0.11.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / skillbench_cli-0.2.0-py3-none-any.whl
| Download URL | skillbench_cli-0.2.0-py3-none-any.whl |
|---|---|
| Size | 56.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
545c13c469395b4eee0c1da77820282dc05e57fae3547f36e39c64f7a12b97c0
|
|
BLAKE2b-256 checksum How to use checksums |
319cf88d1f7ec5aaf79f558530dff62f1bb0588007c55dff76ff41245ccc4261
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.13 {"installer":{"name":"uv","version":"0.11.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for skillbench-cli 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.