pytest-skillcheck

Test agent skills against real coding agents. Run a prompt, assert on what actually happened, and grade the rest with an LLM.

A skill is a folder of instructions you hand to a coding agent. Nothing checks that the agent still follows them after you edit the wording. skillcheck runs the real CLI against a throwaway workspace and tells you.

#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = ["pytest-skillcheck"]
# ///
"""Tests for the git-graveyard skill."""

from skillcheck import main


def test_buries_a_public_repo_in_the_public_graveyard(run_skill):
    result = run_skill(
        "Bury ./deadproj",
        fake={"gh": {"wiltaylor/deadproj": {"visibility": "PUBLIC"}}},
        answers=["Yes, that graveyard is right. Go ahead."],
    )

    assert result.called("gh", "repo", "view", "wiltaylor/deadproj")
    assert not result.called("gh", "repo", "delete", "wiltaylor/deadproj")


if __name__ == "__main__":
    raise SystemExit(main(__file__))

Save that as skills/git-graveyard/test.py, chmod +x, and run it:

./skills/git-graveyard/test.py

uv fetches pytest-skillcheck, pytest runs, and the skill under test is the one the file lives in — no skill= argument, no conftest.py.

Contents

• Install
• Usage
• Contributing
• Harnesses
• Settings
• Writing tests
• Linting
• Recorded runs
• Containers
• Results
• Commands
• Cost, and what CI can cover
• Adding a harness
• Licence

Install

uv add pytest-skillcheck          # or: pip install pytest-skillcheck
skillcheck doctor          # which agent CLIs are installed, and what they would run

skillcheck is a pytest plugin, so its fixtures are available anywhere pytest runs once it is installed.

Usage

Scaffold a test beside a skill, then run it:

skillcheck new-test skills/my-skill    # writes skills/my-skill/test.py
chmod +x skills/my-skill/test.py
./skills/my-skill/test.py

The file knows which skill it covers from where it sits, so nothing tells it twice. Each run appends to results.json beside it, and skillcheck status skills/ reads those back.

Contributing

Pull requests are welcome, new harnesses most of all. Run what CI runs before you open one:

uv run ruff check src tests
uv run ruff format --check src tests
uv run pytest tests -q

That suite runs against recorded CLI output and a stub agent, so it needs no credentials and costs nothing. It cannot run a real agent, so anything you change in a harness needs a skill test run by hand as well.

Harnesses

By default skillcheck uses the first of claude, codex, opencode, droid that is installed. Override with SKILLCHECK_HARNESS:

SKILLCHECK_HARNESS=codex ./skills/my-skill/test.py
SKILLCHECK_HARNESS=claude,opencode ./skills/my-skill/test.py
SKILLCHECK_HARNESS=all ./skills/my-skill/test.py

A test can restrict itself with @pytest.mark.harness("claude", "opencode"), and a harness whose CLI is missing skips rather than fails.

Isolation. An agent reads globally installed skills from your home directory, and those shadow the copy under test — so without isolation you would be grading whatever you last deployed. claude and codex are pointed at a throwaway config directory with credentials symlinked in. opencode is not: its auth spans several directories with no single file to link. droid is not either: it takes FACTORY_HOME_OVERRIDE, but its credentials do not survive being moved there. Assert not result.reached_home() to catch this, or run in a container to prevent it.

Settings

Environment first, [tool.skillcheck] in pyproject.toml second, defaults last.

Writing tests

Assertions

run_skill returns a RunResult.

What the workspace looks like now.

result.exists("out.txt")     result.read("out.txt")     result.files()

What the run changed, against a snapshot taken before the agent started. This is how a test says stay out of there, which "the file exists" cannot.

result.created()             # ["docs/plan.md"]
result.modified()            # paths whose contents differ
result.deleted()             # paths that are gone
result.touched()             # all three, sorted
result.untouched("src")      # nothing under src/ changed
result.untouched()           # nothing at all changed

What the agent did.

result.ran(r"cargo build")            result.tool_count("Read")
result.used_skill("name")             result.loaded_skill()      # tool call or SKILL.md read
result.used_agent("reviewer")         result.delegated()
result.read_files()                   # workspace-relative paths, in order
result.ordered("repo view", "repo delete")   # these calls, in this order
result.called("gh", "repo", "view")   result.refusals("gh")      result.calls("gh")
result.acted_before_asking(r"--force")       result.reached_home()

What the agent said.

result.output                result.handbacks           result.questions
result.asked_question()      # it asked something
result.asked("which graveyard")   # it asked about the right thing
result.said(r"dry run")      # anywhere in what it said back, question or not

The git repo it left behind.

result.commits()             # newest first, each with .sha .subject .body
result.committed(r"^feat:")
result.branch()

What it cost.

result.cost_usd              # None on a CLI that reports no price
result.tokens                result.input_tokens        result.output_tokens
result.duration_s            result.turns

One turn at a time. Turn 0 is everything the agent did before the user said anything, so a match there is work done unasked.

result.turn(0).tool_uses     result.turn(1).created()   result.turn(-1).output

When an assertion fails, the transcript, the files that changed, and anything a fake refused are printed under the failure. You should not have to re-run it by hand to see what happened.

Skills that ask questions

answers replies as the agent hands control back, in any of three forms.

A list answers each hand-back in order. Answers are sent whenever a turn ends, not only when the message looks like a question: agents ask without question marks ("Confirm that I should proceed"), and gating on punctuation strands the test.

result = run_skill("Merge these repos", answers=["Yes, go ahead.", "No, keep the originals."])

A mapping answers whichever question its pattern matches, so the test survives the skill reordering what it asks. Each pattern is used once, and a question no pattern matches ends the conversation.

result = run_skill(
    "Bury ./deadproj",
    answers={r"which graveyard": "the public one", r"delete.*remote": "no, keep it"},
)

A brief hands the conversation to a model playing the user, which answers whatever the agent actually asks. Reach for it when you do not want to guess the wording at all — it costs one small model call per turn.

result = run_skill(
    "Bury ./deadproj",
    user="You own deadproj, you want it in the public graveyard, and you are in a hurry.",
)

Then assert on what was asked:

assert result.asked("which graveyard")
assert not result.acted_before_asking(r"--allow-unrelated-histories")
assert len(result.questions) == 1, "the skill asked more than once"

asked needs the wording to hold still. When it should not have to, judge the questions instead — see judging.

Does the skill fire at all?

A description that no agent matches is the most common way a skill fails, and it fails silently. Prompt naturally, without naming the skill, and check it was picked up:

def test_it_fires_on_its_own(run_skill):
    result = run_skill("This old repo is dead, get it out of my way.")
    assert result.loaded_skill()


def test_it_stays_out_of_unrelated_work(run_skill):
    result = run_skill("Rename the variable `tmp` in main.py.")
    assert not result.loaded_skill()

loaded_skill() counts both ways a skill is picked up: a tool call on the CLIs that have one, and reading SKILL.md on the CLIs that do not.

Runs that are not the same twice

Agents are not deterministic, so one green run says less than it looks like. samples= runs the same prompt several times and returns a RunSet:

def test_it_usually_asks_first(run_skill):
    runs = run_skill("Bury ./deadproj", samples=5)

    assert runs.rate(lambda run: run.asked("which graveyard")) >= 0.8
    assert runs.every(lambda run: not run.ran(r"repo delete"))

rate, every, some, most, first, cost_usd, tokens, and explain(predicate) for a line per sample when the assertion fails. Each sample gets its own workspace.

Skills that call out to services

fake puts a stub binary first on PATH, backed by a fixture. The stub answers what the fixture describes and refuses everything else loudly, so a skill reaching for an unanticipated command fails the test rather than doing something real.

result = run_skill("Archive it", fake={"gh": {"me/proj": {"visibility": "PRIVATE"}}})

assert not result.refusals("gh")     # nothing unexpected was attempted

skillcheck ships purpose-built fakes for gh and curl. Any other command is faked by describing it, no Python needed: patterns are matched against the arguments as one string, in the order written.

result = run_skill(
    "Roll out the new image",
    fake={
        "kubectl": {
            "get pods": "NAME   READY\nweb-0  1/1",
            "apply -f .*": {"stdout": "deployment configured", "exit": 0},
            "delete .*": {"stderr": "forbidden", "exit": 1},
        },
        "docker": {"build .*": "built", "push .*": "pushed"},
    },
)

assert result.called("kubectl", "apply", "-f", "deploy.yaml")

A response may be a string (its stdout), an object with stdout, stderr, exit, and writes, or a list — each call taking the next entry, for a command that reports something different once the work has finished. curl is keyed by URL instead, and honours -o, -w '%{http_code}', and -f:

fake={"curl": {"https://api.example.com/.*": {"body": '{"ok": true}', "status": 200}}}

Pass script= to fakebin.install for a fake of your own.

Four things stand between a test and real infrastructure, and the run only starts once all of them hold:

1. The stub goes first on PATH, and skillcheck runs gh --skillcheck-fake itself.
2. The agent then runs the same check in a throwaway session of its own. Checking skillcheck's environment proves nothing about the shell the agent's tools run in, which can rebuild PATH from a profile.
3. GIT_CONFIG_GLOBAL rewrites every github.com and gitlab.com URL to a path that does not exist, so git cannot reach a forge either. It also carries an identity, because without one commits fail and the agent starts improvising with your real name and email.
4. Anything the fake does not recognise exits non-zero.

Nothing patches Python's subprocess: the agent spawns tools from its own shell in its own process, where Python-level mocking cannot reach.

Judging

Some things no assertion catches — whether the skill explained itself, whether it asked the right question. Write a rubric:

verdict = judge("The reply states the code and hedges nothing", result)
assert verdict, verdict.reasoning

Write rubrics as requirements, including any exception the skill itself allows: a rubric that overstates a rule fails compliant work.

Show the judge only the part the rubric is about. A rubric about the questions graded against a wall of tool calls is a rubric being diluted.

judge("It offered both options", result, scope=("prompt", "questions"))
judge.files("The plan names every repo being merged", result)

Sections are prompt, questions, tools, files, and answer.

Did it ask, in substance? result.asked(pattern) needs the wording to hold still. This does not:

verdict = judge.asked("which graveyard the repo should go to", result)
assert verdict, verdict.reasoning

Which wording did better? Run the same prompt against two versions of a skill and compare them, for the edit-and-see-what-happens loop:

before = run_agent("Bury ./deadproj", skill="skills/git-graveyard")
after = run_agent("Bury ./deadproj", skill="drafts/git-graveyard")

assert judge.compare("Asks before deleting anything", before, after).winner == "b"

Across samples, judge.rate(rubric, runs) grades each and returns the fraction that passed.

Every verdict is recorded in results.json with the reasoning that produced it, which is what you want when reviewing a regression six weeks later.

Linting

The cheapest test in the suite: no model, no credentials, no cost, no flakiness. It catches a description an agent will never match, a name that disagrees with its directory, and a link to a reference file that was renamed.

skillcheck lint skills/            # exits 1 on any error
skillcheck lint skills/ --strict   # warnings count as errors too
skillcheck lint skills/ --ignore SK011

This is the part of the suite that belongs in CI unconditionally.

Recorded runs

A recorded run holds what the CLI printed and what the workspace looked like afterwards. Replaying rebuilds the RunResult from it, so every assertion runs again for nothing — the model call was the only part that cost anything.

skillcheck test skills/ --replay=record   # run for real, and keep the recording
skillcheck test skills/ --replay=replay   # never call a model; skip what was not recorded
skillcheck test skills/ --replay=auto     # replay when there is a recording, run when there is not

A recording is keyed by everything that could change the answer: the harness, the model, the prompt, the skill's own files, and whatever the test put in the workspace. Edit the skill and the key changes, so a stale recording is never replayed against new wording — it skips instead, which is the honest answer.

This is what makes a real suite runnable in CI: record locally, commit the cassettes, and let CI replay them. Note what it does not check — replay proves your assertions still hold against the run you recorded, not that today's model still behaves that way. Re-record when the skill changes, on a schedule, or both.

skillcheck cassettes            # what has been recorded
skillcheck cassettes --clean    # throw it all away

Runs answered by a simulated user= brief are never recorded: there is no fixed conversation to record.

Containers

reached_home() tells you afterwards that a run touched the real machine. An image stops it happening:

skillcheck test skills/ --container=ghcr.io/example/agents:1

The container gets the workspace, its throwaway config directory, and a read-only home for credentials. The image supplies the agent CLI; skillcheck only decides what runs inside it.

Results

Each run records beside the test file that produced it, so a skill carries the record of what it was tested on:

skills/git-graveyard/
├── SKILL.md
├── test.py
└── results.json

That makes results reviewable. Someone who runs your skill against a harness you do not have can send the result back as a pull request, and a bug report can point at the row that failed.

skillcheck status skills/

         claude                     codex
SKILL    STATUS  LAST TESTED  TIME  STATUS  LAST TESTED  TIME
-------  ------  -----------  ----  ------  -----------  ----
justfile pass    today        53s   pass    2 days ago   115s

A failure is not automatically a regression: an overloaded API, a timeout, or a model taking a different-but-valid route all show up as red. Re-run the one test before you change the skill.

Commands

skillcheck test skills/my-skill     # run a skill's tests
skillcheck lint skills/             # check skills without running a model
skillcheck status skills/           # what was tested, and when
skillcheck report skills/           # pass rate, time, and cost per skill
skillcheck cassettes                # recorded runs, and --clean to drop them
skillcheck doctor                   # installed harnesses and settings
skillcheck new-test skills/my-skill # scaffold test.py

report is the one that decides what a suite is worth running:

SKILL          HARNESS  TESTS  PASSED  RATE  TIME  COST   TOKENS  LAST RUN
-------------  -------  -----  ------  ----  ----  -----  ------  --------
git-graveyard  claude   6      5/6     83%   412s  $1.20  840k    today
git-graveyard  droid    6      6/6     100%  380s  -      910k    2 days ago

Cost, and what CI can cover

Every live test is at least one model call. Run the tests for the skill you changed, not the whole tree, and put the free checks first:

1. skillcheck lint — no model, runs on every commit.
2. --replay=replay over committed cassettes — no model, re-runs every assertion.
3. Live runs — by hand, on a schedule, or when the skill changes.

skillcheck's own CI covers parsing, the assertion surface, the fakes, the linter, and the plugin end to end against a stub CLI. It cannot run real agents, so live behaviour is checked by hand. Agent CLIs change their flags without notice, and that is where breakage comes from.

Adding a harness

Subclass Harness, implement command, parse, and ideally resume_command, session_id, and usage, then register it in HARNESSES. Pull requests welcome — with a recorded sample of the CLI's output in tests/test_parsing.py, so the day it changes its format, that is what breaks first.

Licence

MIT.