🔬 skillprobe

The eval harness that tells you whether the right SKILL.md loads — and whether the model actually follows it.

skillprobe is an eval harness for agent skills — the SKILL.md folders of Claude Code and the Agent Skills format: a Python CLI that tests whether a skill triggers on the right requests, which skill wins when many compete for the same request, and whether the model follows the skill once it loads. LLM evaluation built for skills: a labeled golden set, repeated trials, 95% confidence intervals and a CI gate, on any model behind an OpenAI-compatible API.

Not "run it twice and eyeball the output." skillprobe runs a labeled golden set against any model, many times over, and turns agent-skill behavior into numbers you can put in a pull request: trigger precision and recall, pass rates with 95% confidence intervals, flaky cases, routing collisions between skills, cost per trial.

Golden set → repeated trials → confidence intervals → per-skill routing table → CI gate. Five model families. Cents per run on flash models.

Quick Start · Results · Routing across 28 public skills · How It Works · Features · FAQ · Roadmap

_{Real output of the bundled example, replayed from the response cache with --cache-only.}

💡 The Problem

A skill is a prompt with a trigger. Both halves fail quietly:

• The trigger. The model only sees the skill's name and description. A vague description means the skill never loads — or loads for requests it has nothing to do with.
• The body. The skill loads, and the model follows half of it. Differently on every run.

And in a real agent a skill never runs alone. It sits in a list next to a dozen others, and every description in that list competes for every request.

Reading a few outputs by eye catches none of this. skillprobe runs a fixed set of cases many times per model and reports numbers you can compare between versions of a skill, between neighbors in the same skill list, and between models.

🔥 Why it wins

📊 Results on the bundled example

examples/commit-message — a skill that enforces a team commit-message convention, plus a code-review skill offered as a distractor. 10 cases: 5 requests that must load the skill and follow it, 5 near-misses that must not load it. 3 repeats each, all models through OpenRouter.

Brackets are 95% Wilson intervals. *The run was stopped manually before the kimi-k3 baseline finished (15 of 30 trials). Full report: examples/commit-message/results/report.md.

What the numbers say:

• Triggering was the easy part here — every model loaded the skill exactly when it should. Following it is where they split.
• Without the skill every model scores 15/30 — only the five near-misses pass. None of the commit requests match the convention on general knowledge, so the gap is the skill's measured effect.
• Qwen follows the shape but not the details: it leaves an empty Refs: line when there is no ticket and writes config: instead of chore(config):.
• GLM and DeepSeek are flaky: in 5 and 4 of 10 cases the same prompt passed on one repeat and failed on another. A single run would have told a different story each time — that is why repeats are the default, not an option.

Ten cases is a small probe; the point is the method. Treat the ranking as specific to this skill.

🧭 Routing across 28 public skills

A real agent never sees one skill. It sees a list — and every description in that list competes for every request. So we took the skills people actually install and asked the question nobody measures: when 14 skills are on the table, does the model pick the right one?

Two probes, real skills, unchanged descriptions:

• examples/public-skills/anthropic — 14 skills from anthropics/skills, 25 labeled cases;
• examples/public-skills/superpowers — 14 skills from obra/superpowers, 21 labeled cases.

Each case says which skill must load and which must not (how the golden sets are labeled). routing_only stops every trial after the first turn, 3 repeats per case, all models through OpenRouter.

_{*kimi-k3 costs about 100x the others per routing trial, so it was run on the first probe only. Full reports: anthropic, superpowers.}

The headline number is high — and that is exactly why the misses are worth reading. They are not random. Every one of them traces back to how a description is written:

And the clean side of the ledger: all 13 Anthropic skills that had a positive case loaded every time they should, on all five models; 10 of them never loaded when they shouldn't. systematic-debugging went 9 for 9 on every model.

What it says:

• Routing between well-separated skills is close to solved — flash models get it right for one to three cents per 75 trials.
• The failures are written, not sampled. Descriptions that shout — "You MUST", "Stop and check this skill before finishing any reply", "any" — win requests that belong to a quieter neighbor. A description is a routing rule; write it like one.
• First-turn routing is the cheap test you can run in CI: label your requests once, run --fail-under, and a new skill that steals traffic from an old one fails the build.

Limits, plainly: one labeler, 46 cases, first turn only — a model that loads a second skill after reading the first one's body is not measured here. Disagree with a label? Change it in cases.yaml and rerun with --cache-only: the report rebuilds from cached responses for free.

⚙️ How It Works

flowchart LR
    A[📋 Golden set<br/>cases.yaml: prompt + expect] --> B[🧰 Skills offered<br/>name + description only]
    B --> C[🤖 Model turn<br/>call load_skill?]
    C -->|loads| D[📄 SKILL.md body<br/>returned as tool result]
    D --> E[💬 Final answer]
    C -->|no call| E
    E --> F[✅ Checks<br/>triggered · loads · not_loads<br/>contains · regex · json_schema]
    F --> G[📊 Report<br/>Wilson CI · flaky · routing · cost]

This mirrors how agent harnesses expose skills: the body stays out of context until the model asks for it. Whether it asks is exactly what the trigger and routing metrics measure. With routing_only: true the trial stops right after the model's first turn.

system: base prompt + list of skills (name: description only)
tools:  load_skill(name)
user:   case prompt + attached files
   |
   v
model calls load_skill?  --yes-->  tool result = SKILL.md body  -->  model answers
   |
   no --> model answers
   |
   v
checks: triggered? loads / not_loads? contains / not_contains / regex / json_schema / max_chars

✨ Features

🚀 Quick Start

Installed from PyPI with pipx install skilldx (the PyPI name: skillprobe there is an unrelated project), the command is skillprobe (or skilldx). To try the bundled examples, take the repo:

# 1) Install
git clone https://github.com/AndreySukhanov/skillprobe.git
cd skillprobe
pip install -e .

# 2) Add a key
export OPENROUTER_API_KEY=...      # or put it in a .env next to probe.yaml

# 3) Run the bundled example on one model
skillprobe run examples/commit-message/probe.yaml --repeats 1 --models qwen3.8-flash

No Python at hand? The same run from the ready image (amd64 and arm64), with your probe folder mounted as /work:

docker run --rm -v "$PWD:/work" -e OPENROUTER_API_KEY ghcr.io/andreysukhanov/skillprobe \
  run examples/commit-message/probe.yaml --repeats 1 --models qwen3.8-flash

Useful flags:

Every response is cached on disk (.skillprobe_cache/), keyed by the full request and the repeat number. Editing checks and re-running costs nothing. max_cost_usd in the config stops a run before it overspends.

🧪 Writing a Probe

probe.yaml:

skills:
  - skills/commit-message     # the skill under test
  - skills/code-review        # distractors are offered too
target: commit-message
cases: cases.yaml
repeats: 3
max_cost_usd: 2.0

models:
  - name: qwen3.8-flash
    model: qwen/qwen3.8-flash           # provider defaults to openrouter
  - name: deepseek-direct
    provider: deepseek                  # DEEPSEEK_API_KEY
    model: deepseek-chat

cases.yaml:

- id: ticket-from-branch
  prompt: I'm on branch PAY-482-invoice-pdf. What should I put in git commit for this?
  files: [fixtures/billing_invoice_pdf.diff]
  expect:
    triggered: true
    regex:
      - '\Afeat\(billing\): '
      - '^Refs: PAY-482\s*$'

- id: review-is-not-commit          # near-miss: a diff, but not a commit request
  prompt: Review this diff before I merge it. Anything dangerous?
  files: [fixtures/sql_injection.diff]
  expect:
    triggered: false
    contains: ["injection"]

Good probes are mostly near-misses. A skill that loads for "write a haiku" is rare; one that loads for "review this diff" when it is a commit-message skill is common.

- id: failing-test-is-debugging
  prompt: test_invoice_total started failing after yesterday's merge - expected 120.00, got 119.99.
  expect:
    loads: [systematic-debugging]                  # must be loaded
    not_loads: [brainstorming, writing-plans]      # must not be loaded; other skills are not scored

- id: plain-question
  prompt: What does HTTP 418 mean?
  expect:
    not_loads: "*"                                 # no skill at all

❓ FAQ

How do I test whether my Claude skill triggers?

Point probe.yaml at your skill folder and its neighbors, write cases.yaml with requests that must load the skill and near-misses that must not, and run skillprobe run probe.yaml. The report gives trigger precision and recall per model, and every failure with the model's output. See Writing a Probe.

What is a routing collision between skills?

Two skills loading for the same request, or the wrong one winning it. Label each case with loads / not_loads and the report adds a per-skill routing table — loaded when required, loaded when forbidden — plus a collisions table. On 28 public skills, "Add a function that validates IBANs" went to brainstorming in 12 of 12 trials instead of test-driven-development: see Routing across 28 public skills.

Which models and providers does it support?

Any model behind an OpenAI-compatible chat API with tool calling — skills load through a load_skill tool, the way agent harnesses expose them. Presets are built in for OpenRouter, OpenAI, DeepSeek, Qwen (DashScope), GLM (Z.ai), Kimi (Moonshot) and YandexGPT; a local vLLM or a corporate gateway is two lines of config.

Do I need an LLM judge?

No. Checks are deterministic — triggered, loads, not_loads, contains, not_contains, regex, json_schema, max_chars — so the same output always gets the same verdict. LLM-as-judge with calibration against human labels is on the roadmap.

How much does a run cost?

Cents on flash models: 30 trials of the bundled example cost $0.007-0.011, and 75 routing trials across 14 skills cost $0.011-0.026. max_cost_usd stops a run before it overspends, and every response is cached — rebuilding a report with --cache-only costs nothing.

Why run every case several times?

Because one run lies. In the bundled example GLM and DeepSeek passed and failed the same prompt on different repeats in 5 and 4 of 10 cases. Repeats are the default, pass rates come with 95% Wilson intervals, and flaky cases are flagged.

How do I measure what my skill actually adds?

Run with --baseline: the same cases with no skills offered. In the bundled example every model scores 15/30 without the skill and up to 30/30 with it — the gap is the skill's measured effect.

Can I run it in CI?

Yes. --fail-under 0.9 exits 1 when any model's pass rate drops below the threshold, so a new skill that steals traffic from an old one fails the build. A ready GitHub Action does it in one step:

In GitHub Actions, the same gate in one step (the key from your repository secrets; the report lands in the job summary):

- uses: AndreySukhanov/skillprobe@v0.3.0
  env:
    OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
  with:
    probe: evals/probe.yaml
    fail-under: "0.9"
    args: --repeats 3

A ready Docker image (ghcr.io/andreysukhanov/skillprobe) works in any other CI.

A skill is a routing rule plus a prompt — skillprobe measures both.

🗺 Roadmap

• Skill loading via tool call, deterministic checks, trigger precision/recall, repeats with intervals, flaky detection, cost and latency, response cache, budget cap
• Routing between many skills: loads / not_loads labels, routing_only first-turn mode, per-skill routing table and collisions, golden sets on public skills
• LLM-as-judge with rubrics, and pairwise comparison of two skill versions
• Judge calibration against human labels — a judge that disagrees with people is flagged, not trusted
• Judge panels from different model families, with disagreements reported instead of averaged
• skillprobe compare — what broke between skill v1 and v2, as a GitHub PR comment
• Adversarial pack — prompt injection through attached files, language switching

Vote with 👍 on issues, or open a new one.

🤝 Contributing

The most valuable contributions are small and self-contained:

• A golden set for a public skill collection — labeled by the rules in LABELS.md
• A provider preset for another OpenAI-compatible API
• A new deterministic check

See CONTRIBUTING.md. Open an issue first for bigger changes — happy to discuss the design.

pip install -e ".[dev]"
pytest          # scripted fake client - no network, no keys

⭐ Support the project

If this project saved you time or showed you something new — star it. It genuinely helps others discover it.

💜 Sponsor development

This is a self-funded project — every benchmark in this README was paid for out of pocket in model credits (the routing run alone cost about $2.40). A donation goes straight into credits for bigger golden sets and more model families.

USDT — TRC-20 (Tron network) TERj89Gjf4qsMW7xyKQWGWtNmBQaDUPnnH ⚠️ Send only USDT via the TRC-20 (Tron) network to this address. Funds sent through any other network will be lost.

Thank you — even a small contribution keeps the probes running. 🙏

📄 License

Source-Available — free to read, run and learn from. Commercial or derivative use requires written permission. See LICENSE.

Third-party skills under examples/public-skills/ keep their own licenses (Apache-2.0, MIT) — see SOURCES.md.

Licensing contact: avsukhanov21@gmail.com