skill-lens

An Agent Skill is a prompt. Prompts regress. skill-lens turns "I think this SKILL.md got better" into a score, a report, and an exit code your pipeline can gate on.

📖 Full documentation: https://emadmokhtar.github.io/skill-evaluator/

Why

You edit a SKILL.md, read the new answer once, and it looks better. Two weeks later a teammate edits the same file, and nobody can say whether the agent still looks an order up before refunding it — or whether it now refunds orders it should refuse.

skill-lens gives that question a real answer. You write eval cases next to your skill. It runs them, scores what came back, and reports one verdict for the whole run.

Your skills stay yours. Skills and their eval cases are inputs to the tool — nothing about a skill under test is vendored here, so any skill repository can adopt skill-lens without embedding it.

What it measures

• What the agent said — substring, regex and exact-match assertions on the output.
• What the agent did — which mock tools it called, in what order, and which ones it must never touch. A refund granted without a lookup is invisible to an output assertion.
• What it cost — per-case ceilings on tokens, dollars and latency.
• How well it said it — a rubric-based LLM judge (large language model grading the output) returns one verdict per rubric line, with the evidence for each. "Explains it plainly" is not a substring.
• Whether the agent reached for the skill at all — mode: offered registers the skill as a tool instead of force-loading it, so triggering becomes an observable choice. Ship the negative control and a skill that fires on everything stops scoring 100%.
• Whether your edit actually helped — run every case twice, once with the skill and once against a baseline (no skill, or its previous version resolved from git), and gate on the delta.

What an eval looks like

# an eval file, beside the SKILL.md it tests
cases:
  - name: refuses a refund outside the return window
    task: I want a refund for order 1234
    tools:
      - name: lookup_order
        description: Look up an order by its id
        parameters:
          order_id: string
        returns: '{"id": "1234", "delivered_days_ago": 61}'
      - name: issue_refund
        description: Issue a refund for an order
        parameters:
          order_id: string
        returns: '{"ok": true}'
    trajectory:
      called: [lookup_order]      # it must look the order up
      forbidden: [issue_refund]   # and must not refund this one
    assertions:
      - kind: contains
        value: "30-day"           # and cite the policy

uv tool install "skill-lens[pydantic-ai]"   # drop the extra for the offline runner alone
skill-lens run ./skills                     # exit 0 passed, 1 failed, 2 your setup is wrong

That first run is free: the default runner is scripted and offline, so nothing is sent anywhere and no API key is read. --runner pydantic-ai, --runner copilot or --runner claude-code scores a real agent instead — see Runners.

Start at Getting started, which takes one skill from nothing to a CI gate.

Documentation

Contributing

Contributions are welcome, and the project is set up so that helping is cheap:

uv sync
uv run pytest        # the whole suite: offline, no API key, no spend
uv run ruff check .

Every test passes with no network access. Tests that would hit a real provider are opt-in (-m integration) or replay recorded traffic, so you can work on any part of this without an API key or a bill.

Three conventions to know before your first pull request — all three are explained in Contributing:

1. Test-driven. Write the failing test first.
2. Conventional Commits for commit messages and pull-request titles. Releases are derived from history, and pull requests are squash-merged, so the title becomes the commit.
3. Documentation ships with the change, not as a follow-up. Continuous integration checks it.

Good places to start: a new example skill with its eval suite, an adapter for another agent framework, or anything on the roadmap. Not sure whether an idea fits? Open an issue and ask — that is a perfectly good first contribution.

Status

Discovery, scoring, judging, comparison, real-file workspaces, reporting, gating and the automated release pipeline all ship and are tested. Versions are derived from the commit history and published to PyPI on merge. This is 0.x: a minor release may still change behaviour, so pin what you depend on. See the roadmap for what is shipped and what is planned.

License

MIT — see LICENSE.