Skill Lab

Agent Skills Evaluation Framework

Your agent's skills are probably broken in at least one way — and you don't know it yet. Skill Lab catches skills that drain tokens, never fire, or leak data before they cause damage.

pip install skill-lab

Why Skill Lab

Performance — A badly-written skill can triple your token usage with zero gain. We score every skill 0–100 and show exactly what it costs. sklab evaluate ./my-skill

Security — A malicious skill can exfiltrate company data to an external endpoint. Static checks catch that before the conversation starts. sklab scan ./my-skill

Trigger Testing — If your description doesn't have enough trigger examples, the skill sits there doing nothing. We generate and run ~13 tests automatically. sklab trigger ./my-skill

Quick Start

# Install
pip install skill-lab

# First run — scans your repo and shows the getting started guide
sklab

Commands

What Gets Checked

37 checks across 5 dimensions. Run sklab list-checks to browse all of them with severity labels.

Structure (13)

• SKILL.md Exists · Valid Frontmatter · Standard Frontmatter Fields
• Allowed Tools Format · Compatibility Length · License Format · Metadata Format
• Scripts Folder Valid · Scripts Self-Contained · Scripts No Interactive Input · Scripts Help Support
• References Folder Valid · Files Outside Spec Dirs

Naming (3)

• Name Required · Name Format (kebab-case) · Name Matches Directory

Description (3)

• Description Required · Description Not Empty · Description Max Length

Content (13)

• Body Not Empty · Has Examples · Description Actionable · Line Budget · Token Budget
• Metadata Token Budget · Reference Depth · Asset Paths Exist · Script Paths Exist
• Scripts Referenced · Compatibility Prerequisites · Broken Internal Links · Orphaned Files

Security (5)

• Prompt Injection & Jailbreak · Evaluator Manipulation · Unicode Obfuscation · YAML Anomalies · Suspicious Size & Structure

Trigger Testing

Skill Lab generates ~13 test cases per skill across 4 types — explicit, implicit, contextual, and negative — then runs them against a live LLM via Claude CLI.

Requires Claude CLI: npm install -g @anthropic-ai/claude-code

# .sklab/tests/triggers.yaml
skill: my-skill
test_cases:
  # should fire
  - id: explicit-1
    type: explicit
    prompt: "$my-skill do the thing"
    expected: trigger
  # should NOT fire
  - id: negative-1
    type: negative
    prompt: "unrelated question"
    expected: no_trigger

Telemetry

sklab collects anonymous usage data (command names, duration, exit codes, scores, token counts). No skill content, file paths, or flag values are ever collected. To opt out:

sklab telemetry disable

See docs/PRIVACY.md for the full privacy policy.

Development

pip install -e ".[dev]"
pytest tests/ -v
mypy src/
ruff check src/
ruff format src/

Apache License 2.0