SkillSeal

Test your Agent Skills before your agents do.

SKILL.md files can be syntactically valid and still be bad: a vague description that never routes correctly, an oversized file that eats context, a curl | sh buried in a code block, a hardcoded /Users/you/... path that only works on your machine. None of that shows up until an agent picks the wrong skill, or picks the right one and runs something it shouldn't.

SkillSeal is a local-first, offline-first CLI that lints, scores, and routing-tests SKILL.md files, so you catch that before an agent does. It's deliberately scoped to what's useful today: static linting across four categories, deterministic (LLM-optional) routing tests, and CI-friendly exit codes and JSON output. No dashboard, no registry, no cloud — see Roadmap for what's intentionally not here yet.

Installation

Requires Python 3.12+. Install from PyPI with uv, pipx, or pip:

uv tool install skillseal
# or: pipx install skillseal
# or: pip install skillseal

No install at all, one-off run:

uvx skillseal check ./skills

For local development, clone the repo instead:

git clone https://github.com/pespinel/skillseal
cd skillseal
uv sync

Quickstart

skillseal check examples
skillseal test examples
skillseal conflicts examples/conflicting-skills

(From a repo clone without installing, prefix these with uv run.)

All three commands accept a path to a single SKILL.md file, a single skill directory, or a directory containing many skills (searched recursively).

Commands

skillseal check <path>

Runs every rule (SPECIFICATION, QUALITY, SECURITY, PORTABILITY) against each discovered skill and prints a per-skill report with a 0-100 score.

skillseal test <path>

Runs the routing test cases declared in each skill's skillseal.yaml against a RoutingEvaluator, and reports accuracy against should_trigger and should_not_trigger prompts. Skills without a skillseal.yaml are skipped, not failed.

skillseal conflicts <path>

Scans every skill under path together rather than one at a time, and flags two things check/test can't see in isolation:

• Duplicate names — two skills declaring the same frontmatter name (usually a copy-paste leftover).
• Routing overlap — two skills whose vocabulary (name + description + keywords:) is similar enough that an agent likely can't reliably tell them apart, using the same term-matching HeuristicRoutingEvaluator uses for routing tests, compared pairwise via Jaccard similarity.

$ uv run skillseal conflicts examples/conflicting-skills

Duplicate names

✗ "alpha-reviewer" used by 2 skills:
  - examples/conflicting-skills/alpha-reviewer/SKILL.md
  - examples/conflicting-skills/alpha-reviewer-2/SKILL.md

Routing overlap

✗ "alpha-reviewer" and "beta-reviewer"
  examples/conflicting-skills/alpha-reviewer/SKILL.md
  examples/conflicting-skills/beta-reviewer/SKILL.md
  Similarity: 53% (threshold: 50%)
  Shared terms: bug, code, potential, quality, review, reviewer, skill, style

Exit codes (all commands)

A typo'd path can never silently report success: exit 2 is reserved for "SkillSeal couldn't even run the check," distinct from "the check ran and found problems" (exit 1).

Example output

$ uv run skillseal check examples/bad-skill

examples/bad-skill/SKILL.md

Specification  WARN
Quality        WARN
Security       FAIL
Portability    WARN

Issues

WARN  name-directory-mismatch
      Frontmatter 'name' does not match the skill's directory name.
      name: 'helper', directory: 'bad-skill'

WARN  description-too-vague
      Description may not provide enough information for reliable routing.
      matched vague phrase: "helps with tasks"

FAIL  rm-rf
      Potential risk: recursive force-delete command found in a code block.
      1 occurrence(s), e.g. "rm -rf"

FAIL  pipe-to-shell
      Potential risk: downloads remote content and pipes it directly into a shell.
      1 occurrence(s), e.g. "curl https://example.com/install.sh | sh"

WARN  absolute-path
      Skill assumes absolute filesystem paths, which won't exist on other machines.
      /Users/someone/projects/output, /Users/someone/projects/output/tmp

  ... (more findings omitted for brevity — run it yourself to see the rest)

SkillSeal Score: 68/100

Specification   90
Quality         60
Security        40
Portability     90

$ uv run skillseal test examples/bad-skill

helper

Should trigger       5/5
Should NOT trigger   5/7

Accuracy             83.3%

Failures:

✗ "Help me write a poem"
  Expected: NOT TRIGGER
  Actual: TRIGGER
  Likely reason:
  Matched terms: help

skillseal.yaml format

Place a skillseal.yaml next to a SKILL.md to define its routing tests:

version: 1

routing:
  should_trigger:
    - "Review this payment implementation"
    - "Check whether this Stripe integration is secure"

  should_not_trigger:
    - "Write a React button"
    - "Explain Kubernetes"

• A missing skillseal.yaml means that skill is skipped, not failed.
• Empty should_trigger/should_not_trigger lists are skipped too (no 0/0 false pass or divide-by-zero).
• Malformed YAML is a usage error (exit 2), not a crash.

Using it in CI

As a reusable GitHub Action (action.yml):

- uses: pespinel/skillseal@v0.2.1
  with:
    path: ./skills
    fail-on: error

Or driven directly, e.g. to also run routing tests:

- uses: astral-sh/setup-uv@v3

- name: Check Agent Skills
  run: uvx skillseal check ./skills --fail-on error

- name: Test Agent Skill Routing
  run: uvx skillseal test ./skills

This repo's own .github/workflows/ci.yml does the same against examples/, plus lint/type-check/unit tests.

pre-commit

repos:
  - repo: https://github.com/pespinel/skillseal
    rev: v0.2.1
    hooks:
      - id: skillseal

Runs skillseal check . whenever a SKILL.md changes.

Releasing

Publishing to PyPI is automated via .github/workflows/release.yml using PyPI Trusted Publishing — no API token stored anywhere.

1. Bump version in pyproject.toml.
2. Commit, then tag: git tag vX.Y.Z && git push origin vX.Y.Z.
3. The workflow verifies the tag matches pyproject.toml, builds the sdist and wheel, signs a SLSA build provenance attestation, and publishes to PyPI via OIDC.

To verify a release artifact was actually built by this repo's workflow (not hand-uploaded) before installing it:

gh attestation verify dist/skillseal-*.whl --owner pespinel

The score

Deterministic, no LLM involved. Each of the four categories starts at 100 and loses points per finding:

INFO findings (like "requires docker") are purely descriptive — declaring a real dependency isn't a defect, so it doesn't cost points. Rules aggregate repeated occurrences of the same issue into one finding with a count, so a long file can't rack up an artificially low score just from file size.

The total is a weighted sum of the four category scores:

Architecture

src/skillseal/
├── models.py           # pydantic models: Skill, Finding, SkillReport, routing models
├── parser.py            # discover_skills(), parse_skill() — never raises on bad YAML
├── linter.py             # ties parser + rules + scoring together
├── scoring.py             # deterministic 0-100 scoring
├── conflicts.py            # cross-skill: duplicate names, routing-overlap (Jaccard)
├── rules/
│   ├── base.py             # Rule protocol, FuncRule, registry, text helpers
│   ├── metadata.py          # SPECIFICATION rules
│   ├── quality.py            # QUALITY rules
│   ├── security.py           # SECURITY rules
│   └── portability.py         # PORTABILITY rules
├── routing/
│   ├── evaluator.py           # HeuristicRoutingEvaluator, LLMRoutingEvaluator, LLMProvider
│   └── runner.py               # loads skillseal.yaml, runs cases
├── reporters/
│   ├── terminal.py             # Rich terminal output
│   └── json_reporter.py         # stable JSON schema
└── cli.py                        # typer app: check, test, conflicts

A Rule is id, category, severity, description, and check(skill) -> list[Finding]. Most rules are built with FuncRule, which wraps a plain function so adding a check doesn't require a new class.

Routing evaluation is behind a RoutingEvaluator protocol with two implementations:

• HeuristicRoutingEvaluator (default): fully offline, no API key needed. Scores how much of a prompt's distinctive vocabulary (after stopword removal and light suffix stripping) is covered by the skill's own name, description, and keywords:. It's deliberately simple — not real NLP — which is also why it's fast, free, and explainable ("Matched terms: ...").
• LLMRoutingEvaluator: delegates the trigger/no-trigger decision to an LLMProvider (complete(prompt) -> str). OpenAICompatibleProvider implements this against any OpenAI-compatible /chat/completions endpoint, configured via SKILLSEAL_BASE_URL, SKILLSEAL_API_KEY, and SKILLSEAL_MODEL. Use --provider llm to opt in — it's never required.

Limitations

• Rules are regex/heuristic-based, not a real parser or NLP model — they will have false positives and false negatives. Findings are phrased as potential risk, never certainty.
• The heuristic routing evaluator uses simple tokenization and suffix stripping, not real stemming or embeddings; words like "secure" and "security" won't match each other.
• Token counts are a rough len(text) // 4 estimate, not a real tokenizer.
• No sandboxing or dynamic execution — nothing in a skill is ever run.
• No compatibility testing against real agents (Claude Code, Codex, Gemini, etc.) — see the roadmap.

Roadmap

Near-term, likely next:

• A config file (skillseal.toml) to override thresholds (size, description length, etc.) without forking a rule

Documented, not implemented, on purpose — deliberately out of scope for now:

• Real execution against Claude Code, Codex, Gemini, and other agents
• A compatibility matrix across agents/environments
• Sandboxed dynamic analysis of skill-invoked commands
• Auto-fix for common findings
• Version-to-version comparison for a skill
• A GitHub App
• A web dashboard
• A skill registry / marketplace
• A hosted/cloud service
• Telemetry
• Skill certification

License

MIT