skillrecall

Measure how reliably an agent skill gets picked when it is installed next to other skills, and get a short, ranked list of edits that make it get picked more.

A skill is a directory with a SKILL.md. The host reads only the name and description when it decides which skill to use, so those two fields are what an author is really designing. skillrecall puts your skill in a room with its competitors, sends in a sample of realistic requests, counts how often yours wins, tries concrete edits to the description, and tells you which ones measurably help.

You edit, run it again, and it tells you what moved.

Install

uv tool install git+https://github.com/modiqo/skillrecall

Or pin a release wheel:

uv tool install https://github.com/modiqo/skillrecall/releases/download/v0.1.0/skillrecall-0.1.0-py3-none-any.whl

pipx install git+https://github.com/modiqo/skillrecall works the same way. The package is not on PyPI yet; uv tool install skillrecall will work once it is.

The core has no dependencies beyond Python 3.11. Optional extras:

Use

skillrecall assess path/to/my-skill

That competes your skill against the closest skills in the public catalog. The skill does not have to be on disk. A skills.sh link, a GitHub link, or an owner/repo/skill id works the same way; the skill's files are fetched once and cached:

skillrecall assess https://skills.sh/owner/repo/my-skill
skillrecall assess https://github.com/owner/repo/tree/main/skills/my-skill
skillrecall assess owner/repo/my-skill

Add the skills already installed on your machine, or a directory of your own:

skillrecall assess path/to/my-skill --installed
skillrecall assess path/to/my-skill --corpus ./our-team-skills
skillrecall assess path/to/my-skill --installed --corpus ./our-team-skills --neighbours 60

Real requests beat generated ones. If you have them, put one per line in a file:

skillrecall assess path/to/my-skill --tasks-file requests.txt

What you get

Skill: landing-clarity
Competition: 28 other skills (28 from the public catalog)

How often you get picked
  When a task is yours, you are chosen about 6 in 10.
  Confidence: moderate (100 sample tasks).
  In bigger requests that need several skills, you are among the top 3 about 7 in 10.

Who takes your tasks
  landing-page-review (acme/skills)       takes about 1 in 10   44 installs
  landing-page (jezweb/claude-skills)     takes about 1 in 10   38 installs

Whose tasks you take
  landing-pages (someone/skills)          you answer about 1 in 10 of its tasks by mistake

Do these, in order
  1. Say the words people use when they ask for this (headline, subhead); for example add “Also covers headline and subhead.”
     Expected: picked more often, shows up more in bigger tasks.
  2. Remove sentence 4: “Trigger even if the user only pastes a URL, a screenshot, or a block of copy.”
     Expected: shorter by 20 tokens, no loss.

  If you apply them together, the header becomes:
    description: Audit a landing page or hero section for clarity ... Also covers headline and subhead.
  Expected: picked about 9 in 10 (from about 6 in 10); mistaken pickups almost never; 9 tokens shorter.

Structure
  [consider] The description never says what the skill produces.
        Add one clause naming the result, for example “returns a ranked list of fixes”.

Every edit shown was applied to a copy of your description and rescored on the same requests. Edits that would hurt are not shown. When you rerun after editing, the report opens with what changed.

Output formats

skillrecall assess my-skill                       # human, simple
skillrecall assess my-skill --detail detailed     # adds sizes, every competitor, every edit tried, sample tasks
skillrecall assess my-skill --explain             # appends how the numbers were measured
skillrecall assess my-skill --format json         # machine-readable, same sections
skillrecall assess my-skill --format json --detail detailed -o report.json
skillrecall history my-skill                      # previous runs for this skill

The JSON has a stable top level: skill, competition, pickup, takes_your_tasks, you_take_theirs, edits, suggested, structure, since_last_run. Detailed adds neighbours, all_edits, terms, sample_tasks, scorers, and options.

Options that matter

Catalog searches and fetched competitor files are cached under ~/.cache/skillrecall for an hour and a week respectively. Run history lives under ~/.local/state/skillrecall. Set GITHUB_TOKEN to raise the rate limit when many competitors live in repositories with unusual layouts.

Use from Python

from skillrecall import Options, assess

result = assess(Options(skill_path="path/to/my-skill", include_installed=True))
print(result.recall.value, result.confidence)
for edit in result.edits:
    print(edit.instruction, "->", edit.verdict)
report = result.as_dict(detail=True)

How it works

The method is described step by step in ALGORITHM.md. In one paragraph: requests are sampled from your skill's body and from the bodies of its closest competitors; each request is routed across every competitor's name and description by a lexical scorer that treats hand-off sentences as rules, plus an optional embedding scorer; you are "picked" when you score highest; rates carry bootstrap intervals; and every suggested edit is a change that was applied and rescored on the same requests before being shown to you.

Development

uv venv && uv pip install -e ".[dev]"
uv run pytest
uv build

License

MIT