skill-conflict-detector

Stop your agent from doing the wrong thing — before users notice.

中文版 · Apache-2.0 · PyPI release coming soon — pip install git+https://github.com/frkentno1-cloud/skill-conflict-detector.git

---

Your agent is only as reliable as its skill routing. When two skills both claim "use me when the user asks for X", the router picks one — often the wrong one. Your agent quietly does the wrong thing, and nobody knows why until trust is gone.

skill-conflict-detector finds those collisions automatically, proposes concrete fixes, applies them with your approval (with automatic backup), and re-tests until your library is clean. One command. Minutes. A few cents.

The problem, concretely

Every skill declares when to trigger in its description. With a handful of skills, overlaps are easy to spot. With 50+? 200+? They're inevitable — and hand-maintaining non-overlapping descriptions at that scale is impossible.

The failure is silent: no error, no crash, just a wrong tool doing the wrong job. It's the #1 hidden killer of agent reliability, and it compounds as your library grows.

What this tool does

┌─────────┐   ┌──────────┐   ┌──────────┐   ┌────────┐   ┌────────┐
│  check  │ → │ suggest  │ → │  apply   │ → │ verify │ → │(loop)  │
│  detect │   │ propose  │   │(with     │   │ re-test│   │ until  │
│  clashes│   │  edits   │   │ backup)  │   │        │   │ clean  │
└─────────┘   └──────────┘   └──────────┘   └────────┘   └────────┘

1. check — one LLM call over your entire library (name + description of every skill). Whole-library view: it spots clusters, not just pairs. Output: JSON + HTML report.
2. suggest — for each clash, concrete edits: which file, which field, change it to what. It also classifies each conflict: text-fixable (rewrite descriptions to draw a boundary) vs architectural (the skills genuinely overlap — flagged for manual merge, never auto-applied).
3. apply — every edit is shown to you first (y/n). Originals are backed up automatically.
4. verify — re-runs the analysis. Clean? Done. Not clean? Loops (default max 3 rounds).

Everything is reversible: rollback restores any change from backup.

Why you can trust the approach

• Validated by published research. A Single Rewrite Suffices: Empirical Lessons from Production Skill Description Optimization (arXiv 2606.30775) shows a single LLM rewrite of overlapping descriptions reaches ~79% F1 on routing accuracy — on par with hand-tuned descriptions (79.4%) — at ~32× lower effort (3.8 min vs 120 min per skill). It also separates text-fixable collisions from genuinely overlapping scopes — exactly what our fix_type field encodes.
• Precision over recall. We'd rather miss a conflict than cry wolf: mere description similarity (daily report vs weekly report) is not a conflict. Real user requests must be genuinely ambiguous. In our own smoke test on an internal 37-skill library: 8/8 conflict clusters found, 0 false positives.
• Deterministic. temperature=0, whole-library single-pass, structured JSON output with fabrication guards (invented skill IDs are dropped).

Quick start (2 minutes)

# PyPI release coming soon — install from source for now:
pip install git+https://github.com/frkentno1-cloud/skill-conflict-detector.git
export LLM_API_KEY=sk-...          # any OpenAI-compatible or Anthropic key

# Point it at your skills directory and run the full loop:
skill-conflict-detector audit --path skills/

audit walks you through every proposed edit (y/n), applies the ones you approve, backs up the originals, and re-scans until clean — or until nothing further can be auto-applied (remaining conflicts need manual handling). That's the whole workflow in one command.

Tired of the long name? alias scd="skill-conflict-detector".

Commands

Command	What it does
skill-conflict-detector check	Detect conflicts → JSON + HTML report
skill-conflict-detector suggest	Generate fix suggestions (optionally fed misrouting evidence, below)
skill-conflict-detector audit	Full loop: detect → suggest → confirm → apply → re-test
skill-conflict-detector apply <suggestions.json>	Apply suggestions from a JSON file (interactive; add --yes for non-interactive CI)
skill-conflict-detector rollback	Restore the last change (--session <ts> / --suggestion <id> / --all)
skill-conflict-detector backups	List backup sessions
skill-conflict-detector validate	Hard checks, no LLM (duplicate names, description quality)

Configuration (skill-conflict-detector.toml)

[llm]
provider = "openai"            # openai / anthropic / deepseek / any OpenAI-compatible gateway
model = "gpt-4.1-mini"         # cheap models work; quality comes from the prompt
api_key_env = "LLM_API_KEY"    # key read from environment variable
base_url = ""                  # set for a self-hosted gateway

[scan]
paths = ["skills/"]
exclude = []

[report]
out_dir = "./skill-conflict-detector-reports"

[apply]
backup_dir = "./.skill-conflict-detector-backups"
confirm = "interactive"        # interactive / all
max_loop = 3                   # re-test loop cap

Optional: feed it real misrouting evidence

Have logs of actual routing mistakes? Condition the fixes on them:

skill-conflict-detector suggest --path skills/ --evidence misrouting.json

misrouting.json: [{"user_request": "...", "routed_skill_id": "a", "expected_skill_id": "b", "note": "..."}]

CI integration

Two different CIs are worth distinguishing:

• This repo's CI (.github/workflows/skill-conflict-detector.yml, shipped): installs the package and runs the unit-test suite on every push/PR.
• Your skill library's CI — keep conflicts out of main forever. Add a workflow to your skills repo:

on:
  push:
    paths: ['skills/**', 'skill-conflict-detector.toml']
  pull_request:
    paths: ['skills/**', 'skill-conflict-detector.toml']
jobs:
  skillcheck:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - name: Install
        run: pip install -e git+https://github.com/frkentno1-cloud/skill-conflict-detector.git
      - name: Validate skills (no LLM)
        run: skill-conflict-detector validate --path skills/   # errors block the build
      - name: Conflict detection (LLM, optional)
        if: env.LLM_API_KEY != ''
        env: { LLM_API_KEY: ${{ secrets.LLM_API_KEY }} }
        run: skill-conflict-detector check --path skills/ --json

1. validate — hard checks, no LLM; duplicate skill names are errors and block the build (they make detection itself ambiguous).
2. conflict detection — runs when secrets.LLM_API_KEY is set, skipped otherwise.
3. reports — check writes JSON + HTML under skill-conflict-detector-reports/; upload them as artifacts.

Run everything on your own repo first (check / audit locally) before wiring it into CI — the LLM needs a configured provider ([llm] in skill-conflict-detector.toml).

FAQ

Is this a syntax linter for SKILL.md? No — structural linters exist. This targets semantic clashes: descriptions that would both match the same request. Syntax errors never broke a routing; ambiguity does.

Which skill formats does it support? Any library of Markdown files with YAML frontmatter (name + description) — the emerging open format adopted across Claude Code, Codex, Cursor, Gemini CLI, and many other agents. If your agent reads SKILL.md, this works.

Does it need an LLM key? Yes for conflict detection; validate works without one. Cost is trivial: a full analysis of 200 skills is roughly ¥1–3 ($0.15–0.45) on cheap models.

Will it edit my skills without asking? Never. Every edit is shown for approval, originals are backed up, and rollback restores them. fix_type: scope / manual items (merges, renames, platform-level issues) are never auto-applied, even if the model says otherwise. Skills with duplicate names in the library are never auto-edited (which copy would you be fixing?).

Why does check refuse to run on my library? Duplicate skill names are hard validation errors — detection with two skills sharing an id is meaningless. Run skill-conflict-detector validate --path skills/ to see the offending files (identical = redundant copies; version_conflict = historical versions, pick one), fix them, and re-run.

I'm a PM, not an engineer. Then audit is your friend: one command, human-readable questions, human-readable HTML report. No code to write, nothing to configure beyond one API key.

Design & development

• DESIGN.md — full design rationale (detection mechanism, closed loop, related work)
• Tests: python tests/test_skeleton.py && python tests/test_rollback.py && python tests/test_robustness.py && python tests/test_validate_real.py (54 cases; test_validate_real.py gracefully skips its real-library section when the internal /skills tree is absent)
• testdata/ and extract_skills.py are intentionally gitignored (internal-library samples)

License

Apache-2.0. See LICENSE.