Nikhail

Lint agent capabilities. Nikhail statically analyzes the two artifact types that determine how well AI agents discover and use capabilities:

• Agent Skills — SKILL.md packages (frontmatter, instructions, bundled scripts and references)
• MCP tools — tool definitions as returned by an MCP server's tools/list

It is designed to run locally as a CLI and in CI as a GitHub Action, with rules grounded in published research: the skill-smells study (arXiv:2607.01456) and the MCP description-quality study (arXiv:2602.14878).

The pieces (8)

Commands

Setup (once)

cd Nikhail
uv venv                      # create .venv (or: python3 -m venv .venv)
uv pip install -e ".[dev]"   # editable install + pytest
source .venv/bin/activate    # or prefix commands with .venv/bin/

Run the linter

nikhail fixtures/skills/smelly-skill        # lint one skill (findings, exit 0 — warnings only)
nikhail fixtures/mcp/tools-smelly.json      # lint MCP tool definitions (errors, exit 1)
nikhail fixtures/                            # walk a tree: finds every SKILL.md + tools JSON
nikhail ~/.claude/skills                     # try it on your real skills
nikhail . --fail-on warning                  # stricter gate: warnings also fail the run
nikhail . --format json                      # machine-readable output
nikhail . --sarif nikhail.sarif            # also write SARIF (what the Action does)
nikhail . --html report.html                 # self-contained interactive HTML report
                                               #   skills: findings + source excerpts
                                               #   MCP: per-tool cards pairing each finding
                                               #   with the description the model reads
                                               #   (embeds source excerpts — treat like source)
nikhail . --handoff handoff.md               # self-contained fix pack: findings + full
                                               #   source per flagged artifact, absolute
                                               #   paths — enough context for a fixing
                                               #   agent in a fresh session
                                               #   (embeds sources — treat like source)
nikhail --list-rules                         # show the rule catalog with citations
nikhail . --disable S007                     # skip a rule, by id or name; repeatable,
                                               #   comma lists OK (--disable S007,S018)
nikhail . --kind mcp                         # lint only MCP tool JSON (or --kind skill)
nikhail . --advisory                         # + LLM-assisted advisory checks (see below)
nikhail . --advisory --fail-on advisory      # opt in to gating on advisory findings

Exit codes: 0 clean (below the --fail-on threshold), 1 findings at/above threshold, 2 usage error. Lint an MCP server you run by dumping its tools/list response to JSON and passing that file.

Run the tests

pytest -q          # full suite: rule behavior on fixtures + CLI exit codes + SARIF shape

Test the GitHub Action locally (before GitHub ever sees it)

The testing ladder, cheapest first:

# 1. The CLI itself — this IS 95% of the Action (constant, instant)
nikhail fixtures/skills/good-skill --fail-on warning && echo PASS

# 2. The workflow wiring — act runs .github/workflows/ci.yml in Docker (occasional)
brew install act           # one-time; needs Docker running
act push -j tests          # pytest job locally
act push -j fixture-contract  # exit-code contract job locally
act push -j action-selftest   # composite action via 'uses: ./' — see act note below

# 3. Server-side rendering (PR annotations, code-scanning upload) — only on GitHub (rare)
#    Push to a scratch repo and reference the action by branch:
#    uses: prove-ai/Nikhail@main

act notes: the .actrc here selects catthehacker/ubuntu:act-latest (closer to GitHub's real runner than act's slim default) and forces linux/amd64 (needed on Apple Silicon). The tests and fixture-contract jobs deliberately use the runner's system Python in a venv instead of actions/setup-python, because setup-python's post-job cache step hits a known act bug (node not found — nektos/act#107) that fails the job cosmetically. action-selftest still uses setup-python (inside the composite action, as real consumers will), so under act its main steps succeed but the post step may report that same cosmetic failure — its true end-to-end check is the scratch-repo step below. What act cannot emulate: PR annotations, the Checks UI, and SARIF upload to code scanning — that's what the scratch repo is for. Validate SARIF locally first at https://sarifweb.azurewebsites.net/Validation .

Use the Action in another repo (once pushed to GitHub)

- uses: actions/checkout@v4
- uses: prove-ai/Nikhail@v1
  with:
    path: .claude/skills
    fail-on: warning

v1 rule catalog

Skills (SKILL.md) — S008 parseable frontmatter · S001 has description · S002 description long enough to support activation · S003 description under the 1024-char limit · S004 kebab-case name matching the directory · S005 body within the progressive-disclosure budget (500 lines) · S006 referenced files exist · S007 no orphan bundled files · S009 description states when to use the skill · S010 name under the 64-char limit · S011 no XML/HTML tags in the description · S012 forward-slash paths only · S013 body under 5,000 words · S014 description in third person · S015 name conveys the capability · S016 no rigid multi-command sequences · S017 long workflows decomposed into steps · S018 substantial skills include an example · S019 no time-anchored statements.

Coverage of the 26 skill smells (arXiv:2607.01456, Table III)

The paper classifies only 5 of its 26 smells as statically detectable; the other 21 required an LLM-based detector even for the authors. Nikhail v1 is deterministic, so:

• Fully covered — the paper's entire static set (5/5): Lengthy Skill Description → S003, Lengthy Skill Body → S013 (+ stricter S005 line budget), Lengthy Skill Name → S010, XML Included Description → S011, Backslash Path → S012.
• Covered by conservative deterministic heuristics (7): Confusing Skill Description → S001+S002+S009, Non Third Person Description → S014, Unclear Skill Name → S015 (+S004), Series of Commands → S016, Stepless Workflow → S017, Missing Example → S018, Time Sensitive Skill → S019.
• Covered by the opt-in LLM advisory tier (14) — A001–A014: Option Buffet, Missing Utility Script, Missing Decision Tree, No Validation Step, Execute Without a Plan, Never Asks Human, Rationalization Loophole, No Progress Tracking, Undelegated Detail, No Guardrails, Buried Gotchas, Missing Usage Rules, Missing Caveats, Missing Template. These are absence-based semantic judgments no deterministic rule can make; per the paper's own prevalence data they occur in 60–94% of real skills, so any keyword-absence approximation would be pure CI noise.

The full mapping also lives in the skills.py module docstring.

The advisory tier (v2, --advisory)

The 14 semantic smells are detected the way the paper's own SSD detector works: one binary LLM classification per smell per file, each prompt carrying that smell's definition plus a positive and a negative example. Because an LLM verdict is a judgment, not a fact, the tier is engineered to be safe next to the deterministic rules:

• Opt-in and non-gating. Nothing runs without --advisory; findings carry the advisory severity, which ranks below info and never affects the exit code unless you explicitly pass --fail-on advisory. In SARIF they map to note.
• Reproducible by construction. Every verdict is cached keyed by (file content hash, smell, model, prompt version) under ~/.cache/nikhail/advisory (--cache-dir / NIKHAIL_CACHE_DIR). Unchanged files are never re-evaluated, so CI re-runs are byte-identical cache hits; nondeterminism can only enter when a file actually changes.
• Evidence-anchored. When the model claims a smell is present because of specific text, it must quote it — and the quote is verified against the file. A hallucinated quote kills the finding mechanically. Absence-based verdicts (you can't quote an omission) are labeled unanchored in the message.
• Cheap deterministic pre-gates. Some classifiers only run when the question applies (e.g. No Progress Tracking requires an actual multi-step workflow).
• Parallel. The per-smell calls for a file run concurrently in batches of 10, so a cold file classifies in a few seconds; cached files are instant.
• Model: the latest Haiku (claude-haiku-4-5 alias — tracks new Haiku snapshots; the resolved model is part of the cache key, so upgrades invalidate stale verdicts). Override with --model / NIKHAIL_MODEL. Credentials resolve however the Anthropic SDK normally does (ANTHROPIC_API_KEY, ant auth login profile, …).
• Install: pip install 'nikhail[advisory]'. Eval: pytest -m live runs both sides against the real API (skips cleanly without credentials): recall — every fixtures/smells/ positive must be flagged by its mapped A-rule — and precision — every fixtures/smells-clean/ counterpart (a skill containing the mitigation for its smell) must NOT be flagged by that rule. The eval re-earns all verdicts in a throwaway cache each run by design; that cost is eval-only, not CLI usage.

MCP tools — M001 has description · M002 description ≥ 10 words · M003 snake_case name · M004 has inputSchema · M005 parameters described · M006 parameters typed · M007 no near-duplicate descriptions across tools (Jaccard overlap) · M008 description ≤ 300 words · M009 states when to use the tool · M010 states limitations/caveats · M011 not example-only content · M012 more than one sentence · M013 states what the call returns or changes.

Coverage of the six MCP description smells (arXiv:2602.14878)

The paper derives six smells from a six-component rubric (component score < 3 = smelly): Unclear Purpose → M001 + M013 · Missing Usage Guidelines → M009 · Unstated Limitation → M010 · Opaque Parameters → M005 + M006 · Exemplar Issues (examples-vs-description balance) → M011 · Underspecified or Incomplete → M002 + M012. M009/M010/M012/M013 are conservative cue heuristics at info severity (the paper's own scanner is FM-based); M011 flags the confident degenerate case at warning.

All v1 rules are deterministic: no LLM calls, no network, stable output — safe to gate CI on.

License

Versions 0.1.x are released under the MIT License. Future versions may be released under a different license.

Roadmap status

v1 (deterministic rules) and the first slice of v2 (LLM-assisted advisory tier for the 14 semantic skill smells) are implemented. v3 (does fixing findings measurably help?) has its first results: on a real 53-skill repo (garrytan/gbrain), applying description-level findings improved skill-routing accuracy consistently across two router models, with the gain concentrated on requests that don't use the skill's own vocabulary — see bench/README.md for the numbers and caveats. Still deliberately out of scope: security scanning (delegate to Snyk agent-scan / Cisco skill-scanner instead of competing), declared-vs-observed script consistency (v2), cross-skill composition analysis (v2), cross-model selection benchmarking (v3), automatic rewriting with A/B measurement (v4).