Security scanners tell you if your MCP server is dangerous. mcp-quality tells you if it's good — and blocks the merge when it gets worse. It grades any server across five dimensions into a single MCP Quality Score, runs deterministically in CI with no LLM and no key, and prints a badge.

$ mcp-quality run "python my_server.py" --legibility

  MCP Quality Score  67   Grade D    ⚠ hard-gate: 'legibility' capped the grade at C

  Cost         A   100   122 toolset tokens; $0.0004/task
  Legibility   F     0   50% right-tool selection; archive_record⇄delete_record 100%
  Contract     A   100   2 tools conform

  Disambiguation matrix  (row = correct tool · cell = % of times chosen)
  ┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
  ┃ correct ↓ / chose → ┃ delete ┃ archiv ┃
  ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
  │ delete_record       │   100% │      · │
  │ archive_record      │   100% │     0% │   ← agents pick delete when archive was right
  └─────────────────────┴────────┴────────┘
  ↳ proposed rewrite: "Move a record to the archive (reversible); use delete_record to remove permanently."

Two tools described "Remove a record by id." — a data-loss bug living in your tool descriptions, invisible to every test you have. mcp-quality catches it and writes the fix.

Quickstart

pip install mcp-quality

mcp-quality run "python my_server.py"                       # graded A–F report
mcp-quality run "python my_server.py" --json --fail-under B # CI gate (exit 1 if < B)
mcp-quality static ./server.mcp.json                        # offline / air-gapped

The zero-LLM fast path (Contract + Cost) is deterministic and needs no model, key, or network beyond your server. Add --legibility --model ollama:qwen2.5-3b for the disambiguation matrix, --all for every family, or --deep-security to fold in mcp-scan / Cisco findings.

Use it in CI

# .github/workflows/mcp-quality.yml
name: MCP Quality
on: [push, pull_request]
jobs:
  mcp-quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install mcp-quality
      - run: mcp-quality run "python my_server.py" --fail-under B --no-regressions

Commit a baseline with mcp-quality snapshot "…" and --no-regressions fails the PR when a commit silently breaks a tool or drops a score. mcp-quality badge emits an mcp-quality: A SVG + shields.io endpoint for your README.

The check families

Overall score = a weighted, versioned rubric; measured families are renormalized, and an F in any family hard-gates the grade to C.

Reliability overlay — --reliability K

Accuracy asks "can it work?"; reliability asks "will it work every time?" — the number a deploy gate actually cares about. --reliability K reruns the nondeterministic families K times and reports pass^k: the projected probability all K runs pass. A server that works 9 times in 10 is a 10% incident rate — pass^k over 5 runs is 59%, not an A. Deterministic families (Contract, Cost, Safety) short-circuit to 100% with no reruns, so the fast path stays free.

Experimental — --experimental (spec-surface)

Beyond tools, MCP servers can talk back: sampling/createMessage, elicitation/create, and advertised resources — the surfaces where a server can smuggle instructions into your model, phish your user, or dangle links that don't resolve. --experimental adds a spec-surface family that drives the read-only tools, captures those server-originated messages, and grades them (sampling-injection, over-broad context, sensitive-form elicitation, unresolvable resources). It's opt-in and carries zero rubric weight — it's reported but never moves your grade while these spec surfaces are still stabilizing.

Where it fits

mcp-quality doesn't compete with the security scanners or the point tools — it unifies the quality question into a CI gate. Credit to mcp-xray, which pioneered token-tax + tool-confusion scoring (we borrow its leave-one-out method).

Commands

Leaderboard

mcp-quality run against real public MCP servers (full table):

_{¹ Its one tool is stateful, so the determinism probe (identical args → different results) flags undeclared nondeterminism. Fix = declare the output volatile. A lower grade is an invitation to a PR, not a verdict.}

Part of the Swarm Proof toolkit

Trust infrastructure for the agent economy — seven projects, one thesis.

Docs

Demo · Leaderboard · Architecture · Requirements (PRD) · Design decisions · Roadmap · Changelog · Contributing

License

Apache-2.0. Provider-agnostic and Ollama-friendly; the CI-critical path needs no LLM. Citable via CITATION.cff.