mcp-quality

The CI quality suite for MCP server builders

Lint, contract-test, benchmark, and load-test your MCP server before you ship it. The pytest + lighthouse for the servers agents depend on.

▶ See the live demo — the token tax and the disambiguation matrix, captured from real runs. Or the leaderboard: mcp-quality run against real public MCP servers.

$ mcp-quality run "python my_server.py" --legibility
Legibility  F   archive_record⇄delete_record confused 100% of the time   ← caught before you ship
Cost        D   8,792 toolset tokens — paid on every turn, before anything works

Status: v0.1 implemented — all five families, CLI, CI, badge. See the roadmap.

Why

Security scanners already answer "is this server malicious?" (mcp-scan, Cisco's mcp-scanner, and others own that lane). The quality question — "is this server any good?" — now has point tools too: mcp-xray scores token tax and tool confusion into a single grade, and Cisco's mcp-scanner added readiness heuristics. But those are things you run by hand. There is still no lighthouse/pytest for MCP: no CI-native suite that unifies the quality dimensions (contract, legibility, cost, performance, light security) into one graded gate, catches quality regressions across commits, prints a badge, and hands off to a full behavioral simulation.

mcp-xray is an X-ray you run by hand; mcp-quality is the pytest in .github/workflows that blocks the merge. It is a quality-and-reliability suite — it treats security as one check, defers deep security to the specialists (integrate, don't reinvent), and unifies the quality point tools rather than competing with them (credit where due — they prove the demand).

Quickstart

pip install mcp-quality
mcp-quality run "python my_server.py"        # graded A–F report + MCP Quality Score
mcp-quality run "python my_server.py" --json --fail-under B   # CI gate
mcp-quality static ./server.mcp.json         # offline / air-gapped CI

Use it in CI

# .github/workflows/mcp-quality.yml
name: MCP Quality
on: [push, pull_request]
jobs:
  mcp-quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install mcp-quality
      - run: mcp-quality run "python my_server.py" --fail-under B --no-regressions

The zero-LLM fast path (Contract + Cost) runs deterministically with no model and no network beyond your server. Add --all for the full five families, --legibility --model ollama:qwen2.5-3b for the agent-comprehension score, or --deep-security to fold in mcp-scan / Cisco findings. Commit a baseline with mcp-quality snapshot "…" so --no-regressions can catch a silently-broken tool in the PR.

The five check families

Contract (LLM-free): spec conformance, schema validity, determinism probe, snapshot regression. · Legibility (the differentiator): agent-comprehension score, the disambiguation matrix (delete vs archive confusion rate), description lints with proposed rewrites. · Cost: token weight of your whole toolset, per-tool bloat, $-per-task estimates. · Performance: concurrent-agent load with real MCP semantics (not naive HTTP), p50/p95/p99, connection-leak detection. · Security-lite: OWASP MCP Top 10 basics, with --deep-security shelling out to mcp-scan / Cisco.

Outputs a terminal report, JSON for CI, and an mcp-quality: A badge for your README. See SPEC.md and ROADMAP.md.

Part of the Swarm Proof toolkit

Trust infrastructure for the agent economy — seven projects, one thesis.

License

Apache-2.0. Zero-LLM fast path for Contract/Cost/Performance; legibility checks are provider-agnostic and Ollama-friendly. Citable via CITATION.cff.