mcp-reportcard

A health check for MCP servers. Lint your tools, measure their token cost, fuzz their error handling — and get a graded report card with a concrete fix for every issue.

When an agent picks the wrong tool, sends malformed arguments, or burns context on bloated definitions, the cause is usually in the server — and the model only ever sees what your tools/list says. mcp-reportcard checks exactly that, in seconds, for any MCP server in any language.

Try it in the browser: huggingface.co/spaces/builditwithgk/mcp-reportcard — paste a tools/list JSON, get the report card.

Why it's different

It grades what the model sees, not what your code does.

The MCP Inspector is for poking at a server by hand. mcp-reportcard is automated and opinionated: it scores tool names, descriptions, schemas and safety annotations against the things that actually make models misbehave — vague descriptions, undocumented parameters, near-duplicate tools, missing readOnlyHints, stack traces in error replies — and it runs in CI.

Every finding comes with a rule ID, a severity and a specific fix. No LLM is involved in the checks, so results are deterministic and safe to gate a merge on.

Example output

Running against the deliberately sloppy server in examples/bad_server.py:

┌──────── mcp-reportcard · acme-orders (bad example) ─────────┐
│ Grade D  ·  score 67/100                                 │
│ 7 tools  ·  1 errors  ·  16 warnings  ·  5 info          │
│ ~351 tokens per turn (tiktoken o200k_base (approximate)) │
└──────────────────────────────────────────────────────────┘
  F002 lookup_invoice Error output exposes internals: `Traceback (most recent call last)`
       fix: Log stack traces server-side; return a short, safe message to the client.
  X001 server `find_items` and `search_items` are described almost the same way (88% word overlap)
       fix: Say in each description when to use it instead of the other, or merge them.
  ...

The same service written properly, examples/good_server.py, scores A (100).

Field-tested: run against six official MCP reference servers (Node and Python, stdio and HTTP) — 159 fuzz cases, zero crashes, and two over-eager rules fixed as a result. Numbers and details in FINDINGS.md.

Install & run

Requires Python 3.10+.

pip install "mcp-reportcard[tokens] @ git+https://github.com/builditwithgk/mcp-reportcard"

# …or from source
git clone https://github.com/builditwithgk/mcp-reportcard
cd mcp-reportcard
pip install -e ".[dev]"

A PyPI release (pip install mcp-reportcard) is coming. The [tokens] extra adds tiktoken for more accurate token counts; without it, a characters/4 estimate is used.

Command line

# A stdio server (any language): pass the command that starts it
mcp-reportcard check "python server.py"
mcp-reportcard check "npx -y @modelcontextprotocol/server-everything"

# A Streamable HTTP server
mcp-reportcard check http://localhost:8000/mcp

# A saved tools/list JSON (offline, no server needed)
mcp-reportcard check tools.json

# Also test error handling with bad inputs (read-only tools only — see below)
mcp-reportcard check "python server.py" --fuzz

# Reports for PRs and CI
mcp-reportcard check "python server.py" -f markdown -o report.md
mcp-reportcard check "python server.py" -f json --fail-under 80

# Save a server's tool list (for offline checks or the web demo)
mcp-reportcard dump "python server.py" -o tools.json

# List every check
mcp-reportcard rules

Use in CI

--fail-under exits 1 when the score drops below the threshold:

# .github/workflows/mcp-reportcard.yml
on: pull_request
jobs:
  mcp-reportcard:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install "mcp-reportcard @ git+https://github.com/builditwithgk/mcp-reportcard"
      - run: mcp-reportcard check "python server.py" --fuzz -f markdown -o mcp-reportcard.md --fail-under 80

This repo dogfoods it: CI runs mcp-reportcard against examples/good_server.py with --fail-under 90 on Linux, macOS and Windows.

What it checks

Scoring: each tool starts at 100 and loses 25 per error, 10 per warning and 3 per info. The server score is the average tool score minus server-level penalties (capped at 40). Grades: A ≥ 90, B ≥ 80, C ≥ 70, D ≥ 60, F below.

Fuzzing safety

--fuzz calls your tools with missing arguments, wrong types, out-of-enum values, 10,000-character strings and tricky Unicode, then checks for clean isError responses.

• By default it only calls tools annotated readOnlyHint: true (and not destructiveHint: true). Everything else is skipped — and listed in the report, never silently.
• --allow-unsafe fuzzes every tool. Only use it against a test instance, never production data.
• A stdio server's own stderr is hidden to keep the report readable; add --server-logs to see it.

Web demo

The demo runs the static checks only — it never connects to servers, calls tools or contacts an LLM, and nothing pasted is stored. Because the core is pure standard-library Python, it ships two ways from the same code:

• In the browser — space/ is a static page that runs src/mcp_reportcard client-side with Pyodide. That's the Hugging Face Space: no server, free tier. bash space/build.sh assembles it into space/_site/; bash space/deploy.sh publishes it. The same folder works on GitHub Pages or any static host.
• As a Flask app — web/, for Python hosting such as cPanel: create a Python app with web/ as the root and passenger_wsgi.py as the startup file, install web/requirements.txt, and upload src/ and examples/ alongside.

pip install flask
python web/app.py     # http://localhost:5000

Limitations

• Token counts are estimates. Each provider wraps tool definitions in its own format; use the numbers to compare tools and track changes, not as a bill.
• Description checks are heuristics. A D005 ("no usage guidance") is a nudge, not a hard rule.
• Leak detection (F002) looks for stack traces only. File paths in error messages are not flagged — file and git tools legitimately name paths.
• Fuzzing can't prove a tool is correct; it shows how it fails on bad input.
• v1 checks tools only, not resources or prompts.

Roadmap

• Tool-selection eval (bring your own API key): generate realistic tasks per tool and measure how often a model picks the right one.
• README badge (mcp-reportcard: A).
• Resource and prompt checks.
• PyPI release.

Development

python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
pytest

Tests cover every lint rule, scoring, leak detection, fuzz-case generation, the CLI and the web demo — plus end-to-end runs against both example servers, in-process and as real stdio subprocesses.

Author

Built by Gopi K Aitham (builditwithgk) — see scaleup-solutions.in. Available for freelance and contract work on AI, agent, and MCP tooling.

License

MIT