mcp-reportcard

A health check for MCP servers. Lint your tools, measure their token cost, fuzz their error handling — and get a graded report card with a concrete fix for every issue.

When an agent picks the wrong tool, sends malformed arguments, or burns context on bloated definitions, the cause is usually in the server — and the model only ever sees what your tools/list says. mcp-reportcard checks exactly that, in seconds, for any MCP server in any language.

Try it in the browser: huggingface.co/spaces/builditwithgk/mcp-reportcard — paste a tools/list JSON, get the report card.

Why it's different

It grades what the model sees, not what your code does.

The MCP Inspector is for poking at a server by hand. mcp-reportcard is automated and opinionated: it scores tool names, descriptions, schemas and safety annotations against the things that actually make models misbehave — vague descriptions, undocumented parameters, near-duplicate tools, missing readOnlyHints, stack traces in error replies — and it runs in CI.

Every finding comes with a rule ID, a severity and a specific fix. No LLM is involved in the checks, so results are deterministic and safe to gate a merge on.

Example output

Running against the deliberately sloppy server in examples/bad_server.py:

┌──────── mcp-reportcard · acme-orders (bad example) ─────────┐
│ Grade D  ·  score 67/100                                 │
│ 7 tools  ·  1 errors  ·  16 warnings  ·  5 info          │
│ ~351 tokens per turn (tiktoken o200k_base (approximate)) │
└──────────────────────────────────────────────────────────┘
  F002 lookup_invoice Error output exposes internals: `Traceback (most recent call last)`
       fix: Log stack traces server-side; return a short, safe message to the client.
  X001 server `find_items` and `search_items` are described almost the same way (88% word overlap)
       fix: Say in each description when to use it instead of the other, or merge them.
  ...

The same service written properly, examples/good_server.py, scores A (100).

Field-tested: run against six official MCP reference servers (Node and Python, stdio and HTTP) — 159 fuzz cases, zero crashes, and two over-eager rules fixed as a result. Numbers and details in FINDINGS.md.

Install & run

Requires Python 3.10+.

pip install "mcp-reportcard[tokens]"

# …or run it without installing (uv)
uvx --from "mcp-reportcard[tokens]" mcp-reportcard check "python server.py"

# …or from source
git clone https://github.com/builditwithgk/mcp-reportcard
cd mcp-reportcard
pip install -e ".[dev]"

The [tokens] extra adds tiktoken for more accurate token counts; without it (pip install mcp-reportcard), a characters/4 estimate is used.

Command line

# A stdio server (any language): pass the command that starts it
mcp-reportcard check "python server.py"
mcp-reportcard check "npx -y @modelcontextprotocol/server-everything"

# A Streamable HTTP server
mcp-reportcard check http://localhost:8000/mcp

# A saved tools/list JSON (offline, no server needed)
mcp-reportcard check tools.json

# Also test error handling with bad inputs (read-only tools only — see below)
mcp-reportcard check "python server.py" --fuzz

# Reports for PRs and CI
mcp-reportcard check "python server.py" -f markdown -o report.md
mcp-reportcard check "python server.py" -f json --fail-under 80

# Save a server's tool list (for offline checks or the web demo)
mcp-reportcard dump "python server.py" -o tools.json

# List every check
mcp-reportcard rules

GitHub Action

Grade your server on every pull request. The action writes the report card to the job summary, keeps one sticky PR comment up to date, and fails the check when the score drops below your bar:

# .github/workflows/mcp-reportcard.yml
on: pull_request
permissions:
  contents: read
  pull-requests: write   # for the PR comment
jobs:
  mcp-reportcard:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5          # or setup-node, etc. — whatever your server needs
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt   # your server's own dependencies
      - uses: builditwithgk/mcp-reportcard@v1
        with:
          server: python server.py             # a command, an http(s) URL, or a saved tools/list JSON
          fuzz: "true"
          fail-under: "80"

Outputs: grade, score, report (Markdown path) and json (JSON path). mcp-reportcard installs into its own virtualenv, so the Python your server runs on is left untouched.

pre-commit

Run it before every commit with pre-commit:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/builditwithgk/mcp-reportcard
    rev: v0.1.2
    hooks:
      - id: mcp-reportcard
        args: ["python server.py", "--fail-under", "80"]   # or a tools/list JSON file
        files: ^server\.py$                               # optional: only when these files change

Prefer the plain CLI? pip install mcp-reportcard and mcp-reportcard check "python server.py" --fail-under 80 (exits 1 below the bar; --json-out writes JSON alongside a Markdown report).

This repo dogfoods both: CI runs the CLI and the action against examples/good_server.py with --fail-under 90 on Linux, macOS and Windows.

What it checks

Scoring: each tool starts at 100 and loses 25 per error, 10 per warning and 3 per info. The server score is the average tool score minus server-level penalties (capped at 40). Grades: A ≥ 90, B ≥ 80, C ≥ 70, D ≥ 60, F below.

Fuzzing safety

--fuzz calls your tools with missing arguments, wrong types, out-of-enum values, 10,000-character strings and tricky Unicode, then checks for clean isError responses.

• By default it only calls tools annotated readOnlyHint: true (and not destructiveHint: true). Everything else is skipped — and listed in the report, never silently.
• --allow-unsafe fuzzes every tool. Only use it against a test instance, never production data.
• A stdio server's own stderr is hidden to keep the report readable; add --server-logs to see it.

Web demo

The demo runs the static checks only — it never connects to servers, calls tools or contacts an LLM, and nothing pasted is stored. Because the core is pure standard-library Python, it ships two ways from the same code:

• In the browser — space/ is a static page that runs src/mcp_reportcard client-side with Pyodide. That's the Hugging Face Space: no server, free tier. bash space/build.sh assembles it into space/_site/; bash space/deploy.sh publishes it. The same folder works on GitHub Pages or any static host.
• As a Flask app — web/, for Python hosting such as cPanel: create a Python app with web/ as the root and passenger_wsgi.py as the startup file, install web/requirements.txt, and upload src/ and examples/ alongside.

pip install flask
python web/app.py     # http://localhost:5000

Limitations

• Token counts are estimates. Each provider wraps tool definitions in its own format; use the numbers to compare tools and track changes, not as a bill.
• Description checks are heuristics. A D005 ("no usage guidance") is a nudge, not a hard rule.
• Leak detection (F002) looks for stack traces only. File paths in error messages are not flagged — file and git tools legitimately name paths.
• Fuzzing can't prove a tool is correct; it shows how it fails on bad input.
• v1 checks tools only, not resources or prompts.

Roadmap

• Tool-selection eval (bring your own API key): generate realistic tasks per tool and measure how often a model picks the right one.
• README badge (mcp-reportcard: A).
• Resource and prompt checks.

Development

python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
pytest

Tests cover every lint rule, scoring, leak detection, fuzz-case generation, the CLI and the web demo — plus end-to-end runs against both example servers, in-process and as real stdio subprocesses.

Author

Built by Gopi K Aitham (builditwithgk) — see scaleup-solutions.in. Available for freelance and contract work on AI, agent, and MCP tooling.

License

MIT