MCPProbe

Tells an MCP server author, in one run, whether their server is spec-compliant, robust to bad input, and usable by a real model.

Status: v1.0.0. Conformance (CONF), schema fuzzing (FUZZ), robustness (ROB) and security (SEC) suites over stdio and Streamable HTTP, for the stateless 2026-07-28 revision and the legacy initialize era (2025-11-25, 2025-06-18). Each server is classified as modern, legacy, dual-era or unknown, and only the checks for the era(s) it speaks run.

Every check cites the spec section it enforces, every failure is saved as a byte-for-byte replayable case, and the same seed always gives the same report.

Install

Requires Python 3.11+. No compiler needed on any platform.

pipx install mcpprobe-kit        # or: uv tool install mcpprobe-kit

The package is mcpprobe-kit; the command is mcpprobe. (PyPI refuses "mcpprobe" and "mcpprobe-cli" as too similar to unrelated existing projects.)

Full documentation: https://boyndasouth.github.io/MCPProbe/ (quickstart, CI recipes, configuration, the check catalog).

GitHub Action

- uses: boyndasouth/MCPProbe@v1
  with:
    stdio: node dist/index.js
    seed: "42"

Runs every suite, uploads SARIF to code scanning, attaches JSON, JUnit and HTML reports as an artifact, writes a job summary, and fails the job on error findings. Add baseline: mcpprobe-baseline.json to fail only on new ones. See docs/ci.md for every input.

Configuration

Options can live in mcpprobe.toml next to your server (command-line options win). ${NAME} reads an environment variable, so secrets stay out of the file:

[server]
stdio = "python -m my_server"
env = { API_KEY = "${API_KEY}" }

[run]
seed = 42

[checks]
disable = ["CONF-012"]
severity_overrides = { "SEC-002" = "info" }

See docs/configuration.md.

Usage

# A stdio server: give the command that starts it. Pin @latest with npx:
# a bare `npx -y pkg` silently reuses whatever version it cached first.
mcpprobe run --stdio "npx -y @modelcontextprotocol/server-everything@latest"
mcpprobe run --stdio "python -m my_server" --env API_KEY=...

# A Streamable HTTP server
mcpprobe run --http http://127.0.0.1:8000/mcp --header "Authorization: Bearer $TOKEN"

# Reproducible run with a machine-readable report
mcpprobe run --stdio "node dist/index.js" --seed 42 --report json -o probe.json

# Reports for CI: JUnit for test tabs, SARIF for GitHub code scanning, HTML to share
mcpprobe run --stdio "node dist/index.js" --report sarif -o probe.sarif --report html -o probe.html

# See which tools a run would call, without calling any
mcpprobe run --stdio "node dist/index.js" --dry-run

# Adopt in CI without fixing everything first: fail only on new findings
mcpprobe run --stdio "node dist/index.js" --report json -o probe-baseline.json  # once, commit it
mcpprobe run --stdio "node dist/index.js" --baseline probe-baseline.json

# Re-run one failure byte for byte against a fresh server
mcpprobe replay .mcpprobe/cases/CONF-020.json

# Browse the checks
mcpprobe checks list
mcpprobe checks explain CONF-006

Useful options for run:

Eras

Before any check runs, MCPProbe asks server/discover (the 2026-07-28 way) and, failing that, initialize (the legacy way), then probes a modern server with initialize on a fresh process or session to spot dual-era servers.

Checks

error checks enforce a MUST and fail the run; warn enforces a SHOULD or flags a deprecated feature; info is advice and shows as a note.

FUZZ, ROB and SEC run once per server, in the modern era when the server speaks it: they test how the server handles input, which does not depend on the era.

SEC-005 only applies to HTTP servers that require authorization. Give MCPProbe a valid token with --header "Authorization: Bearer ...": SEC-005 then checks what a client without one (or with a bad one) is told. Secrets found by SEC-003 are never printed in full and are masked in saved cases. The liveness probe after every case turns a crash or hang into a finding, whatever the check's severity. | CONF-021 | warn | legacy | initialize answers an unsupported version with a supported one | | CONF-022 | info | legacy | Server negotiates an outdated revision (2025-03-26 or older) |

mcpprobe checks explain <ID> prints what a check sends, what it expects, and a link to the spec section.

Fuzzing

The FUZZ suite calls each tool --examples times (default 50) with arguments generated from its inputSchema by Hypothesis: valid values, and near-misses with one thing wrong (a required field missing, a wrong type, a value just past a bound, an extra property, null, deep nesting, a 100,000-character string, odd Unicode). Each argument set is classified as valid or invalid by validating it against the tool's own schema, so a finding never rests on MCPProbe's guess. One case per tool, FUZZ-001/<tool>, judges all four FUZZ checks from the same calls.

Every finding is shrunk to a minimal input ({"message": ""} rather than a page of random Unicode) and saved as a recording of just that call on a fresh server, so mcpprobe replay reproduces it in one step. Crashes are shrunk too, restarting a stdio server between attempts; requests that are never answered are not, since each attempt would cost a full timeout. Calls carry a progressToken, and a tool that keeps reporting progress gets up to three times --timeout before MCPProbe moves on (without a finding: slow is not stuck).

Generation, shrinking and case order all derive from --seed, and shrinking is bounded by a call budget rather than a clock, so the same seed gives the same report on any machine.

Generation uses hypothesis-jsonschema, which speaks JSON Schema drafts 4 to 7; 2020-12 schemas (the MCP default) are translated for the generator, and keywords with no draft-7 form (unevaluatedProperties) only cost coverage, never a wrong verdict. Tools whose schemas the generator cannot handle (recursive $ref, or a remote $ref, which MCPProbe never fetches) are skipped with the reason.

Calling your tools safely

CONF-008 to 011 and the FUZZ suite call real tools. MCPProbe only calls tools marked readOnlyHint: true or destructiveHint: false (MCP treats unannotated tools as destructive), and never tools marked openWorldHint: true. Any URL, email or hostname it generates points at a reserved .invalid name, and IP addresses come from the documentation ranges. --allow-destructive and --allow-open-world lift those rules; don't use them against a server connected to real data or credentials. Fuzzing sends many unusual inputs, so test against a staging copy.

Developer tools

record sends a file of JSON-RPC messages (one per line, byte for byte) and saves everything that crossed the wire; replay re-sends it to a fresh server and reports any difference.

mcpprobe record --stdio "npx -y @modelcontextprotocol/server-everything" \
  --send examples/legacy-session.jsonl -o everything.json --timeout 60
mcpprobe replay everything.json --timeout 60

Secrets passed with --env or sensitive --header values are masked in every saved recording; pass them again to replay.

Roadmap

Development

Requires uv.

uv sync
uv run pytest
uv run ruff format . && uv run ruff check . && uv run mypy

See CONTRIBUTING.md and checks/README.md.

src/mcpprobe/
  cli.py          Typer entry point
  transport/      raw stdio, HTTP and SSE wire client + byte recorder
  protocol/       request building, era detection, schema instances
  engine/         runner, seeds, check catalog, record and replay
  suites/         conf/, fuzz/, rob/, sec/ (eval/ in M5)
  report/         terminal, json
checks/           one YAML file per check
fixtures/servers/ reference server; --break <ID> violates one check

License

MIT