mcp-fuzz

Runtime behavioral testing for MCP servers.

mcp-doctor reads an MCP server's source code and checks whether its tools are well-documented. mcp-fuzz does the opposite: it actually launches the server and calls its tools, with inputs derived from each tool's own declared JSON schema, and checks whether the server behaves the way that schema and its description claim — does a missing required field get a structured error back, or does the server crash? Does a wrong-typed field get rejected cleanly, or does it hang?

Static analysis can't see any of that. Only running the code can.

It still deliberately stops short of judging whether a successful call's response is actually correct — a schema-only placeholder value usually isn't realistic enough to fairly judge that. mcp-reality-check is the third tool in the family that picks up exactly that: does it fail safely, and does it actually work.

Install

pip install mcp-runtime-check

(The PyPI distribution name is mcp-runtime-check — mcp-fuzz and close variants were blocked by PyPI's anti-typosquat check as too similar to existing packages, same naming friction mcp-doctor hit. The installed CLI command is still mcp-fuzz, and the repo/import package are unchanged.)

Use

mcp-fuzz -- python server.py
mcp-fuzz -- npx -y some-mcp-server

mcp-fuzz launches the command you give it as an MCP server over stdio, lists its tools, and for each one runs three kinds of calls built purely from that tool's own inputSchema — no LLM, no network calls of its own:

• valid — one plausible value per property (respecting type, enum, minimum/maximum, format, ...) — should succeed.
• missing required — the valid call, with one required property removed at a time — should come back as a structured MCP error, not a crash.
• wrong type — the valid call, with one property swapped to a value of a different JSON type — same expectation.

Any call that crashes the server, hangs past --timeout (default 15s), or leaves the connection unusable triggers a full reconnect before the next case runs, so one bad tool doesn't invalidate the rest of the report.

Safety — read this before pointing it at anything real

mcp-fuzz actually executes tool calls. Unlike mcp-doctor, it has real side effects if a tool does. By default, only tools annotated readOnlyHint: true are tested — everything else is skipped and listed as such in the report. Pass --include-destructive to test everything, but only against a server you're confident is safe to call blindly (a local sandbox, a test/staging backend) — never a server wired to production data, a real inbox, a real payment system, etc. Many real-world servers don't set readOnlyHint accurately or at all, in which case those tools are conservatively skipped rather than assumed safe.

What the score means

The reported "crash resilience" percentage covers only the missing-required and wrong-type cases — the fraction that came back as a structured error instead of a crash or hang. It does not grade whether the tool's "valid" call produced a correct result: a synthetic, schema-only-derived value (a placeholder string where the field really expects a real arXiv ID, or a URL that has to actually resolve) often isn't realistic enough for a failure there to be a fair judgment. A failed "valid" call is reported separately, flagged explicitly as "may be a synthetic-input false positive, not a confirmed bug" — worth a manual look, not proof of a bug.

JSON output / CI

mcp-fuzz --json -- python server.py
mcp-fuzz --fail-under 90 -- python server.py   # non-zero exit if crash resilience < 90%

Real-world spot check

A bug in mcp-fuzz itself, and the two-step fix it took to get right

The firecrawl false-0%/F above led to a genuinely tricky classification bug, worth documenting honestly rather than glossing over. The client SDK raises the same exception type, MCPError, for two completely different situations:

1. A real, well-formed JSON-RPC error response actually received from the server — e.g. -32602 INVALID_PARAMS when schema validation (zod, pydantic, ...) rejects a bad call. This is exactly the "structured error back" a well-behaved server is supposed to return — not a crash.
2. A client-synthesized error for a transport failure (-32000 CONNECTION_CLOSED) or the SDK's own internal request timeout (-32001 REQUEST_TIMEOUT) — no real response was ever received. Still a crash/timeout.

The first fix attempt only excluded case 2 and treated every other MCPError as graceful. That over-corrected: re-running it against antvis/mcp-server-chart (the 37.85%/F finding above) silently flipped it to a false 100%/A. The real cause: many frameworks use a third code, -32603 INTERNAL_ERROR, to wrap an unhandled exception from the tool's own business logic (a raw TypeError: Cannot read properties of null) so it doesn't kill the whole process — a completed round trip, but not the server "behaving the way its schema claims" either. The correct rule, verified against both repos simultaneously (each has the other as its regression test): only -32602 INVALID_PARAMS is trusted as "properly handled"; every other code, including -32603, stays a crash. Re-checked against both real repos afterward: firecrawl-mcp-server 100%/A, antvis/mcp-server-chart back to the original, correct 37.85%/F (exactly 133/214) — the fix removed the false positive without touching the real finding.

Hardened against malformed tool metadata (not just bad tool-call arguments)

Everything above tests whether a target server's tool handlers fail safely on bad input. A comment on the trilogy writeup pointed out the gap one level up: nothing tested whether mcp-fuzz itself stays standing when the tools/list response it's fuzzing from is malformed — the contract between mcp-fuzz and the server, not the server's own tool logic.

It didn't. Feeding the schema generator adversarial tools/list metadata by hand turned up three real crashes in mcp-fuzz's own code, all in the same place — real MCP servers just don't happen to send this kind of metadata, so nothing here had exercised it before:

• properties (or the whole inputSchema) coming back as a non-object — AttributeError: 'list' object has no attribute 'items'.
• required coming back as a string instead of a list — silently iterated character-by-character instead of raising, which is arguably worse than a crash (wrong output with no signal anything was off).
• A schema nested a few thousand levels deep — RecursionError, well within what a misbehaving code generator could produce by accident.

Fixed by treating any of these the same way the rest of the module already treats "no usable type information": generate a safe placeholder or nothing at all, never propagate the exception. A depth cap (50 levels — no real tool schema goes anywhere near that) stops the recursion case; type checks before every .items()/iteration stop the rest. 4 new regression tests (41 total).

Known limitations

• Input generation is schema-only. A property with no type (or a genuinely ambiguous anyOf) is skipped from the wrong-type test set rather than guessed at.
• No semantic check of what a successful response actually contains — that's a deliberately separate, opt-in, LLM-backed capability planned for a later release, not v1.
• stdio transport only for now; no HTTP/SSE servers yet.

License

MIT