mcp-fuzz

Runtime behavioral testing for MCP servers.

Real output from a live --include-destructive run against the official @modelcontextprotocol/server-memory reference server — not a synthetic example.

mcp-doctor reads an MCP server's source code and checks whether its tools are well-documented. mcp-fuzz does the opposite: it actually launches the server and calls its tools, with inputs derived from each tool's own declared JSON schema, and checks whether the server behaves the way that schema and its description claim — does a missing required field get a structured error back, or does the server crash? Does a wrong-typed field get rejected cleanly, or does it hang?

Static analysis can't see any of that. Only running the code can. It also measures things static analysis structurally can't: how long each tool actually takes to respond (Latency check), how large its response actually is (Response size check), whether it behaves correctly when several callers hit it at once (Concurrency check), and, opt-in, whether a resource actually stays deleted once a tool says it deleted it (Resource lifecycle check).

It still deliberately stops short of judging whether a successful call's response is actually correct — a schema-only placeholder value usually isn't realistic enough to fairly judge that. mcp-reality-check is the third tool in the family that picks up exactly that: does it fail safely, and does it actually work.

Install

pip install mcp-runtime-check

(The PyPI distribution name is mcp-runtime-check — mcp-fuzz and close variants were blocked by PyPI's anti-typosquat check as too similar to existing packages, same naming friction mcp-doctor hit. The installed CLI command is still mcp-fuzz, and the repo/import package are unchanged.)

Use

mcp-fuzz -- python server.py
mcp-fuzz -- npx -y some-mcp-server

mcp-fuzz launches the command you give it as an MCP server over stdio, lists its tools, and for each one runs three kinds of calls built purely from that tool's own inputSchema — no LLM, no network calls of its own:

• valid — one plausible value per property (respecting type, enum, minimum/maximum, format, ...) — should succeed.
• missing required — the valid call, with one required property removed at a time — should come back as a structured MCP error, not a crash.
• wrong type — the valid call, with one property swapped to a value of a different JSON type — same expectation.

Any call that crashes the server, hangs past --timeout (default 15s), or leaves the connection unusable triggers a full reconnect before the next case runs, so one bad tool doesn't invalidate the rest of the report.

By default, only a minimal, safe set of environment variables (PATH, HOME, ...) is passed to the target server — never your full shell environment. Many real servers need an API key to start at all; pass one through explicitly rather than relying on the ambient environment:

mcp-fuzz --env BRAVE_API_KEY=... -- npx -y @brave/brave-search-mcp-server

Safety — read this before pointing it at anything real

mcp-fuzz actually executes tool calls. Unlike mcp-doctor, it has real side effects if a tool does. By default, only tools annotated readOnlyHint: true are tested — everything else is skipped and listed as such in the report. Pass --include-destructive to test everything, but only against a server you're confident is safe to call blindly (a local sandbox, a test/staging backend) — never a server wired to production data, a real inbox, a real payment system, etc. Many real-world servers don't set readOnlyHint accurately or at all, in which case those tools are conservatively skipped rather than assumed safe.

What the score means

The reported "crash resilience" percentage covers only the missing-required and wrong-type cases — the fraction that came back as a structured error instead of a crash or hang. It does not grade whether the tool's "valid" call produced a correct result: a synthetic, schema-only-derived value (a placeholder string where the field really expects a real arXiv ID, or a URL that has to actually resolve) often isn't realistic enough for a failure there to be a fair judgment. A failed "valid" call is reported separately, flagged explicitly as "may be a synthetic-input false positive, not a confirmed bug" — worth a manual look, not proof of a bug.

JSON output / CI

mcp-fuzz --json -- python server.py
mcp-fuzz --fail-under 90 -- python server.py   # non-zero exit if crash resilience < 90%

Latency check

Every call was always timed internally (see Full-fidelity trace export below) — this surfaces that timing as a real, scored part of the report instead of leaving it buried in an opt-in export. For each tested tool's valid call (the one call that does real work, unlike a bad-input call that's typically rejected before any real work happens), mcp-fuzz flags a tool as slow when either is true:

• it takes longer than an absolute threshold (default 5000ms, override with --slow-threshold-ms), or
• it's a clear outlier relative to this same server's other tools — more than 3x the server's own median (only checked once there are at least 3 comparably-tested tools; calling one thing an "outlier" against a sample of one or two isn't a fair comparison).

A tool whose valid call crashed or timed out is excluded from the latency check entirely, not counted as merely "slow" — that failure is already the crash-resilience score's job to report, and its duration is contaminated by reconnect/timeout overhead rather than real service time anyway.

mcp-fuzz --slow-threshold-ms 3000 -- python server.py
mcp-fuzz --fail-under-latency 90 -- python server.py

Read this before treating the latency score as more precise than it is: this is one real call per tool, not a load test — no repeated sampling, no percentiles, no warm-up run. It's a cheap, real signal worth a manual look (a single wildly slow tool among fast siblings usually does mean something real — an unbounded upstream call, a missing timeout, an N+1 query), not a benchmark you'd cite as a tool's actual production latency.

Response size check

Same two-signal design as latency, applied to response size instead of response time: a tool that dumps an unusually large payload burns an agent's context window for no reason an agent can see coming from the tool's own description. For each tested tool's valid call, mcp-fuzz flags a tool as bloated when either is true:

• its response is longer than an absolute threshold (default 20,000 characters, roughly 5,000 tokens on the common ~4-chars/token rule of thumb for English text — a cheap estimate stated as such, not real tokenization; override with --bloat-threshold-chars), or
• it's a clear outlier relative to this same server's other tools — more than 3x the server's own median response length (same minimum-sample-size guard as latency).

Same crash/timeout exclusion as latency, for the same reason: no real response to measure the size of.

mcp-fuzz --bloat-threshold-chars 10000 -- python server.py
mcp-fuzz --fail-under-response-size 90 -- python server.py

Same caveat as latency: one real call per tool, not representative of every possible input a tool could return. A tool genuinely designed to return a lot of content (a full-file read, a config dump) isn't a bug just because it's large relative to its neighbors — this is a "worth a look" signal, not a confirmed problem.

Concurrency check — opt-in

Everything above calls one tool at a time. This is the one check that doesn't: with --concurrency N, for each tested tool, mcp-fuzz also launches N independent connections — each its own subprocess of the target server command — and calls the tool with the same valid arguments on all of them at once. Independent connections, not N calls fanned out over one shared session, deliberately: it's a real test of concurrent access to whatever backend the server itself talks to (a shared file, database, lock, rate limiter), the kind of bug that N sequential calls on one connection structurally can't surface.

mcp-fuzz --concurrency 5 -- python server.py
mcp-fuzz --fail-under-concurrency 90 -- python server.py

Off by default — it means N extra subprocess launches per tool, real cost for a real server. A tool is flagged only when the concurrent failure is actually informative:

• some concurrent calls succeeded and others didn't, on the identical input — proof the failure is load-dependent, not just unrealistic synthetic data (the same argument this tool's own README elsewhere applies leniently to a lone valid-call error doesn't apply here, since the same data demonstrably worked for at least one other concurrent caller), or
• the tool's own sequential call succeeds fine, but it fails under concurrent load — the sharper signal of the two, since it usually means a deadlock or resource-starvation bug rather than a tool that's simply broken.

A tool whose sequential call also already fails, failing the exact same way on every one of the N concurrent calls too, is deliberately not re-flagged here — that's not new information, just the same already-known brokenness (already surfaced by the crash-resilience score) showing up again.

Note on how a failure actually shows up: an application-level exception raised inside a tool handler is commonly caught by the server's own framework (FastMCP, for one, verified directly) and returned as a normal isError: true response, not a raw crash — this check treats that the same as a crash/timeout when it happens under concurrent load, since excluding it would silently miss most real concurrency bugs.

Resource lifecycle check — opt-in, chains a real id

Everything above tests one tool call in isolation, always with synthetic, schema-derived arguments — even a get_item-shaped tool is called with a placeholder id that was never actually created. That structurally can't catch a real and common bug class: does a tool behave correctly on a resource another tool actually created, and — the more interesting case — does a "deleted" resource stay deleted?

--sequential (requires --include-destructive, since it creates and deletes real data) looks for create_X/get_X/delete_X-shaped tool names sharing the same resource (also recognizes add/insert/new, read/fetch/retrieve/describe/show, remove/destroy). For each group found:

1. Calls create_X with a normal schema-derived valid call.
2. Extracts a real id from the real response (a JSON id/{resource}_id/uuid field, or a bare UUID in plain text) — gives up cleanly rather than guessing if nothing id-shaped is found.
3. Calls get_X/delete_X with that real id in place of a synthetic one.
4. If both a read and a delete tool exist, re-calls get_X with the same id after deletion — a stale read: get_X still reporting success on an id delete_X just removed — is the specific finding this check exists to catch.

mcp-fuzz --include-destructive --sequential -- python server.py

Off by default, same reasoning as --concurrency: it's strictly more destructive than testing a single write tool alone (it creates and deletes a real resource). The resource grouping is a deliberately conservative name heuristic — same discipline as this project's other name-based checks — and gives up rather than guessing on anything ambiguous: two create_-shaped tools for the same resource name, no id-shaped field anywhere in a create response, or a dependent tool whose schema doesn't have an unambiguous id-shaped or sole-required-string property. A server whose create/read/delete tools don't follow this naming convention, or whose id isn't returned in the response body at all, produces zero detected groups — reported plainly as "0 groups detected," not treated as a pass.

Full-fidelity trace export

--json's report is deliberately narrow — it exists to answer "what's the crash-resilience score," so it drops every "ok" outcome, the call arguments, and any timing. That's the right shape for the score, wrong shape for reconstructing what a run actually did call by call — useful if you want to feed a real session into an external evidence or trajectory-debugging tool.

mcp-fuzz --full-trace session.jsonl -- python server.py

Writes one JSON line per tool call (tool, case, property, outcome, detail, arguments, startedAt, durationMs), plus one line per skipped tool — independent of, and in addition to, whatever --json/text report you also asked for.

Added after an actual external ask, not speculatively: agent-inspect (a TypeScript agent-trajectory/evidence-debugging tool) asked what it would take to feed a real mcp-fuzz session into its evidence model. Before this flag existed, the only way to do that was a one-off script that duplicated mcp-fuzz's own input generators just to recover the arguments and timing --json throws away — this flag makes that a first-class, supported path instead. Feeding a session through confirmed a real, useful finding along the way: mcp-fuzz's graceful_error/valid_call_errored outcomes (the good, correctly-handled case) map naturally onto a generic trajectory tool's status:error/"failed" vocabulary, which reads a healthy 100%/A crash-resilience session as mostly broken unless the consumer knows to treat those two outcomes as distinct from crash/timeout. Worth remembering for anyone else piping this JSONL into a similar tool.

Resolved upstream, not just reported: filed as agent-inspect#362 with a reproducible 114-call session and a working fix demonstration against AgentInspect's own outcome_observed schema. AgentInspect shipped --preset behavioral-session in v6.26.0+, which keeps a graceful MCP rejection as a tool-level error while scoring the expected-behavior verdict separately, and merged the sanitized session as an attributed fixture (agent-inspect#403, examples/recipes/mcp-behavioral-session/). Reran the exact documented commands against current main (v6.29.1) afterward to check for drift: the merged fixture is byte-identical to the original conversion, and both the default check (still collapses to one run.status finding) and --preset behavioral-session (run.requireCompleted passes, 21 findings from outcome.status) reproduce exactly — no regression through review or merge.

Real-world spot check

A real usability gap: no way to pass an API key to the target server

Trying to fuzz brave/brave-search-mcp-server surfaced a gap that had nothing to do with crash detection: the server refuses to start at all without BRAVE_API_KEY, and mcp-fuzz's CLI had no way to pass any environment variable through to the target. The MCP SDK's own stdio_client deliberately doesn't inherit the operator's full shell environment when none is given — only a minimal, safe allowlist (PATH, HOME, ...) — a real security default against leaking secrets into an arbitrary launched process. But that meant any server needing an API key (Brave, financial-datasets/mcp-server, and presumably many others) simply couldn't be tested. Added --env KEY=VALUE (repeatable), merged onto the SDK's own safe baseline rather than replacing it — losing PATH entirely would break npx before the target server ever ran, a strictly worse outcome than the default. 4 new tests (45 total): the real fix (env passed through → connects), the safety property it must not regress (no env → the server still refuses to start, confirming this isn't quietly leaking the full shell environment instead), and the merge behavior itself as a pure unit test. Shipped as v0.1.4.

A bug in mcp-fuzz itself, and the two-step fix it took to get right

The firecrawl false-0%/F above led to a genuinely tricky classification bug, worth documenting honestly rather than glossing over. The client SDK raises the same exception type, MCPError, for two completely different situations:

1. A real, well-formed JSON-RPC error response actually received from the server — e.g. -32602 INVALID_PARAMS when schema validation (zod, pydantic, ...) rejects a bad call. This is exactly the "structured error back" a well-behaved server is supposed to return — not a crash.
2. A client-synthesized error for a transport failure (-32000 CONNECTION_CLOSED) or the SDK's own internal request timeout (-32001 REQUEST_TIMEOUT) — no real response was ever received. Still a crash/timeout.

The first fix attempt only excluded case 2 and treated every other MCPError as graceful. That over-corrected: re-running it against antvis/mcp-server-chart (the 37.85%/F finding above) silently flipped it to a false 100%/A. The real cause: many frameworks use a third code, -32603 INTERNAL_ERROR, to wrap an unhandled exception from the tool's own business logic (a raw TypeError: Cannot read properties of null) so it doesn't kill the whole process — a completed round trip, but not the server "behaving the way its schema claims" either. The correct rule, verified against both repos simultaneously (each has the other as its regression test): only -32602 INVALID_PARAMS is trusted as "properly handled"; every other code, including -32603, stays a crash. Re-checked against both real repos afterward: firecrawl-mcp-server 100%/A, antvis/mcp-server-chart back to the original, correct 37.85%/F (exactly 133/214) — the fix removed the false positive without touching the real finding.

Hardened against malformed tool metadata (not just bad tool-call arguments)

Everything above tests whether a target server's tool handlers fail safely on bad input. A comment on the trilogy writeup pointed out the gap one level up: nothing tested whether mcp-fuzz itself stays standing when the tools/list response it's fuzzing from is malformed — the contract between mcp-fuzz and the server, not the server's own tool logic.

It didn't. Feeding the schema generator adversarial tools/list metadata by hand turned up three real crashes in mcp-fuzz's own code, all in the same place — real MCP servers just don't happen to send this kind of metadata, so nothing here had exercised it before:

• properties (or the whole inputSchema) coming back as a non-object — AttributeError: 'list' object has no attribute 'items'.
• required coming back as a string instead of a list — silently iterated character-by-character instead of raising, which is arguably worse than a crash (wrong output with no signal anything was off).
• A schema nested a few thousand levels deep — RecursionError, well within what a misbehaving code generator could produce by accident.

Fixed by treating any of these the same way the rest of the module already treats "no usable type information": generate a safe placeholder or nothing at all, never propagate the exception. A depth cap (50 levels — no real tool schema goes anywhere near that) stops the recursion case; type checks before every .items()/iteration stop the rest. 4 new regression tests (41 total).

Known limitations

• Input generation is schema-only. A property with no type (or a genuinely ambiguous anyOf) is skipped from the wrong-type test set rather than guessed at.
• No semantic check of what a successful response actually contains — that's a deliberately separate, opt-in, LLM-backed capability planned for a later release, not v1.
• stdio transport only for now; no HTTP/SSE servers yet.
• The resource-lifecycle check (--sequential) only chains a single create → read/delete step per detected group, not an arbitrary multi-step plan across several different resources — a real bug in a longer, cross-resource workflow (an agent's actual multi-tool plan, not just one resource's own lifecycle) is out of scope for this check specifically.

License

MIT