Skip to main content

MCP-CrashTest

Unit tests check single tool calls. Real agents fail on multi-step contracts.

State-aware random walks with POSIX inode-preserving rollback for Model Context Protocol (MCP) servers.

Version Python CI License: MIT

中文文档 (README_CN.md)


Why MCP-CrashTest?

Tools like MCP Inspector and standard unit tests validate single-step static calls. In production, AI agents execute 5–20 step tool chains where outputs from discovery/list tools feed downstream mutations, entities collide, and stdio pipes hang.

Inspired by reversing the state-aware random walk paradigm from ToolSpace (arXiv:2611.05036)—instead of keeping successful trajectories for model training and discarding rollbacks—MCP-CrashTest mines a dependency graph from tools/list, runs guided multi-step random walks inside ephemeral OS sandboxes, rolls back on failure to explore neighboring branches, prunes noisy intermediate steps, and outputs deduplicated minimal JSON-RPC reproducers.

What It Catches in Real MCP Servers

  1. Masked Errors (isError: false on SQL/Runtime Failures) — Protocol Violation (CRITICAL)
    In mcp-server-sqlite (verified on 2026-11-20 with MCP-CrashTest v0.2.1 in both --adapter none pure zero-shot mode and --adapter auto), SQL errors return isError: false, causing agents to treat error strings as valid query results (context poisoning):

    {
      "method": "tools/call",
      "params": {
        "name": "write_query",
        "arguments": {
          "query": "INSERT INTO users (id, name, email) VALUES (1, 'CollisionBob', 'bob_new@example.com');"
        }
      },
      "response": {
        "text": "Database error: UNIQUE constraint failed: users.id",
        "isError": false
      }
    }
    
  2. CLI Output Leaks Across Tools — Cross-Tool Contract Break (HIGH)
    In mcp-server-git, git_branch outputs human CLI formatting ("* main"). Passing that branch name directly to git_checkout, git_diff, git_show, or git_merge fails:

    [
      { "tool": "git_branch", "response": { "text": "* main", "isError": false } },
      { "tool": "git_merge", "arguments": { "source_branch": "* main" }, "response": { "text": "Error executing tool git_merge: Ref '* main' did not resolve to an object", "isError": true } }
    ]
    
  3. Silent State Corruption & Stdio Deadlocks — Protocol Violations (CRITICAL)
    Read-after-write verification catches tools that report isError: false without persisting state, as well as hung stdio buffers or severed child-process pipes (with automatic subprocess recycling).

  4. Agent Contract Smells — UX & Consistency Pitfalls (MEDIUM / LOW / INFO)
    Explicitly separated from protocol bugs: undocumented glob-only search_files matching, relative vs. parent-joined path contracts, no-upsert skip behavior (create_entities), and intra-server isError inconsistencies.


Quick Start

Requires Python >= 3.10.

1. Run Immediately via uvx or pip

# Run directly without cloning (once published on PyPI)
uvx mcp-crashtest --target sqlite --walks 15 --depth 6

# Or install from source
git clone https://github.com/SnowyLumiere/mcp-crashtest.git
cd mcp-crashtest
pip install -e ".[dev]"

2. Test Any Custom MCP Server (Zero-Shot Generic Mode)

Use -- to pass the server command cleanly without option collisions, and --adapter none to run pure zero-shot generic stateful fuzzing with zero domain-specific assumptions:

mcp-crashtest \
  --target my-server \
  --adapter none \
  --sandbox directory \
  --seed 42 \
  --walks 20 \
  --depth 6 \
  --fail-on-crash \
  --output reports/my_server_report.md \
  -- python3 -m my_mcp_server --root {sandbox}

For stateless/API-only servers, use --sandbox none (context-memory rollback and branch exploration still run):

mcp-crashtest \
  --target my-api-server \
  --adapter none \
  --sandbox none \
  --walks 20 \
  -- node dist/index.js --env test

3. Built-in Official Server Presets

Requires npx (filesystem / memory) or uvx + git (sqlite / git):

mcp-crashtest --target sqlite --walks 15 --depth 6 --output reports/sqlite_crash_report.md
mcp-crashtest --target git --walks 15 --depth 6 --output reports/git_crash_report.md
mcp-crashtest --target filesystem --walks 15 --depth 6 --output reports/filesystem_crash_report.md
mcp-crashtest --target memory --walks 15 --depth 6 --output reports/memory_crash_report.md

Architecture: Generic Engine vs. Domain Adapters

MCP-CrashTest separates the generic zero-shot stateful engine from optional domain contract adapters (mcp_crashtest/adapters/):

Capability Generic Mode (--adapter none) Built-in Domain Adapters (--adapter auto)
Schema Graph Mining Infers CREATE/READ/MUTATE/DELETE + DATA/STATE/RETRY edges from tools/list; handles Pydantic v2 anyOf/oneOf/enum and properties: null Adds domain-specific edge bindings for filesystem, sqlite, memory, git
Context Flow & Fuzzing Extracts IDs/keys/URIs/paths from args & JSON responses; injects boundary strings, traversal paths, extreme ints Adds domain-specific probes (e.g. PK collisions, glob-vs-substring queries, raw * main branch passing)
Post-Condition Probes Probe A (RETRY_TRAP), Generic Probe B (Read-After-Write on id/key/uri), Probe C (SCHEMA_BREAK), Probe D (STDIO_DEADLOCK) Adds calibrated CONTRACT_SMELL checks (e.g. search_files glob pitfall, describe_table [], create_entities no-upsert skip)
Report Provenance Every finding is tagged with source: generic Every adapter finding is explicitly tagged source: adapter:<name>

Sandbox Placeholders & Environment Injection

--sandbox CLI Placeholders Injected Env Var Seeded Initial Data
directory (default) {sandbox}, {work_dir} MCP_SANDBOX_DIR sample.txt, data_dir/config.json, data_dir/nested_note.txt
git {sandbox}, {work_dir} MCP_SANDBOX_DIR Initialized main branch + initial commit (commit.gpgsign=false)
sqlite {sandbox}, {db_path} MCP_SANDBOX_DB_PATH test_target.db with seeded users table (2 rows)
memory {sandbox}, {memory_path} MEMORY_FILE_PATH Isolated temp path memory_graph.json
none — — No local filesystem sandbox (context rollback still active)

Live Audit Summary (v0.2.1 · 4 Official MCP Servers)

Verified on 2026-11-20 (--walks 15 --depth 6 --rollbacks 3 --seed 42, Python 3.12, mcp SDK v2.3.0 client with full v1.x/v2.x compatibility; pruned reports in reports/):

Server Mode Protocol Violations Contract Smells Headline Minimal Reproducers Report
mcp-server-sqlite --adapter none (pure zero-shot) 3 (CRITICAL) 0 write_query, create_table, read_query return "Error: ..." with isError: false (1-step each, source: generic) —
mcp-server-sqlite --adapter auto 3 (CRITICAL) 1 (MEDIUM) SQL errors return isError: false (×3, 1-step); describe_table returns [] on missing table (1-step) sqlite
mcp-server-git --adapter auto 0 4 (HIGH SCHEMA_BREAK) git_branch leaks '* main', breaking git_checkout, git_diff, git_show, git_merge (2-step each) git
@modelcontextprotocol/server-filesystem --adapter auto 0 4 (MEDIUM) list_directory bare filenames vs read_file/read_text_file/get_file_info (2-step); search_files glob-only miss (1-step) filesystem
@modelcontextprotocol/server-memory --adapter auto 0 3 (LOW / INFO) create_entities skips existing entities with isError: false (2-step); delete_* idempotent 0 differs from add_* (1-step) memory

Notes:

  • The sqlite preset launches uvx --with "mcp<2" mcp-server-sqlite because upstream mcp-server-sqlite currently uses the mcp 1.x @server.list_resources() API and fails to boot under mcp 2.x. MCP-CrashTest itself supports both mcp SDK v1.x (inputSchema/isError) and v2.x (input_schema/is_error).
  • reports/MCP_CRASHTEST_COMPREHENSIVE_DIAGNOSIS_20261106.md is preserved as the historical v0.2.0 archive.

4 Contract Rules for Agent-Safe MCP Servers

  1. Strict isError Semantics: Never return isError: false on SQL errors, constraint violations, or failed mutations.
  2. Cross-Tool I/O Symmetry: Discovery/list tool outputs must be directly consumable by downstream tools without CLI decorations (* main) or stripped parent paths.
  3. Explicit Matching & Upsert Contracts: Document glob vs. substring matching and provide explicit upsert/conflict errors instead of ambiguous isError: false skips.
  4. Actionable Errors & Pipe Backpressure: Return offending parameter names and suggested corrective tools; paginate large responses to prevent stdio deadlocks.

CLI Reference

Flag Default Description
--target "custom" Target label or built-in preset (filesystem, sqlite, memory, git)
--adapter "auto" Domain adapter mode: auto, none (pure generic), or explicit adapter name(s)
--cmd / -- <cmd...> None Server executable (prefer POSIX -- passthrough to avoid flag clashes)
--sandbox "directory" directory | sqlite | git | memory | none
--walks 15 Number of random walk chains
--depth 6 Max step depth per walk
--rollbacks 3 Max rollback retries per walk
--fuzz-ratio 0.25 Boundary fuzzing probability (0.0–1.0)
--timeout 8.0 Per-step & tools/list timeout in seconds
--init-timeout 20.0 Session initialize() timeout in seconds
--seed None Local deterministic RNG seed (does not mutate global random state)
--lang "en" Report language: en | zh
--fail-on-crash False Exit with code 1 if findings matching --fail-on are detected (for CI gates)
--fail-on "any" CI failure scope: any (all findings) | protocol (PROTOCOL_VIOLATION only)
--output "reports/mcp_crash_report.md" Output Markdown report path
--version — Print version and exit

Known Limitations & Roadmap

  • Transport: Currently supports stdio transport only. Streamable HTTP / SSE support is on the roadmap.
  • SQLite Sandbox: In-place binary file rollback assumes per-call SQLite connections (such as mcp-server-sqlite). Long-lived connection pools with in-memory page caches require session restarts between walks.
  • Process Memory: For servers keeping unpersisted in-process state, sandbox rollback restores disk state only.

Python API & Running Tests

import asyncio
from mcp_crashtest.cli import run_crash_test

async def main():
    crashes = await run_crash_test(
        cmd="python3",
        args=["my_server.py", "--root", "{sandbox}"],
        target_name="my-server",
        adapter_mode="none",
        sandbox_type="directory",
        seed=42,
        lang="en",
        output_report_path="reports/my_server.md",
    )
    print(f"Hits: {len(crashes)} | Unique: {len({c.fingerprint for c in crashes})}")

asyncio.run(main())

Run the test suite (zero third-party dependencies required for unittest):

PYTHONPATH=. python3 -m unittest discover -s tests -v
# or with pytest
PYTHONPATH=. pytest tests/ -v

License

Released under the MIT License.

Metadata

Release files for mcp-crashtest 0.2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mcp-crashtest 0.2.1
File Size Uploaded Provenance Actions
mcp_crashtest-0.2.1.tar.gz 68.1 kB Without provenance Details for mcp_crashtest-0.2.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for mcp-crashtest 0.2.1
File Interpreter ABI Platform Actions
mcp_crashtest-0.2.1-py3-none-any.whl Python 3 none any Details for mcp_crashtest-0.2.1-py3-none-any.whl

Total release size: 121.8 kB

Release files / mcp_crashtest-0.2.1.tar.gz

Download URL mcp_crashtest-0.2.1.tar.gz
Size 68.1 kB
Tags Source
SHA-256 checksum
How to use checksums
97dc8421e71fe575077fd867d40f4413c91fec58ddc54f31c5c7d91b8cc074d3
BLAKE2b-256 checksum
How to use checksums
88ca54ab7669acdc620e3ab667dd248d7bf1710bd9ffbc26ad69f9ecc0cb75cf
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Release files / mcp_crashtest-0.2.1-py3-none-any.whl

Download URL mcp_crashtest-0.2.1-py3-none-any.whl
Size 53.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f40c656bb2fdde1ac4ca7d048e4cec6cc4446b99f6242753186c2b66d6affda0
BLAKE2b-256 checksum
How to use checksums
f2a8960fcf483223b163aa86af424b5d04f37ba4974dca8463f84c417855ad1e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Security for mcp-crashtest 0.2.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page