MCP-CrashTest
Unit tests check single tool calls. Real agents fail on multi-step contracts.
State-aware random walks with POSIX inode-preserving rollback for Model Context Protocol (MCP) servers.
Why MCP-CrashTest?
Tools like MCP Inspector and standard unit tests validate single-step static calls. In production, AI agents execute 5–20 step tool chains where outputs from discovery/list tools feed downstream mutations, entities collide, and stdio pipes hang.
Inspired by reversing the state-aware random walk paradigm from ToolSpace (arXiv:2611.05036)—instead of keeping successful trajectories for model training and discarding rollbacks—MCP-CrashTest mines a dependency graph from tools/list, runs guided multi-step random walks inside ephemeral OS sandboxes, rolls back on failure to explore neighboring branches, prunes noisy intermediate steps, and outputs deduplicated minimal JSON-RPC reproducers.
What It Catches in Real MCP Servers
-
Masked Errors (
isError: falseon SQL/Runtime Failures) — Protocol Violation (CRITICAL)
Inmcp-server-sqlite(verified on 2026-11-20 withMCP-CrashTest v0.2.1in both--adapter nonepure zero-shot mode and--adapter auto), SQL errors returnisError: false, causing agents to treat error strings as valid query results (context poisoning):{ "method": "tools/call", "params": { "name": "write_query", "arguments": { "query": "INSERT INTO users (id, name, email) VALUES (1, 'CollisionBob', 'bob_new@example.com');" } }, "response": { "text": "Database error: UNIQUE constraint failed: users.id", "isError": false } }
-
CLI Output Leaks Across Tools — Cross-Tool Contract Break (
HIGH)
Inmcp-server-git,git_branchoutputs human CLI formatting ("* main"). Passing that branch name directly togit_checkout,git_diff,git_show, orgit_mergefails:[ { "tool": "git_branch", "response": { "text": "* main", "isError": false } }, { "tool": "git_merge", "arguments": { "source_branch": "* main" }, "response": { "text": "Error executing tool git_merge: Ref '* main' did not resolve to an object", "isError": true } } ]
-
Silent State Corruption & Stdio Deadlocks — Protocol Violations (
CRITICAL)
Read-after-write verification catches tools that reportisError: falsewithout persisting state, as well as hungstdiobuffers or severed child-process pipes (with automatic subprocess recycling). -
Agent Contract Smells — UX & Consistency Pitfalls (
MEDIUM/LOW/INFO)
Explicitly separated from protocol bugs: undocumented glob-onlysearch_filesmatching, relative vs. parent-joined path contracts, no-upsert skip behavior (create_entities), and intra-serverisErrorinconsistencies.
Quick Start
Requires Python >= 3.10.
1. Run Immediately via uvx or pip
# Run directly without cloning (once published on PyPI)
uvx mcp-crashtest --target sqlite --walks 15 --depth 6
# Or install from source
git clone https://github.com/SnowyLumiere/mcp-crashtest.git
cd mcp-crashtest
pip install -e ".[dev]"
2. Test Any Custom MCP Server (Zero-Shot Generic Mode)
Use -- to pass the server command cleanly without option collisions, and --adapter none to run pure zero-shot generic stateful fuzzing with zero domain-specific assumptions:
mcp-crashtest \
--target my-server \
--adapter none \
--sandbox directory \
--seed 42 \
--walks 20 \
--depth 6 \
--fail-on-crash \
--output reports/my_server_report.md \
-- python3 -m my_mcp_server --root {sandbox}
For stateless/API-only servers, use --sandbox none (context-memory rollback and branch exploration still run):
mcp-crashtest \
--target my-api-server \
--adapter none \
--sandbox none \
--walks 20 \
-- node dist/index.js --env test
3. Built-in Official Server Presets
Requires npx (filesystem / memory) or uvx + git (sqlite / git):
mcp-crashtest --target sqlite --walks 15 --depth 6 --output reports/sqlite_crash_report.md
mcp-crashtest --target git --walks 15 --depth 6 --output reports/git_crash_report.md
mcp-crashtest --target filesystem --walks 15 --depth 6 --output reports/filesystem_crash_report.md
mcp-crashtest --target memory --walks 15 --depth 6 --output reports/memory_crash_report.md
Architecture: Generic Engine vs. Domain Adapters
MCP-CrashTest separates the generic zero-shot stateful engine from optional domain contract adapters (mcp_crashtest/adapters/):
| Capability | Generic Mode (--adapter none) |
Built-in Domain Adapters (--adapter auto) |
|---|---|---|
| Schema Graph Mining | Infers CREATE/READ/MUTATE/DELETE + DATA/STATE/RETRY edges from tools/list; handles Pydantic v2 anyOf/oneOf/enum and properties: null |
Adds domain-specific edge bindings for filesystem, sqlite, memory, git |
| Context Flow & Fuzzing | Extracts IDs/keys/URIs/paths from args & JSON responses; injects boundary strings, traversal paths, extreme ints | Adds domain-specific probes (e.g. PK collisions, glob-vs-substring queries, raw * main branch passing) |
| Post-Condition Probes | Probe A (RETRY_TRAP), Generic Probe B (Read-After-Write on id/key/uri), Probe C (SCHEMA_BREAK), Probe D (STDIO_DEADLOCK) |
Adds calibrated CONTRACT_SMELL checks (e.g. search_files glob pitfall, describe_table [], create_entities no-upsert skip) |
| Report Provenance | Every finding is tagged with source: generic |
Every adapter finding is explicitly tagged source: adapter:<name> |
Sandbox Placeholders & Environment Injection
--sandbox |
CLI Placeholders | Injected Env Var | Seeded Initial Data |
|---|---|---|---|
directory (default) |
{sandbox}, {work_dir} |
MCP_SANDBOX_DIR |
sample.txt, data_dir/config.json, data_dir/nested_note.txt |
git |
{sandbox}, {work_dir} |
MCP_SANDBOX_DIR |
Initialized main branch + initial commit (commit.gpgsign=false) |
sqlite |
{sandbox}, {db_path} |
MCP_SANDBOX_DB_PATH |
test_target.db with seeded users table (2 rows) |
memory |
{sandbox}, {memory_path} |
MEMORY_FILE_PATH |
Isolated temp path memory_graph.json |
none |
— | — | No local filesystem sandbox (context rollback still active) |
Live Audit Summary (v0.2.1 · 4 Official MCP Servers)
Verified on 2026-11-20 (--walks 15 --depth 6 --rollbacks 3 --seed 42, Python 3.12, mcp SDK v2.3.0 client with full v1.x/v2.x compatibility; pruned reports in reports/):
| Server | Mode | Protocol Violations | Contract Smells | Headline Minimal Reproducers | Report |
|---|---|---|---|---|---|
mcp-server-sqlite |
--adapter none (pure zero-shot) |
3 (CRITICAL) |
0 |
write_query, create_table, read_query return "Error: ..." with isError: false (1-step each, source: generic) |
— |
mcp-server-sqlite |
--adapter auto |
3 (CRITICAL) |
1 (MEDIUM) |
SQL errors return isError: false (×3, 1-step); describe_table returns [] on missing table (1-step) |
sqlite |
mcp-server-git |
--adapter auto |
0 |
4 (HIGH SCHEMA_BREAK) |
git_branch leaks '* main', breaking git_checkout, git_diff, git_show, git_merge (2-step each) |
git |
@modelcontextprotocol/server-filesystem |
--adapter auto |
0 |
4 (MEDIUM) |
list_directory bare filenames vs read_file/read_text_file/get_file_info (2-step); search_files glob-only miss (1-step) |
filesystem |
@modelcontextprotocol/server-memory |
--adapter auto |
0 |
3 (LOW / INFO) |
create_entities skips existing entities with isError: false (2-step); delete_* idempotent 0 differs from add_* (1-step) |
memory |
Notes:
- The
sqlitepreset launchesuvx --with "mcp<2" mcp-server-sqlitebecause upstreammcp-server-sqlitecurrently uses themcp 1.x@server.list_resources()API and fails to boot undermcp 2.x.MCP-CrashTestitself supports bothmcpSDK v1.x (inputSchema/isError) and v2.x (input_schema/is_error).- reports/MCP_CRASHTEST_COMPREHENSIVE_DIAGNOSIS_20261106.md is preserved as the historical v0.2.0 archive.
4 Contract Rules for Agent-Safe MCP Servers
- Strict
isErrorSemantics: Never returnisError: falseon SQL errors, constraint violations, or failed mutations. - Cross-Tool I/O Symmetry: Discovery/list tool outputs must be directly consumable by downstream tools without CLI decorations (
* main) or stripped parent paths. - Explicit Matching & Upsert Contracts: Document glob vs. substring matching and provide explicit upsert/conflict errors instead of ambiguous
isError: falseskips. - Actionable Errors & Pipe Backpressure: Return offending parameter names and suggested corrective tools; paginate large responses to prevent
stdiodeadlocks.
CLI Reference
| Flag | Default | Description |
|---|---|---|
--target |
"custom" |
Target label or built-in preset (filesystem, sqlite, memory, git) |
--adapter |
"auto" |
Domain adapter mode: auto, none (pure generic), or explicit adapter name(s) |
--cmd / -- <cmd...> |
None |
Server executable (prefer POSIX -- passthrough to avoid flag clashes) |
--sandbox |
"directory" |
directory | sqlite | git | memory | none |
--walks |
15 |
Number of random walk chains |
--depth |
6 |
Max step depth per walk |
--rollbacks |
3 |
Max rollback retries per walk |
--fuzz-ratio |
0.25 |
Boundary fuzzing probability (0.0–1.0) |
--timeout |
8.0 |
Per-step & tools/list timeout in seconds |
--init-timeout |
20.0 |
Session initialize() timeout in seconds |
--seed |
None |
Local deterministic RNG seed (does not mutate global random state) |
--lang |
"en" |
Report language: en | zh |
--fail-on-crash |
False |
Exit with code 1 if findings matching --fail-on are detected (for CI gates) |
--fail-on |
"any" |
CI failure scope: any (all findings) | protocol (PROTOCOL_VIOLATION only) |
--output |
"reports/mcp_crash_report.md" |
Output Markdown report path |
--version |
— | Print version and exit |
Known Limitations & Roadmap
- Transport: Currently supports
stdiotransport only. Streamable HTTP / SSE support is on the roadmap. - SQLite Sandbox: In-place binary file rollback assumes per-call SQLite connections (such as
mcp-server-sqlite). Long-lived connection pools with in-memory page caches require session restarts between walks. - Process Memory: For servers keeping unpersisted in-process state, sandbox rollback restores disk state only.
Python API & Running Tests
import asyncio
from mcp_crashtest.cli import run_crash_test
async def main():
crashes = await run_crash_test(
cmd="python3",
args=["my_server.py", "--root", "{sandbox}"],
target_name="my-server",
adapter_mode="none",
sandbox_type="directory",
seed=42,
lang="en",
output_report_path="reports/my_server.md",
)
print(f"Hits: {len(crashes)} | Unique: {len({c.fingerprint for c in crashes})}")
asyncio.run(main())
Run the test suite (zero third-party dependencies required for unittest):
PYTHONPATH=. python3 -m unittest discover -s tests -v
# or with pytest
PYTHONPATH=. pytest tests/ -v
License
Released under the MIT License.
Metadata
Release files for mcp-crashtest 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| mcp_crashtest-0.2.1.tar.gz | 68.1 kB | Without provenance | Details for mcp_crashtest-0.2.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| mcp_crashtest-0.2.1-py3-none-any.whl | Python 3 | none | any | Details for mcp_crashtest-0.2.1-py3-none-any.whl |
Total release size: 121.8 kB
Release files / mcp_crashtest-0.2.1.tar.gz
| Download URL | mcp_crashtest-0.2.1.tar.gz |
|---|---|
| Size | 68.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
97dc8421e71fe575077fd867d40f4413c91fec58ddc54f31c5c7d91b8cc074d3
|
|
BLAKE2b-256 checksum How to use checksums |
88ca54ab7669acdc620e3ab667dd248d7bf1710bd9ffbc26ad69f9ecc0cb75cf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.16
|
Release files / mcp_crashtest-0.2.1-py3-none-any.whl
| Download URL | mcp_crashtest-0.2.1-py3-none-any.whl |
|---|---|
| Size | 53.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f40c656bb2fdde1ac4ca7d048e4cec6cc4446b99f6242753186c2b66d6affda0
|
|
BLAKE2b-256 checksum How to use checksums |
f2a8960fcf483223b163aa86af424b5d04f37ba4974dca8463f84c417855ad1e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.16
|
Security for mcp-crashtest 0.2.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.