toolproof

Automated tests for Model Context Protocol servers. Check tool definitions, write YAML or pytest tests, fuzz every tool from its schema, benchmark latency, check that models pick the right tools, and fail the build when something breaks.

pip install mcp-toolproof
toolproof fuzz -- python my_server.py

Documentation · Getting started · Configuration · CLI · Changelog

Features

• Static checks: every tool needs a name, a description, a valid JSON Schema and required fields that exist
• YAML tests: assert on errors, text, regex, JSONPath values, latency and the tool's own outputSchema
• Fuzzing: valid and invalid inputs generated from each tool's schema with Hypothesis; crashes, hangs, internal errors and leaked tracebacks, each shrunk to the smallest failing input
• Benchmarks: p50/p95/p99 latency, throughput and error rate under concurrency, with thresholds
• Tool-selection evals: give a model your tool list and check it picks the right tool and arguments for real requests; Anthropic, OpenAI-compatible or your own program, with cached answers
• pytest plugin: an mcp_server fixture for tests written in Python
• Built for CI: a GitHub Action, exit codes, JUnit, JSON and single-file HTML reports, a job summary, repeatable fuzz seeds, ${VAR} secrets
• Any server: stdio or streamable HTTP, written in Python, TypeScript, Go or anything else; crashed servers are restarted and their stderr is shown

Quick start

See what a server offers and whether its tool definitions are sound:

toolproof inspect -- python my_server.py
toolproof inspect --url http://localhost:8000/mcp

Write tests in toolproof.yaml:

server:
  command: ["python", "my_server.py"]

tests:
  - name: toronto weather
    tool: get_weather
    args: { city: Toronto }
    expect:
      is_error: false
      jsonpath: { "$.temp_c": { type: number, min: -60, max: 60 } }
      max_latency_ms: 2000

  - name: rejects empty city
    tool: get_weather
    args: { city: "" }
    expect: { is_error: true }

bench:
  thresholds: { p95_ms: 200 }

fuzz:
  seed: 1

toolproof run toolproof.yaml --junit report.xml

Check that a model picks the right tool (pip install "mcp-toolproof[anthropic]" or [openai]):

evals:
  provider: anthropic
  model: your-model-name
  cases:
    - prompt: Is it raining in Montreal right now?
      expect_tool: get_weather
      expect_args: { city: Montreal }

toolproof eval -c toolproof.yaml

In GitHub Actions:

- uses: actions/setup-python@v7
  with:
    python-version: "3.12"
- run: pip install -e .              # your server and its dependencies
- uses: NilkanthSuthar/toolproof@v0.3.1
  with:
    config: toolproof.yaml
    python-version: ""               # run in the environment set up above

Or in Python, with the pytest plugin:

def test_weather(mcp_server):
    r = mcp_server.call("get_weather", city="Toronto")
    assert not r.is_error
    assert r.data["temp_c"] > -60

Fuzzing finds bugs without writing tests

examples/buggy_server.py has five planted bugs. toolproof fuzz finds all of them on its own:

$ toolproof fuzz --seed 3 -- python examples/buggy_server.py

Fuzz (seed 3)
┌──────┬─────────────────┬───────┬────────────────────────────┬─────────────────┬─────────────────────────────────────────────┐
│      │ Tool            │ Calls │ Finding                    │ Smallest input  │ Details                                     │
├──────┼─────────────────┼───────┼────────────────────────────┼─────────────────┼─────────────────────────────────────────────┤
│ FAIL │ get_weather     │    21 │ crash (valid)              │ {"city": ""}    │ connection lost, server probably crashed    │
│      │                 │       │                            │                 │ (Connection closed)                         │
│      │                 │       │ accepted-invalid (invalid) │ {"city": []}    │ invalid input returned a success result     │
│      │                 │       │ crash (invalid)            │ {}              │ connection lost, server probably crashed    │
│      │                 │       │                            │                 │ (Connection closed)                         │
│ FAIL │ search_cities   │     0 │ bad-schema (valid)         │ {}              │ can't generate inputs: 'strng' is not valid │
│      │                 │       │                            │                 │ under any of the given schemas              │
│ FAIL │ slow_report     │     1 │ hang (valid)               │ {"city": ""}    │ timed out after 1.0s                        │
│ FAIL │ get_temperature │    24 │ output-schema (valid)      │ {"city": ""}    │ output schema: $.temp_c: '12.5' is not of   │
│      │                 │       │                            │                 │ type 'number'                               │
│      │                 │       │ accepted-invalid (invalid) │ {}              │ invalid input returned a success result     │
│ FAIL │ lookup          │    15 │ internal-error (valid)     │ {"city_id": {}} │ JSON-RPC error -32603: Internal server      │
│      │                 │       │                            │                 │ error                                       │
│      │                 │       │ internal-error (invalid)   │ {}              │ JSON-RPC error -32603: Internal server      │
│      │                 │       │                            │                 │ error                                       │
└──────┴─────────────────┴───────┴────────────────────────────┴─────────────────┴─────────────────────────────────────────────┘
Repeat this run with --seed 3

FAILED  |  fuzz: 9 findings in 5 tools  |  24.33s

Warning: fuzzing calls your tools for real, many times, with odd arguments. Point it at a test instance, and skip tools that send, delete or spend. See the fuzzing guide.

Tested on

toolproof is run against the official MCP reference servers: everything, filesystem, git, memory, sequential-thinking, time and fetch. In October 2026 that was about 3,000 fuzz calls, 15 security boundary tests and benchmarks. All of them held up; the reference servers validate their input well.

How it compares

MCP Inspector is for exploring a server by hand; toolproof is for tests that run unattended in CI. Other projects cover declarative YAML tests, snapshot regression gates, agent evaluations and security scanning. toolproof's focus is behaviour testing with schema-driven fuzzing and benchmarks, offline and without an LLM; only the optional evals use a model. See the comparison for details.

Requirements

• Python 3.11 to 3.14 on Linux, macOS or Windows
• The server can be written in any language

Contributing

Bug reports, fixes and new checks are welcome. See CONTRIBUTING.md for the development setup, and SECURITY.md to report a security problem privately.

License

MIT