toolproof

pytest for MCP servers. Write repeatable tests for any Model Context Protocol server's tools and run them in CI.

• Static checks on every tool definition: missing descriptions, invalid JSON Schemas, duplicate names, required fields that don't exist
• YAML test cases: call a tool and assert on errors, text, regex, JSONPath values, latency and the declared outputSchema
• Crash detection: if the server dies mid-test, toolproof reports it with the server's stderr, restarts the server and keeps going
• Fuzzing: generates valid and invalid inputs from each tool's schema with Hypothesis, finds crashes, hangs and leaked tracebacks, and shrinks each one to the smallest failing input
• Benchmarks: p50/p95/p99 latency, throughput and error rate under concurrency, with thresholds that fail the build
• CI-friendly output: exit code 0/1, JUnit XML and JSON reports
• pytest plugin: an mcp_server fixture for tests written in Python
• Works with any server over stdio or streamable HTTP, in any language

Install

pip install mcp-toolproof

Python 3.11+. The package is called mcp-toolproof on PyPI. The command and the import are both toolproof.

30-second quickstart

Look at a server:

toolproof inspect -- python my_server.py
toolproof inspect --url http://localhost:8000/mcp

Write toolproof.yaml:

server:
  command: ["python", "my_server.py"]

tests:
  - name: toronto weather
    tool: get_weather
    args: { city: Toronto }
    expect:
      is_error: false
      contains: "Toronto"
      jsonpath: { "$.temp_c": { type: number } }
      max_latency_ms: 2000

  - name: rejects empty city
    tool: get_weather
    args: { city: "" }
    expect: { is_error: true }

Run it:

toolproof run toolproof.yaml --junit report.xml

Fuzz and benchmark it, no test cases needed:

toolproof fuzz -- python my_server.py
toolproof bench --tool get_weather --args '{"city": "Toronto"}' -- python my_server.py

What a failing run looks like

examples/buggy_server.py has five planted bugs. Here is toolproof run examples/buggy.yaml:

toolproof - buggy-weather 0.0.1

Static checks
┌──────┬───────────────┬─────────────────┬──────────────────────────────────────────────────────────────────┐
│      │ Tool          │ Check           │ Problem                                                          │
├──────┼───────────────┼─────────────────┼──────────────────────────────────────────────────────────────────┤
│ FAIL │ search_cities │ description     │ tool has no description                                          │
│ FAIL │ search_cities │ input-schema    │ input schema is not valid JSON Schema: 'strng' is not valid      │
│      │               │                 │ under any of the given schemas                                   │
│ FAIL │ lookup        │ required-fields │ required fields not listed in properties: city_id                │
└──────┴───────────────┴─────────────────┴──────────────────────────────────────────────────────────────────┘

Tests
┌──────┬────────────────────────────────────┬─────────────────┬────────┬────────────────────────────────────┐
│      │ Test                               │ Tool            │   Time │ Details                            │
├──────┼────────────────────────────────────┼─────────────────┼────────┼────────────────────────────────────┤
│ PASS │ weather works for a normal city    │ get_weather     │    5ms │                                    │
│ FAIL │ empty city should be an error, not │ get_weather     │   23ms │ no result from server: connection  │
│      │ a crash                            │                 │        │ lost, server probably crashed      │
│      │                                    │                 │        │ (Connection closed)                │
│ PASS │ server still works after the crash │ get_weather     │    3ms │                                    │
│ FAIL │ report comes back in time          │ slow_report     │ 1001ms │ no result from server: timed out   │
│      │                                    │                 │        │ after 1.0s                         │
│ FAIL │ temperature matches its output     │ get_temperature │    2ms │ output schema: $.temp_c: '12.5' is │
│      │ schema                             │                 │        │ not of type 'number'               │
└──────┴────────────────────────────────────┴─────────────────┴────────┴────────────────────────────────────┘

FAILED  tests: 2 passed, 3 failed  |  checks: 3 errors, 0 warnings  |  7.21s

The same run as JUnit XML (--junit report.xml), which GitHub, GitLab and Jenkins show as a normal test report:

<testsuite name="tests" tests="5" failures="3" errors="0" time="1.034">
  <testcase classname="toolproof.get_weather" name="weather works for a normal city" time="0.005" />
  <testcase classname="toolproof.get_weather" name="empty city should be an error, not a crash" time="0.023">
    <failure message="no result from server: connection lost, server probably crashed (Connection closed)">...</failure>
  </testcase>
  ...
</testsuite>

YAML reference

server:
  command: ["python", "server.py"]   # start the server over stdio...
  # url: http://localhost:8000/mcp   # ...or connect over streamable HTTP (exactly one)
  env: { API_KEY: test }             # extra environment variables (stdio only)
  cwd: .                             # working directory, relative to this file (default: this file's folder)
  startup_timeout_s: 30

timeout_ms: 10000        # default per-test timeout
retries: 0               # retry failing tests this many times

checks:
  enabled: true
  max_description_length: 1024
  strict: false          # treat warnings as failures

tests:
  - name: a readable name
    tool: tool_name
    args: { any: json }
    timeout_ms: 2000     # optional override
    retries: 1           # optional override
    expect:
      is_error: false
      equals: { result: 42 }         # whole result: structured content, JSON text, or plain text
      contains: "Toronto"            # or a list: ["Toronto", "cloudy"]
      regex: "\\d+ C"
      jsonpath:
        "$.temp_c": { type: number, min: -50, max: 50 }
        "$.city": { equals: Toronto }
        "$.wind": { exists: false }
      max_latency_ms: 2000
      output_schema: true            # validate against the tool's outputSchema (default on)

bench:                   # optional; `toolproof run` includes it when present
  calls: 100             # per target
  concurrency: 10
  warmup: 3
  timeout_ms: 10000
  thresholds:            # all optional
    p50_ms: 50
    p95_ms: 200
    p99_ms: 500
    max_error_rate: 0.01
    min_throughput: 100  # calls per second
  targets:               # default: every test case that expects success
    - tool: get_weather
      args: { city: Toronto }
      thresholds: { p95_ms: 100 }   # per-target override

fuzz:                    # optional; `toolproof run` includes it when present
  max_examples: 50       # per tool, for valid and for invalid inputs
  timeout_ms: 2000
  max_time_s: 60         # per tool
  seed: 1                # fix it for repeatable CI runs
  tools: []              # only these (default: all)
  skip: [send_email]
  include_destructive: false
  valid_must_succeed: false
  invalid_must_fail: true

Notes:

• jsonpath looks at the tool's structured content if it returned any, otherwise at its text parsed as JSON.
• type is a JSON Schema type: string, number, integer, boolean, array, object, null.
• If a tool declares an outputSchema, every successful call is validated against it automatically.
• Paths in command and cwd are relative to the YAML file, so toolproof run path/to/toolproof.yaml works from anywhere.

Static checks

Errors fail the run. Warnings only fail it with --strict.

Fuzzing

toolproof fuzz calls every tool with inputs generated from its input schema, using Hypothesis and hypothesis-jsonschema:

• valid inputs match the schema, mixed with edge cases that still match it: empty, whitespace, huge (100k characters), emoji, right-to-left text, null bytes, SQL and path-traversal strings
• invalid inputs take a valid input and break it: drop a required field, set a field to null, or give it the wrong type

Each call is judged like this:

Hypothesis shrinks every failure, so you get the smallest input that triggers it. Crashed servers are restarted automatically. Every run prints its seed; pass --seed to repeat it exactly.

Running it on examples/buggy_server.py finds all five planted bugs with no hand-written tests:

Fuzz (seed 3)
┌──────┬─────────────────┬───────┬────────────────────────────┬─────────────────┬───────────────────────────────────────────────┐
│      │ Tool            │ Calls │ Finding                    │ Smallest input  │ Details                                       │
├──────┼─────────────────┼───────┼────────────────────────────┼─────────────────┼───────────────────────────────────────────────┤
│ FAIL │ get_weather     │    22 │ crash (valid)              │ {"city": ""}    │ connection lost, server probably crashed      │
│      │                 │       │ accepted-invalid (invalid) │ {"city": []}    │ invalid input returned a success result       │
│      │                 │       │ crash (invalid)            │ {}              │ connection lost, server probably crashed      │
│ FAIL │ search_cities   │     0 │ bad-schema (valid)         │ {}              │ can't generate inputs: 'strng' is not valid   │
│ FAIL │ slow_report     │     1 │ hang (valid)               │ {"city": ""}    │ timed out after 2.0s                          │
│ FAIL │ get_temperature │    24 │ output-schema (valid)      │ {"city": ""}    │ output schema: $.temp_c: '12.5' is not of     │
│      │                 │       │                            │                 │ type 'number'                                 │
│      │                 │       │ accepted-invalid (invalid) │ {}              │ invalid input returned a success result       │
│ FAIL │ lookup          │    15 │ internal-error (valid)     │ {"city_id": {}} │ JSON-RPC error -32603: Internal server error  │
│      │                 │       │ internal-error (invalid)   │ {}              │ JSON-RPC error -32603: Internal server error  │
└──────┴─────────────────┴───────┴────────────────────────────┴─────────────────┴───────────────────────────────────────────────┘
Repeat this run with --seed 3

Be careful with tools that change things. Fuzzing calls every tool many times with odd arguments. Tools annotated with destructiveHint: true are skipped unless you pass --include-destructive. For anything else that sends email, deletes data or spends money, add it to fuzz.skip or point toolproof at a test instance.

Benchmarks

toolproof bench calls each target calls times with concurrency calls in flight over one connection, after a few warm-up calls, and reports p50/p95/p99 latency, mean and max, calls per second and error rate:

Bench
┌──────┬─────────────────────────────────┬───────┬───────┬────────┬────────┬─────────┬────────┬─────────┐
│      │ Target                          │ Calls │   p50 │    p95 │    p99 │ Calls/s │ Errors │ Details │
├──────┼─────────────────────────────────┼───────┼───────┼────────┼────────┼─────────┼────────┼─────────┤
│ PASS │ toronto weather                 │ 50 x5 │ 8.3ms │ 10.8ms │ 11.9ms │   583.0 │   0.0% │         │
│ PASS │ city names are case insensitive │ 50 x5 │ 8.8ms │ 24.5ms │ 24.9ms │   480.9 │   0.0% │         │
│ PASS │ five day forecast               │ 50 x5 │ 9.3ms │ 20.1ms │ 21.3ms │   477.8 │   0.0% │         │
│ PASS │ list cities                     │ 50 x5 │ 8.1ms │ 10.1ms │ 14.3ms │   600.7 │   0.0% │         │
└──────┴─────────────────────────────────┴───────┴───────┴────────┴────────┴─────────┴────────┴─────────┘

Without --tool or bench.targets, every test case that expects success is benchmarked. A threshold that's broken fails the run, so you can catch a slow tool in CI.

CLI reference

toolproof inspect [--url URL | --config FILE | -- COMMAND...] [--json]
toolproof run [CONFIG] [--junit PATH] [--json PATH] [--timeout-ms N] [--retries N] [--strict] [--no-checks]
              [--skip-bench] [--skip-fuzz]
toolproof fuzz [--url URL | --config FILE | -- COMMAND...] [--examples N] [--timeout-ms N] [--max-time S]
               [--seed N] [--tool NAME]... [--include-destructive] [--valid-must-succeed] [--junit PATH] [--json PATH]
toolproof bench [--url URL | --config FILE | -- COMMAND...] [--tool NAME --args JSON] [--calls N] [--concurrency N]
                [--timeout-ms N] [--p50-ms N] [--p95-ms N] [--p99-ms N] [--max-error-rate R] [--min-throughput N]
                [--junit PATH] [--json PATH]

fuzz, bench and inspect read ./toolproof.yaml when no server is given. CLI flags override the YAML.

Exit codes: 0 all passed, 1 a test, check, threshold or fuzz finding failed, 2 bad config or usage.

pytest plugin

Installing toolproof registers a pytest plugin with an mcp_server fixture. Point it at your server in pyproject.toml:

[tool.pytest.ini_options]
toolproof_command = "python my_server.py"
# toolproof_url = "http://localhost:8000/mcp"
# toolproof_config = "toolproof.yaml"
# toolproof_timeout = "30"

Then write normal tests:

def test_weather(mcp_server):
    r = mcp_server.call("get_weather", city="Toronto")
    assert not r.is_error
    assert r.data["temp_c"] > -50


def test_forecast_length(mcp_server):
    r = mcp_server.call("get_forecast", {"city": "Calgary", "days": 5})
    assert len(r.data["days"]) == 5


def test_schema(mcp_server):
    assert mcp_server.tool("get_weather").input_schema["required"] == ["city"]

call() returns a CallResult with is_error, text, data (structured content or parsed JSON), latency_ms and transport_error. The server starts once per test session.

To set the server up in code, override the mcp_server_config fixture in conftest.py:

import pytest
from pathlib import Path
from toolproof import ServerConfig


@pytest.fixture(scope="session")
def mcp_server_config():
    return ServerConfig(command=["python", "my_server.py"], env={"MODE": "test"}), Path.cwd()

Using it in GitHub Actions

- run: pip install mcp-toolproof
- run: toolproof run toolproof.yaml --junit report.xml
- uses: actions/upload-artifact@v5
  if: always()
  with:
    name: toolproof-report
    path: report.xml

How is this different from MCP Inspector?

MCP Inspector is an interactive debugger: you open a UI, click a tool, type arguments and look at the result. It's great while building a server.

toolproof is for what comes after: tests you write once, keep in the repo and run on every pull request. It has no UI. It gives you assertions, an exit code and a JUnit report, so a broken tool fails the build instead of being found by a user.

Try it on the examples

git clone https://github.com/NilkanthSuthar/toolproof
cd toolproof
pip install -e .
toolproof run examples/toolproof.yaml          # passes: tests, bench and fuzz
toolproof run examples/buggy.yaml              # fails, on purpose
toolproof fuzz -- python examples/buggy_server.py   # finds all five planted bugs

Roadmap

• v0.1: inspect, static checks, YAML tests, console/JUnit/JSON reports, pytest plugin
• v0.2: toolproof fuzz (schema-based fuzzing with Hypothesis) and toolproof bench (p50/p95/p99 latency, throughput)
• v0.3: LLM tool-selection evals, a reusable GitHub Action, an HTML report, docs site

Development

pip install -e ".[dev]"
pytest
ruff check . && ruff format --check .
mypy

The test suite starts the example servers for real, over stdio and HTTP.

License

MIT