pytest-mcp-honesty

Bidirectional contract testing for FastMCP servers, in your own pytest suite.

A tool's schema is a contract, and servers break it in both directions:

• The schema promises less than the implementation demands. A param is str | None = None in the signature — so the generated schema calls it optional — but the handler errors without it. Every agent that trusts the schema burns a failed call to learn the truth. (Found in the wild: a server whose resolve_team declared required: [] and then errored "provide club_name or org_id".)
• The implementation enforces less than the schema promises. Required params can be omitted, or structurally wrong values are accepted, and garbage flows into your handler.

check_contracts probes every tool over FastMCP's in-memory client — no network, no auth, hermetic — so a failed probe is a finding about your contract, not your environment:

from pytest_mcp_honesty import check_contracts
from my_app import server  # your FastMCP instance

async def test_tool_contracts():
    report = await check_contracts(server)
    assert not report.violations, report.summary()

Three probe families, all deterministic (same schema → same probe):

The wrong-value probe deliberately avoids values Pydantic's lax coercion legally repairs ("5" → 5 is not a violation); it sends values nothing can coerce.

What this is not

• Not protocol conformance — the transport/spec layer is a different job (see the official MCP Inspector, or mcp-java-testkit on the JVM).
• Not a fuzzer — probes are deterministic transcriptions of the contract, not random inputs (see mcp-fuzzer for security-style fuzzing).
• Not schema portability — whether your schema survives specific clients is mcp-portability-lint's job; this checks whether your schema tells the truth about your own server.

Development

Maintained by one person with AI assistance; where AI tooling contributed is logged per commit in docs/ai-maintenance-log.md.

License

Apache-2.0