MCP Verdict

Test how your MCP servers behave when things go wrong.

MCP Verdict is an open-source Python CLI for testing failure handling and selected security behavior in Model Context Protocol (MCP) servers. It runs repeatable YAML-defined checks against servers you own or are authorized to assess.

MCP Verdict is a test runner, not a general MCP client or a comprehensive security scanner. It does not provide full OAuth/JWT auditing, compliance certification, sandboxing, or remote target discovery.

Why MCP Verdict?

MCP tools can fail in ways that are difficult to exercise repeatedly: a server may return success while reporting a failure, omit an expected tool error, or handle authentication and authorization boundaries incorrectly. These behaviors can regress as a server changes.

MCP Verdict makes those cases explicit in version-controlled configuration. Instead of maintaining one-off failure-testing scripts, developers can run the same local STDIO or Streamable HTTP scenarios in development and CI.

Features

• STDIO and Streamable HTTP MCP connections.
• MCP tool outcome assertions based on the protocol isError field.
• HTTP authentication and authorization status assertions, including 401 and 403 expectations.
• YAML-defined single or multiple checks with per-check timeouts.
• PASS, FAIL, and ERROR results with CI-friendly exit codes.
• Human-readable terminal output and schema-versioned JSON reports.
• Atomic report files and bounded diagnostic redaction.

Quick start

Requirements: Python 3.11 or 3.12. From the repository root:

python -m pip install -e .
mcp-verdict check --config examples/basic_stdio/verdict.yaml

The Basic STDIO example uses a local FastMCP server and needs no credentials or network access. Its output is:

ping-success: PASS: Expected and observed success
add-success: PASS: Expected and observed success
Summary: 2 passed, 0 failed, 0 errored

For development and tests, install the test tools too:

python -m pip install -e '.[test]'

Configuration

Existing configurations default to STDIO. A minimal check is:

server:
  command: [python, examples/basic_stdio/server.py]
tool:
  name: ping
  arguments: {}
expected: success

server.command starts the local MCP server, tool.name selects a tool, arguments supplies its JSON-compatible arguments, and expected is either success or tool_error.

See Configuration for multiple checks, HTTP servers, timeouts, authentication environment variables, and validation rules.

Results and exit codes

The process exits with 0 when all checks pass, 1 when at least one check fails without execution errors, and 2 when any check errors. See Reporting for terminal output, JSON schema version 1, and atomic report files.

Documentation

• Configuration guide
• STDIO and Streamable HTTP transports
• Security testing
• Reporting and JSON schema
• Architecture guide
• Runnable examples
• Contributing
• Security policy
• Changelog

Project status

The current development version is 0.1.0; initial release preparation is in progress. The implemented checks are intended for repeatable local and authorized testing, not comprehensive security auditing or compliance certification.

Contributing

Bug reports, focused test scenarios, and documentation improvements are welcome. Start with CONTRIBUTING.md for setup, quality checks, and fixture conventions.

Security

Test only servers you own or are authorized to assess. STDIO YAML configurations can execute local commands with the current user's permissions; never run an untrusted YAML file. Keep credentials in environment variables and treat diagnostic output carefully. Read the security policy and the security-testing guide before testing protected servers.

License

MIT License

Disclaimer

MCP Verdict is an independent open-source project and is not affiliated with or endorsed by the Model Context Protocol maintainers.