mcp-test-harness 2.0.1

MCP Test Harness -- pytest-style testing framework for MCP servers with automated test discovery, assertions, fixtures, and CI reporting.

MCP Test Harness

Release 2.0.0 — install from PyPI (pip install mcp-test-harness) or use the OCI image on GitHub Container Registry: ghcr.io/vaquarkhan/mcp-test-harness. Tags: latest and 2.0.0 (runtime, mcp-test entrypoint), dev and 2.0.0-dev (pytest + dev extras). Browse tags on GHCR · docker run --rm ghcr.io/vaquarkhan/mcp-test-harness:latest --version · docs/DOCKER.md · docs/RELEASING.md

Release 2.0 — what shipped end-to-end

Everything below is implemented, tested, and documented in this repo (690+ tests, 100% lib coverage gate, e2e dogfood in CI).

Area	Feature	Where to start
Diagnostics	MCP trace — per-test JSON-RPC timeline in HTML/JSON reports; stdio pollution hints	example_mcp_trace.md
Resiliency	Chaos testing — @marker(tags=["chaos"], chaos_faults=[...]) (delay, 503, truncate, schema drift)	example_chaos_testing.md
Productivity	mcp-test generate — draft tests from live tools/list + optional drift JSON	example_generate_scaffold.md
Platform QA (v1.2)	Tool coverage map, unified portal in HTML/JSON, security payload packs, resiliency assertions, performance baselines, assert_throughput SLO params (max_p99_ms, max_error_rate)	docs/POSITIONING.md · docs/SECURITY_TESTING.md
CI / security	SARIF export, OWASP MCP rule metadata, PR summary markdown + GitHub Action pr-comment	docs/CI_AND_REPORTS.md
Self-test	E2E dogfood — pytest spawns real mcp-test CLI against bundled FastMCP fixtures; 100% coverage enforced on every PR	We eat our own dogfood
Examples	Runnable platform-qa demo pack (trace + chaos + generate)	feature-demo/platform-qa
Site	GitHub Pages deploys html/ only (marketing site at vaquarkhan.github.io/mcp-test-harness)	html/index.html

Full history: CHANGELOG.md · roadmap: docs/ROADMAP.md.

Author: Vaquar Khan

MCP Test Harness is a pytest-style testing framework for MCP servers. It provides the mcp-test CLI to discover, run, and report on tests automatically, replacing much of the manual validation you might do in the MCP Inspector. Repository: github.com/vaquarkhan/mcp-test-harness.

Documentation: structured hub and adoption paths. Start with QUICK_START, then DEVELOPER_GUIDE, CI & reports, performance / latency tests, performance strategy and product positioning, how we compare to other MCP tools, and ecosystem / registry discovery (release checklist). Community: open an Issue for bugs, a PR for docs and examples.

License: the core project is under the MIT License; see NOTICE. CITATION.cff suggests how to cite the software in papers (optional, not a license condition). (Optional packages under packages/ may list their own terms in each package’s pyproject.toml.)

For CI-native, code-first MCP test automation, MCP Test Harness fills that gap. For spec conformance, LLM-in-the-loop evals, and model benchmarks, other tools exist; see docs/COMPARISON.md.

---

Visual guide

One command from your repo to a gated MCP test run — local or in CI.

Developer journey pip install → mcp-test init → write test_*.py → mcp-test → merge with confidence.

Three testing modes Functional correctness, snapshot regression, and SLO-style performance — not three separate tools.

Under the hood: config → discovery → scheduling → real MCP sessions → deterministic assertions → multi-format reports.

All visual assets

Every diagram lives in docs/images/. Quick reference:

Image	What it shows
hero-banner.png	Project hero — pytest-style MCP testing
end-to-end-flow.png	Full pipeline: CLI → server → assertions → reports
developer-journey.png	Install → init → write tests → run → CI
three-testing-modes.png	Functional, regression, and performance pillars
architecture-flow.svg	Internal modules and data path (vector)
mcp-testobarness-feature.png	Core feature map
assertions-grid.png	Assertion library at a glance
report-formats.png	Console, JUnit, JSON, and HTML outputs
transport-options.png	stdio, SSE, and HTTP transports
parallel-execution.png	Multi-worker scheduling and module grouping
ci-pipeline.png	GitHub Actions PR gate workflow
docker-distribution.png	PyPI, GHCR container, and standalone binary
ecosystem-map.png	Position vs Inspector, conformance, evals, Bastion
harness-bastion-pairing.png	Test in CI, secure in production
testherness.png	Quick-start overview
dogfood-e2e.svg	Dogfood: pytest runs mcp-test against bundled MCP fixtures (690+ tests, 100% lib coverage)

---

We eat our own dogfood

A test harness should prove itself. This repo runs 690+ pytest cases with a 100% line coverage gate on src/mcp_test_harness, plus end-to-end dogfood tests that spawn the real mcp-test CLI against a bundled FastMCP server:

# Same gate CI enforces on every PR
coverage run -m pytest tests/ --ignore=tests/test_workspace.py -q
coverage report --fail-under=100

# E2E dogfood only
python -m pytest tests/test_harness_dogfood_e2e.py -m e2e -v

Fixtures live under tests/fixtures/ (minimal_mcp_server.py, harness_self_test/). See docs/DEVELOPER.md and CONTRIBUTING.md.

---

Why teams adopt this

MCP Test Harness is the deterministic, CI-native gate for MCP servers — one run proves correctness, speed, and security baselines without an LLM in the loop. Full positioning: docs/POSITIONING.md.

Four differentiators

#	Differentiator	What you get
1	Tri-modal CI architecture	Functional + regression + performance in one mcp-test run — assert_latency (p95/p99 + warmup), assert_throughput (min_rps, max_p99_ms, max_error_rate), assert_tool_idempotent, JSON performance baselines
2	Protocol-aware fixtures	mcp_server / mcp_server_session — spawn, handshake, inject session, teardown; parallel module grouping
3	Code-first workflows	Multi-step agent flows in async Python (anti-DSL); @marker(tags=[...]) for smoke/security/perf
4	Unified portal	One HTML/JSON artifact: category scores, tool coverage map (advertised vs tested vs missing auth), security/resiliency packs

Three testing modes in one tool

• Functional: assert_tool_call, assert_resource_read, schema validation
• Regression: assert_snapshot, assert_tool_idempotent
• Performance: assert_latency, assert_throughput, assert_latency_within_baseline

Pair with mcp-shark for IDE config security and MCP-Bastion for runtime protection — see docs/COMPARISON.md.

MCP Test Harness supports Responsible AI and governance programs: schema contract checks, security payload packs (@marker(tags=["security"])), coverage gaps in reports, and EU AI Act demo packs.

Documentation

Hub (table of all guides + suggested reading order): docs/README.md

Document	Contents
docs/QUICK_START.md	Fastest path - install, mcp-test init, run
docs/DEVELOPER_GUIDE.md	Canonical reference - setup, config, stdio/parallel/validation, assertions, reporting
docs/CI_AND_REPORTS.md	CI, JUnit, JSON, HTML - do you need to publish test reports? (usually: no)
docs/POSITIONING.md	Why we're different - four differentiators, enterprise governance, mcp-shark pairing
docs/PERFORMANCE_TESTING_STRATEGY.md	Product pitch - why MCP performance testing belongs in the harness; roadmap and scope
docs/ROADMAP.md	Roadmap - now/next/later plan plus prioritized value bets (unified reports, security packs, resiliency, coverage map, SLO load testing)
docs/SECURITY_TESTING.md	Security testing - MCP-aware security assertions and CI guidance
docs/CONTRACT_AND_COMPAT.md	Contracts & compatibility - drift protection and protocol/client matrix strategy
docs/ENTERPRISE_GOVERNANCE.md	Enterprise - audit/policy/tenant governance guidance (including EU AI Act evidence mapping notes)
docs/PLUGIN_REGISTRY.md	Plugin registry - extension catalog and integration categories
docs/TUTORIAL.md	Step-by-step tutorial
docs/DECISIONS.md	Architecture and product decisions
docs/IMPLEMENTATION_CHECKLIST.md	Maintainer: features vs. code locations
docs/COMPARISON.md	Ecosystem - where this harness fits alongside conformance/eval/benchmark categories
docs/LLM_TEST_GENERATION.md	LLM + tests - draft-with-review: good; auto trusted in CI: bad fit for this harness
docs/COLLECTIONS.md	Postman / Newman–style multi-step flows, “environments”, and roadmap (declarative collections not in core yet)
docs/DISCOVERY.md	Registries and promotion - internal checklist (PyPI, server.json, awesome lists)
docs/DOCKER.md	Docker & OCI - PyPI, GHCR / GitHub Packages links, build targets, docker run
docs/RELEASING.md	Ship v* - PyPI trusted publishing + GHCR images in one tag
docs/ARCHITECTURE.md	Mermaid diagrams: CLI → scheduler → lifecycle → session → tests
docs/EDITORS.md	Visual Studio Code & Cursor - snippets, Mermaid preview, recommended extensions
docs/MARKDOWN_CONVENTIONS.md	Markdown - [!TIP] / **Feature** callouts and fenced code for readable docs
CHANGELOG.md	Release history (Keep a Changelog)
CONTRIBUTING.md	How to contribute (tests, coverage, release bumps)
CITATION.cff	Optional citation - machine-readable metadata (not required by the license)
Dockerfile (and .dockerignore)	Container image for mcp-test - see Docker

For production security (prompt injection defense, PII redaction, rate limiting, RBAC), see MCP-Bastion - the companion security middleware; this repo is for test automation.

Core Features

Feature	Description
Test discovery	Finds test_*.py files and test_ functions automatically (pytest conventions); broken files log a warning with path and exception
MCP assertions	assert_tool_call, assert_resource_read, assert_prompt, assert_capabilities, assert_snapshot, plus assert_tool_schema, assert_protocol_version, assert_tool_idempotent, assert_latency (p95/p99/mean + warmup), assert_throughput (concurrent load + min_rps, max_p99_ms, max_error_rate), assert_latency_within_baseline, security payload packs — see docs/PERFORMANCE.md, docs/SECURITY_TESTING.md
Fixture system	Built-in mcp_server / mcp_server_session, custom fixtures; cycle detection for dependency errors
Schema validation	JSON-RPC envelope checks; with schema_validation: true (default), post-connect checks on initialize, tools/list (+ tool inputSchema), resources / prompts list shapes, and a best-effort call_tool probe to validate content item shapes
Snapshot testing	Compare responses; ignore_fields and mask_patterns for unstable data
Parallel execution	Multiple workers; tests from the same file stay on one worker so per-module fixtures remain correct
Watch mode	mcp-test --watch re-runs when test *.py files change (configurable poll interval; debounce coalesces rapid saves)
Markers	@marker(timeout=60, retry=3, tags=["smoke"]) and @skip(reason="...")
Reports	Console summary, JUnit XML (GitHub Actions/Jenkins/GitLab), JSON/HTML with MCP trace timelines, unified portal / coverage map, SARIF for Code Scanning
Plugin system	Extend with custom assertions, fixtures, reporters, and transport adapters
Transport support	stdio, SSE, streamable HTTP -- test local and remote servers
GitHub Action	One-line CI integration with artifact upload
Docker	Pre-built on ghcr.io/vaquarkhan/mcp-test-harness (:latest / version tags + :dev); local Dockerfile with mcp-test (runtime) or pytest + dev extras via --target dev (see Docker)
Standalone binary	Single binary via PyInstaller, no Python required on target
MCP trace (v1.3)	Per-test JSON-RPC event log + interactive HTML timeline — example_mcp_trace.md
Chaos testing (v1.3)	@marker(tags=["chaos"], chaos_faults=[...]) — delay, 503, truncate, schema drift — example_chaos_testing.md
mcp-test generate (v1.3)	Draft tests from live tools/list + optional drift report — example_generate_scaffold.md

Beginner demo packs by testing type: functional-testing · regression-testing · performance-testing (each includes runnable tests plus JSON/JUnit/HTML report config).
Platform QA (trace + chaos + generate): platform-qa
Full scenario index: examples/feature-demo/README.md

Ecosystem (Conformance, evals, benchmarks)

MCP Test Harness is deterministic (your tests call the protocol directly; no LLM required). The wider MCP space includes protocol conformance suites, agent/LLM evaluation frameworks, and model benchmarks. A concise map of tools, when to use each, and how they complement (not replace) the harness is in docs/COMPARISON.md.

Installation

Current stable version: 2.0.0 (see CHANGELOG.md). Core harness (lightweight: mcp + YAML + anyio; no MCP-Bastion / Presidio stack):

pip install mcp-test-harness
# pin, if you need a fixed version:
# pip install mcp-test-harness==2.0.0

Same release as a container (GHCR, no local Python): docker pull ghcr.io/vaquarkhan/mcp-test-harness:2.0.0 or :latest — see the image section for docker run and dev tags.

Optional mcplint / MCP-Bastion pin helpers (transitive set can be large; same as a full Bastion install):

pip install mcp-test-harness[mcplint]
# or the historical PyPI name for the monorepo shim:
pip install mcplint

Or from source:

git clone https://github.com/vaquarkhan/mcp-test-harness.git
cd mcp-test-harness
python -m venv .venv && source .venv/bin/activate  # or .venv\Scripts\activate on Windows
pip install -e ".[dev]"
mcp-test --version

Docker

One-page guide (PyPI, container registries, Mermaid build diagram, docker run copy-paste): docs/DOCKER.md · System diagram (flow + sequence): docs/ARCHITECTURE.md · Visual Studio Code & Cursor (snippets, Mermaid, extensions): docs/EDITORS.md

Pre-built runtime and dev (test tooling) images are defined in the repo Dockerfile (and .dockerignore keeps the build context small). Each v* git tag triggers CI that pushes ghcr.io/vaquarkhan/mcp-test-harness: e.g. :2.0.0, :latest, :2.0.0-dev, :dev. Quick pull: docker pull ghcr.io/vaquarkhan/mcp-test-harness:latest. All versions (GHCR) · docs/RELEASING.md · docs/DOCKER.md.

Build	Description
Default (runtime)	mcp-test and core dependencies only - smallest image.
--target dev	Adds the same optional packages as pip install -e ".[dev]" (e.g. pytest, jsonschema, PyInstaller). Use this when you want to run the project’s tests/ inside a container.

Build the default image (from the repository root; requires Docker):

docker build -t mcp-test-harness:local .

Smoke the CLI (the image entrypoint is mcp-test):

docker run --rm mcp-test-harness:local --version

Run mcp-test against a project mounted into the working directory (paths below use POSIX shells; on Windows, use PowerShell and replace $PWD with your project path, e.g. ${PWD} in Git Bash, or the full C:\... form):

docker run --rm -v "$PWD":/work -w /work mcp-test-harness:local

The default command shows mcp-test --help. Pass the same arguments you use locally, for example docker run --rm -v "$PWD":/work -w /work mcp-test-harness:local . to discover and run tests in the current config.

Build the dev image and run the test suite (requires your test tree mounted into /work):

docker build -t mcp-test-harness:dev --target dev .
docker run --rm -v "$PWD":/work -w /work --entrypoint pytest mcp-test-harness:dev tests/ -q

For coverage as in pyproject.toml, you can add --cov=src/mcp_test_harness if your tests/ and config are on the mount.

Windows (PowerShell), using Docker Desktop, mount the current directory, for example:

docker run --rm -v "${PWD}:/work" -w /work mcp-test-harness:local

Size and first-build time depend on what you install: the default runtime image matches mcp-bastion-python-free core dependencies. If you add [dev], [mcplint], or pip install mcp-bastion-python in the same environment, the resolver may pull a large transitive set (e.g. Presidio / NLP and, on many Linux x86_64 wheels, very large ML/CUDA-related packages). The first docker build with those extras can take a long time and produce a multi-gigabyte image. That is expected for a full install of the Bastion tree, not a bug in the Dockerfile when you opt into that stack.

Note: The Docker image is optional; many teams use pip install in CI. Use the image when you need a reproducible, Python-isolated environment without a local venv, or a portable mcp-test in pipelines that standardize on containers.

Quick Start

0. Scaffold a starter (optional)

With mcp-test on your PATH (after pip install):

mcp-test init

This writes tests/test_mcp_server_example.py and a minimal mcp-test.yaml. Set your real launch command, for example:

mcp-test init --server-command "python -m your_package.mcp"

Options: mcp-test init --help (custom --dir, --filename, --no-config, --force).

Check the server first (no tests): mcp-test doctor uses the same mcp-test.yaml (or --server-command) to start the server, run the MCP handshake, print the protocol version, list tools / resources / prompts, and optionally run the same post-connect schema checks as a normal run. Exits 0 when healthy, 1 on startup or schema errors. See mcp-test doctor --help.

Editor snippets: the repo includes .vscode/mcp-test-harness.code-snippets - in VS Code or Cursor, type prefixes like mcp-assert-tool or mcp-test-async in a *.py file to insert common patterns.

1. Write a test

Create tests/test_my_server.py:

from mcp_test_harness import assert_tool_call, assert_capabilities

async def test_server_has_tools(mcp_server):
    """Verify the server advertises tool capabilities."""
    await assert_capabilities(mcp_server, {"tools": {}})

async def test_echo_tool(mcp_server):
    """Call the echo tool and check it works."""
    result = await assert_tool_call(mcp_server, "echo", {"message": "hello"})
    assert result is not None

The mcp_server parameter is a built-in fixture. The harness automatically starts your server, connects via MCP, performs the initialize handshake, and injects a ready-to-use session.

2. Run it

mcp-test --server-command "python my_server.py" tests/

Output:

  [PASS] test_server_has_tools (45.2ms)
  [PASS] test_echo_tool (120.8ms)

2 passed, 0 failed, 0 errored, 0 skipped
Total time: 312.5ms

3. Add a config file

Create mcp-test.yaml:

server:
  command: python my_server.py
  transport: stdio

test:
  dirs: [tests/]
  timeout: 30

report:
  format: junit
  output: reports/results.xml

Then just: mcp-test

Assertion Reference

assert_tool_call -- invoke a tool and validate the response

# Basic: fail if the tool returns an error
await assert_tool_call(mcp_server, "echo", {"message": "hello"})

# With expected output
await assert_tool_call(mcp_server, "add", {"a": 1, "b": 2},
    expected=[{"text": "3", "isError": False}])

# Validate arguments against the tool’s inputSchema (requires `jsonschema`)
await assert_tool_call(
    mcp_server, "add", {"a": 1, "b": 2},
    validate_against_input_schema=True,
)

# Use the return value
result = await assert_tool_call(mcp_server, "get_data", {})
assert len(result.content) > 0

Other helpers: assert_tool_schema, assert_protocol_version, assert_tool_idempotent, assert_latency, assert_tool_call_validates_input, assert_tool_denied, assert_authorization_boundary - see Part 3b in the Developer Guide.

assert_resource_read -- read a resource and check content/MIME type

await assert_resource_read(mcp_server, "file:///config.json",
    expected_content='{"debug": true}',
    expected_mime_type="application/json")

assert_prompt -- get a prompt and validate messages

await assert_prompt(mcp_server, "summarize",
    arguments={"text": "The quick brown fox."},
    expected_messages=[{"role": "assistant", "content": "Summary: A fox."}])

assert_capabilities -- verify server capabilities

await assert_capabilities(mcp_server, {"tools": {}, "resources": {}})

assert_snapshot -- regression detection via stored snapshots

from pathlib import Path
from mcp_test_harness import assert_snapshot

async def test_stable_output(mcp_server):
    result = await mcp_server.call_tool("generate_report", {})
    await assert_snapshot(result, "report_output", test_file=Path(__file__))

# Drop volatile fields or mask dynamic strings (regex patterns)
async def test_noisy_output(mcp_server):
    result = await mcp_server.call_tool("with_ids", {})
    await assert_snapshot(
        result,
        "noisy",
        test_file=Path(__file__),
        ignore_fields=["requestId", "timestamp"],
        mask_patterns=[r"req_[a-f0-9]+"],
    )

First run creates the snapshot. Later runs compare against it. Update with mcp-test --update-snapshots.

All assertions produce diff output on failure:

  [FAIL] test_echo (18.5ms)
      Tool 'echo' response mismatch
      --- expected
      +++ actual
      @@ -1,3 +1,3 @@
       [
      -  {"text": "hello", "isError": false}
      +  {"text": "HELLO", "isError": false}
       ]

Fixtures

Built-in fixtures:

Fixture	Scope	Description
mcp_server	Per-test	Fresh MCP session for each test
mcp_server_session	Per-module	Shared session across all tests in a file

Custom fixtures:

from mcp_test_harness.fixtures import fixture, FixtureScope

@fixture
async def api_key():
    return "test-key-12345"

@fixture
async def database():
    db = await connect()
    yield db              # test runs here
    await db.close()      # teardown

@fixture(scope=FixtureScope.PER_MODULE)
async def shared_client():
    client = await create_client()
    yield client
    await client.close()

# Injected by parameter name
async def test_query(mcp_server, database, api_key):
    result = await mcp_server.call_tool("query", {"db": database.url, "key": api_key})

Markers

from mcp_test_harness import marker, skip

@marker(timeout=120)                    # custom timeout
@marker(retry=3)                        # retry on failure
@marker(tags=["smoke", "critical"])     # tags for filtering
@marker(timeout=60, retry=2, tags=["integration"])  # combine

@skip                                   # skip unconditionally
@skip(reason="Bug #42")                # skip with reason

Filter from CLI:

mcp-test -m smoke           # run only smoke-tagged tests
mcp-test -k "test_echo"     # run tests matching name
mcp-test -k "*workflow*"    # glob patterns

Reports

# JUnit XML for CI (GitHub Actions, Jenkins, GitLab)
mcp-test --report-format junit --report-output results.xml

# JSON with full metadata (server capabilities, retry history, schema violations)
mcp-test --report-format json --report-output results.json

Console output is always printed:

  [PASS] test_echo (45.2ms)
  [FAIL] test_divide (18.5ms)
      Division by zero
  [SKIP] test_future (0.0ms)

2 passed, 1 failed, 0 errored, 1 skipped
Total time: 200.0ms

Parallel Execution

mcp-test --parallel              # use all CPU cores
mcp-test --parallel --workers 4  # specify worker count

Each worker gets its own server instance. If one crashes, others continue.

Module grouping: tests from the same file are always scheduled on the same worker, so per-module fixtures (mcp_server_session, etc.) stay valid. Do not rely on test order across different files in parallel mode.

Transport Support

Transport	Use case	Example
stdio	Local servers (default)	--server-command "python server.py"
SSE	Remote servers via Server-Sent Events	--transport sse --server-command "http://localhost:8080/sse"
HTTP	Remote servers via streamable HTTP	--transport http --server-command "http://localhost:8080/mcp"

With authentication:

server:
  command: http://your-server.example.com/mcp
  transport: http
  transport_options:
    headers:
      Authorization: "Bearer your-token"

GitHub Action

# .github/workflows/mcp-tests.yml
name: MCP Server Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Test MCP Server
        uses: vaquarkhan/mcp-test-harness/.github/actions/mcp-test@main
        with:
          server-command: "python my_server.py"
          test-directory: "tests/"
          report-format: "junit"

Input	Default	Description
server-command	""	Command to start the server
transport	stdio	stdio, sse, or http
test-directory	tests/	Path to test files
config-file	""	Path to config file
report-format	junit	json or junit
harness-version	latest	Version to install

Plugins

Extend MCP Test Harness with custom assertions, fixtures, reporters, and transports:

from mcp_test_harness.plugins import PluginContext
from mcp_test_harness.fixtures import FixtureScope

class MyPlugin:
    name = "my-plugin"

    def register(self, context: PluginContext) -> None:
        context.add_assertion("assert_latency", check_latency)
        context.add_fixture("db", db_factory, FixtureScope.PER_MODULE)
        context.add_reporter("markdown", MarkdownReporter())

plugin = MyPlugin()

Load via config:

plugins:
  - my_plugin.py
  - my_package.plugin_module

Or via Python entry points (auto-discovered):

[project.entry-points.mcp_test_harness]
my-plugin = "my_package.plugin:plugin"

See examples/reference_plugin.py for a complete example.

More examples and patterns: examples/README.md and the per-feature checklist examples/FEATURES_INDEX.md (assertions demo, config validation, reports, transports, GitHub Action, Docker, watch mode, …). Copy-paste tests: patterns_mcp_test.md. For working on the harness source tree, use docs/DEVELOPER.md.

Standalone Binary

pip install -e ".[dev]"
python scripts/build_binary.py
dist/mcp-test --version

No Python required on the target machine. Cross-platform: Linux, macOS, Windows.

Security Testing with MCP-Bastion

MCP Test Harness tests that your MCP server works correctly. For production security, pair it with MCP-Bastion -- an active defense middleware that protects MCP servers at runtime.

Concern	Tool	What it does
Functional testing	MCP Test Harness	Automated tests for tools, resources, prompts, capabilities
Prompt injection defense	MCP-Bastion	Blocks jailbreaks via Meta PromptGuard (local, under 5ms)
PII redaction	MCP-Bastion	Masks SSN, email, phone via Microsoft Presidio
Rate limiting	MCP-Bastion	Token budgets, iteration caps, denial-of-wallet protection
RBAC	MCP-Bastion	Tool-level permissions by role
Schema validation	MCP Test Harness	Validates JSON-RPC responses against MCP spec
Regression detection	MCP Test Harness	Snapshot testing catches unintended changes
Audit logging	MCP-Bastion	Logs who, what, when, blocked/allowed

Use both together for a complete MCP server development workflow:

# Test your server
mcp-test --server-command "python my_server.py" tests/

# Secure your server
pip install mcp-bastion-python

# In your server code
from mcp_bastion import MCPBastionMiddleware

bastion = MCPBastionMiddleware(
    enable_prompt_guard=True,
    enable_pii_redaction=True,
    enable_rate_limit=True,
)

MCP-Bastion supports 16+ framework integrations including FastMCP, LangChain, OpenAI, Anthropic, AWS Bedrock, and more. See the MCP-Bastion README for full docs.

Dependency Management (mcplint shim)

The mcplint sub-package pins MCP-Bastion versions and provides helpers:

from mcplint import bastion_version, bedrock_version

print(bastion_version())    # e.g. "1.0.12"
print(bedrock_version())    # None if bedrock extra not installed

Verify: python scripts/verify_upstream.py

CLI Reference

mcp-test [TEST_PATH] [OPTIONS]

  --server-command CMD     Command to start the MCP server
  --transport TYPE         stdio | sse | http (default: stdio)
  --config PATH            Path to mcp-test.yaml or mcp-test.toml
  --timeout SECONDS        Per-test timeout (default: 30)
  --parallel               Run tests in parallel
  --workers N              Parallel worker count (default: CPU count)
  -k PATTERN               Filter by test name
  -m MARKER                Filter by marker/tag
  --list                   List tests and exit
  --watch                  Re-run on test file changes (poll + debounce via env; not with --list)
  --report-format FORMAT   json | junit | html
  --report-output PATH     Report file path
  --verbose                Full server communication logs
  --update-snapshots       Overwrite stored snapshots
  --version                Print version

Exit codes: 0 = passed, 1 = failures, 2 = config error

Configuration Reference

server:
  command: python my_server.py       # required
  transport: stdio                   # stdio | sse | http
  transport_options: {}              # host, port, headers, etc.

test:
  dirs: [tests/]                     # directories to search
  timeout: 30                        # per-test timeout (seconds)
  parallel: false                    # run in parallel
  workers: 4                         # parallel worker count

report:
  format: junit                      # json | junit
  output: reports/results.xml        # output file path

schema_validation: true              # validate JSON-RPC responses; parallel: only worker 0 runs full checks unless true
validate_schema_each_parallel_worker: false  # set true to run post-connect schema on every worker
schema_probe_call_tool: true         # best-effort call first tool with {} to validate result content
plugins: []                          # plugin paths or module names
redact_patterns: []                  # regex patterns to redact from verbose output

Project Structure

mcp-test-harness/
+-- pyproject.toml
+-- CHANGELOG.md                    # version history (Keep a Changelog)
+-- CONTRIBUTING.md                 # how to contribute; links docs hub + tests
+-- server.json                     # MCP registry / tooling metadata (bump with releases)
+-- mcp_test_harness.spec           # PyInstaller config
+-- src/
|   +-- mcplint/                    # dependency shim
|   +-- mcp_test_harness/           # test framework (14 modules)
|       +-- cli.py                  # mcp-test entry point
|       +-- config.py               # YAML/TOML config loading
|       +-- discovery.py            # test file/function discovery
|       +-- executor.py             # test execution, timeout, retry
|       +-- scheduler.py            # sequential + parallel scheduling
|       +-- lifecycle.py            # server start/stop/monitor
|       +-- transport.py            # stdio, SSE, HTTP adapters
|       +-- stdio_mcp.py            # stdio client + process handle
|       +-- assertions.py           # MCP assertion helpers
|       +-- schema.py               # JSON-RPC / MCP schema validation
|       +-- fixtures.py             # fixture manager
|       +-- plugins.py              # plugin registry
|       +-- reporting.py            # console, JSON, JUnit reporters
|       +-- snapshots.py            # snapshot testing
|       +-- parser.py               # JSON-RPC message parser
|       +-- models.py               # shared data models
+-- examples/
|   +-- README.md                  # catalog + per-feature table
|   +-- FEATURES_INDEX.md         # 1:1 map: README core feature -> example
|   +-- example_*.md              # one feature per file (transports, reports, watch, …)
|   +-- mcp_test_*.yaml            # report + transport copy-paste configs
|   +-- basic_usage.py
|   +-- version_gate.py
|   +-- reference_plugin.py         # complete plugin example
|   +-- assertions_async_demo.py    # assert_* with a fake session
|   +-- validate_mcp_test_config.py # YAML/TOML schema check
|   +-- sample_mcp_test.yaml
|   +-- patterns_mcp_test.md        # copy-paste yaml, markers, snapshots
+-- scripts/
|   +-- verify_upstream.py
|   +-- build_binary.py
+-- tests/                          # 690+ tests; 100% line gate + e2e dogfood (see [docs/DEVELOPER.md](docs/DEVELOPER.md#end-to-end-dogfood))
+-- docs/
|   +-- README.md                   # documentation hub
|   +-- index.md                    # short landing (e.g. GitHub Pages)
|   +-- DISCOVERY.md                # registries / release promotion checklist
|   +-- DEVELOPER_GUIDE.md          # complete API and integration guide
|   +-- TUTORIAL.md                 # step-by-step tutorial
|   +-- DECISIONS.md                # architecture decisions
+-- .github/
    +-- actions/mcp-test/           # reusable GitHub Action
    +-- workflows/validate.yml      # CI pipeline

Testing

# Run all MCP Test Harness tests (pythonpath=src is set in pyproject.toml for pytest)
python -m pytest tests/ -q

# Quick offline check (no heavy deps)
python -m pytest tests/test_pyproject.py -q

# With coverage
python -m coverage run -m pytest tests/ -q
python -m coverage report --show-missing

The repo enforces 100% line coverage on src/mcp_test_harness via coverage report --fail-under=100 in CI, plus e2e dogfood tests that run the real mcp-test CLI against bundled MCP fixtures. See We eat our own dogfood and docs/DEVELOPER.md.

If imports resolve to a different installed copy of the package, run from the repo root so src/ is used, or: pip install -e ".[dev]".

Troubleshooting

Problem	Fix
mcp-test: command not found	Run pip install -e ".[dev]"
Tests hang	Check --timeout; server may not respond to MCP handshake
No tests discovered	Files must match test_*.py or *_test.py; functions must start with test_. Check logs: a warning is emitted if a test file fails to import
Snapshot mismatch	Run mcp-test --update-snapshots after intentional changes
Server crashes during tests	Check server logs; harness marks remaining tests as errored
Config file not found	Harness looks for mcp-test.yaml / mcp-test.toml in cwd, or use --config

Framework Integration Packages

MCP Test Harness provides framework-specific testing helpers. Each package auto-installs mcp-test-harness as a dependency:

Package	Tests for	Version	Downloads
mcp-test-harness	Any MCP server (core)
mcp-test-harness-fastmcp	FastMCP servers
mcp-test-harness-openai	OpenAI function calling
mcp-test-harness-anthropic	Anthropic Claude tool use
mcp-test-harness-bedrock	AWS Bedrock agents
mcp-test-harness-gemini	Google Gemini
mcp-test-harness-langchain	LangChain MCP tools
mcp-test-harness-crewai	CrewAI agents
mcp-test-harness-llamaindex	LlamaIndex tools
mcp-test-harness-groq	Groq inference
mcp-test-harness-mistral	Mistral AI
mcp-test-harness-cohere	Cohere
mcp-test-harness-azure	Azure OpenAI
mcp-test-harness-vertexai	Google Vertex AI
mcp-test-harness-huggingface	Hugging Face Inference
mcp-test-harness-deepseek	DeepSeek AI
mcp-test-harness-together	Together AI
mcp-test-harness-fireworks	Fireworks AI

Note: for optional security-oriented version checks in CI, install mcp-test-harness[mcplint] (or mcplint) to include mcp-bastion-python helpers such as bastion_version().

Related Projects

Project	Purpose
MCP-Bastion	Security middleware for MCP servers (prompt injection, PII, rate limiting, RBAC)
MCP Python SDK	Official Python SDK for building MCP servers and clients
MCP Inspector	Visual debugging tool for MCP servers (manual, browser-based)

Third-party testing and evaluation tools (e.g. official conformance, agent-centric evals, model benchmarks) are mapped in docs/COMPARISON.md so you can pick the right tool for the job.

License and citation

The mcp-test-harness core is distributed under the MIT License - see the LICENSE file in this repository. That includes commercial use, modification, and distribution, subject to preserving the copyright and license notice.

Citing the project (optional): the CITATION.cff file provides metadata for academic or technical citations; it is not a legal requirement of the license.

Optional sub-packages under packages/ may specify different license metadata in their own pyproject.toml files.

Author: Vaquar Khan