mcpevalkit 0.1.0

Evaluation tooling for Model Context Protocol (MCP) servers

mcpevalkit

The eval kit for MCP-based agents. Record real tool-call trajectories, replay them deterministically against mocked servers, and score step-level — not just final answers.

$ make demo
mcpevalkit demo — coding agent demo
====================================================
  PASS  write-greeting
  PASS  write-notes-with-content
  PASS  read-sample
  PASS  list-workspace
  PASS  confirm-write-response
  FAIL  expects-search-but-lists
        tool_called: 'search_files' was never called (calls: ['list_directory'])
  FAIL  expects-different-content
        tool_called_with: no call to 'write_file' matched args {'content': 'beta'}
----------------------------------------------------
  5/7 passed (71%)
  (2 deliberate failure demo(s) shown above)

🚧 Status: v0.1 in active development. The public API is unstable and may change without notice until the first tagged release. Not yet published to PyPI.

---

Why mcpevalkit exists

If your agent speaks MCP, the tool trajectory is the behavior you need to test.

MCP-based agents fail differently. They fail by picking the wrong tool, by calling it with bad parameters, by retrying in a loop, by giving up too early, by recovering well from an error in step 3 but missing the implication in step 7. A final-answer score can't see any of that.

Most eval tooling can judge the final answer. mcpevalkit is narrower: it records and replays MCP tool-call trajectories so you can catch agent regressions in CI before they reach production.

It is opinionated about three things:

1. MCP is a first-class object. Not an adapter, not a plugin — the recorder speaks MCP natively over stdio and SSE.
2. Evals should be deterministic and cheap. Record real trajectories once. Replay them against mocked MCP servers forever. CI runs in seconds, not dollars.
3. Trajectory-level scoring is the point. Judges for tool selection, parameter validity, and retry loops — not just a single rubric over the final response.

---

Who it's for

mcpevalkit is for teams building MCP-based agents who need to answer practical release questions:

• Did the new prompt, model, or tool schema change the agent's tool-use behavior?
• Did the agent call the right tool with the right arguments?
• Did it recover from tool errors, or did it spin in a retry loop?
• Can this behavior be tested in pytest without hitting live services on every CI run?

It is also a lightweight consulting artifact: clone it into a client MCP-agent repo, define the expected tool behavior in YAML or pytest, record a known-good trajectory, and turn agent reliability into a repeatable release check.

mcpevalkit is not a dashboard, hosted observability product, generic RAG benchmark, or framework adapter layer. The project intentionally stays small: MCP trajectory evals, deterministic replay, and test-friendly reporting.

---

Installation

pip install mcpevalkit

Requires Python 3.11+ and an MCP-compatible agent. Works with any LLM provider — bring your own.

Status: v0.1 is in active development and the API may shift. Pin to a specific version in production.

---

Quickstart

Your agent is any async callable that takes a recorder and a prompt and drives MCP tools through recorder.session(...):

# my_agent.py
from mcpevalkit.recorder import McpServerConfig, Recorder

# Servers for the `mcpevalkit record` CLI (the eval suite declares its own).
mcp_servers = {
    "filesystem": McpServerConfig(
        transport="stdio",
        command="npx",
        args=("-y", "@modelcontextprotocol/server-filesystem", "/tmp/sandbox"),
    )
}

async def agent(recorder: Recorder, prompt: str) -> str:
    await recorder.session("filesystem").call_tool(
        "write_file", {"path": "/tmp/sandbox/hello.txt", "content": "hello world"}
    )
    return "done"

Record a real run of your agent against an MCP server — note the :agent attribute on the target:

mcpevalkit record \
  --agent ./my_agent.py:agent \
  --prompt "Write 'hello world' to /tmp/sandbox/hello.txt" \
  --output traces/baseline.jsonl

Define an eval suite:

# .mcpeval.yaml
suite: coding-agent-baseline

mcp_servers:
  filesystem:
    transport: stdio
    command: npx
    args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp/sandbox"]

evals:
  - id: write-file-correctly
    prompt: "Write 'hello world' to /tmp/sandbox/hello.txt"
    expects:
      - tool_called: write_file
      - tool_called_with:
          tool: write_file
          args:
            content: "hello world"
      - final_state_matches:
          kind: file_equals
          path: /tmp/sandbox/hello.txt
          contains: "hello world"
      - no_retry_loop: true

Run it (the agent under test is supplied with --agent):

mcpevalkit run --suite .mcpeval.yaml --agent ./my_agent.py:agent

In pytest

Zero-config: drop a .mcpeval.yaml next to an agent.py that exposes an agent callable, then run pytest with the plugin flag. Each eval becomes its own test, and failures print the trajectory step by step:

pytest --mcpevalkit

Fixtures: for hand-written tests, mcp_agent_run runs an agent and returns the recorded Trace to assert on:

# test_agent.py
from my_agent import agent, mcp_servers
from mcpevalkit.trace import ToolCall

async def test_writes_file(mcp_agent_run):
    trace = await mcp_agent_run(agent, "write hello", servers=mcp_servers)
    calls = [e.tool for e in trace if isinstance(e, ToolCall)]
    assert "write_file" in calls

---

Core concepts

Recorder

Connects to one or more MCP servers as a real client, observes the agent's tool-call sequence during a live run, and persists the full trajectory as a structured JSONL trace.

A trace captures: the prompt, every tool call (name, arguments, latency, result), every error, every retry, the final response, and timing metadata. Traces are designed to be diffed, replayed, and inspected by hand.

Replay

Takes a recorded trace and replays it against a mocked MCP server — same tool surface, deterministic responses sourced from the trace. This makes evaluation:

• Fast — no network, no LLM-tool latency
• Cheap — no real API costs for downstream services
• Deterministic — same inputs every time, even if your tools touch the real world

Replay is the foundation of CI-friendly eval. Record once when something interesting happens; replay forever.

Judges

Three layers of scoring, all composable. Deterministic judges need no LLM; the LLM-judged ones take a bring-your-own callable and report confidence.

• Step judges — ToolSelectedCorrectly, ArgsValid (validates call arguments against a tool's JSON Schema). Deterministic.
• Trajectory judges — ToolCalled, ToolCalledWith, NoRetryLoop (deterministic); TrajectoryEfficient (LLM-judged).
• Outcome judges — FinalStateMatches (deterministic state assertions on disk or the final tool result); FinalResponseSatisfies (LLM-judged rubric over the response).

All layers report individually so you can see where a run failed, not just that it did.

In a .mcpeval.yaml suite, the assertion vocabulary maps onto these judges: tool_called, tool_called_with, no_retry_loop, final_state_matches, and final_response_satisfies. Print the full machine-readable schema any time with mcpevalkit schema.

Pytest plugin

The pytest --mcpevalkit plugin discovers .mcpeval.yaml files in your repo, materializes each eval as a pytest test, and reports pass/fail with per-step diagnostics in the test output. CI integration is one line.

---

Example agent

The repository ships with a small example coding agent in examples/coding_agent/ — a deterministic, LLM-free agent that drives the filesystem MCP server over stdio, scored against a fixed task set. See its agent.py, .mcpeval.yaml, and README.

git clone https://github.com/treble37/mcp-evals
cd mcp-evals
uv sync
make demo

You'll get the scored trajectory report shown at the top of this page in a second or two (it needs Node's npx for the filesystem server, but no API key — the judge defaults to a deterministic fake). The agent is intentionally small (~170 lines): it's a worked consumer of the library, not a separate product. The same suite also runs under pytest examples/coding_agent/ --mcpevalkit.

---

Design notes

Why MCP-first?

A first-class MCP recorder is more accurate than a generic tool-call wrapper because it captures the full protocol surface — server capabilities, resource references, prompts, error semantics. Wrapping MCP behind a framework abstraction loses signal.

Why trajectory-level?

Production agent failures are rarely "the final answer was wrong." They're "the agent picked the wrong tool at step 3 and recovered fine, but then misread its own state at step 6." Final-answer evaluation can't surface either of those. Step-level judges can.

Why deterministic replay?

Eval suites that hit real LLM APIs and real tool servers don't run on CI — they're too slow, too expensive, and too flaky. Record once when something interesting happens; replay forever. This is the load-bearing design choice that makes mcpevalkit usable in real CI.

Why a YAML eval format and a pytest plugin?

YAML is for non-engineering stakeholders defining expected behavior; pytest is for engineers integrating evals into existing test suites. Both compile to the same internal eval representation — you can use either or both.

---

Roadmap

v0.2 — Paired comparison

Answer not just "is this agent good enough?" but "is the new config better than the one in production?"

• Run two configurations against the same eval suite
• Paired statistical comparison with reported confidence
• A ship / don't-ship decision rule with configurable thresholds
• CLI: mcpevalkit compare --baseline traces/v1.jsonl --candidate traces/v2.jsonl

v0.3 and beyond

• Multi-turn conversation eval (currently single-turn only)
• Token / cost reporting per trajectory
• HTML report output for sharing with non-engineering reviewers

---

Status

v0.1 — Active development. API may shift. Pin to a specific version in production.

Issue reports and PRs welcome. The fastest way to influence the shape of the library is to file an issue describing your MCP agent and how you wish you could test it.

---

License

Apache 2.0 © Bruce P

---

Acknowledgements

Built on top of the Anthropic MCP Python SDK.