mcp-evals

Behavioral evaluation and description linting for Model Context Protocol (MCP) servers.

Status: v0.6 — stdio + HTTP JSON-RPC discovery, description lint (including name-description-mismatch + missing-examples), mock/OpenAI-compatible tool-selection evals, and GitHub Action PR comments.

60-second try

pip install mcp-tool-evals
mcp-evals health
mcp-evals run examples/eval-suite.yaml --catalog examples/tools.yaml --model mock
# or with Docker (no local Python):
docker compose run --rm run-example

Why this vs alternatives

Approach	Strength	Gap
mcp-evals	Lint + YAML evals + mock CI runner	Not a full agent harness
MCP Inspector	Interactive debugging	No CI lint/eval suite
Protocol conformance tests	JSON-RPC correctness	Do not test tool selection
Hand-written LLM mocks	Full control	Drift from real descriptions

Problem

Protocol conformance tests verify JSON-RPC correctness. They do not verify whether an LLM agent selects the right tool, passes sensible arguments, or completes the task. Tool descriptions are now load-bearing API design, and there is no standard way to test them.

mcp-evals fills the behavioral layer: lint tool descriptions, define eval cases in YAML, and (next) measure tool-selection accuracy in CI.

Key features (v0.6)

• Description linter — missing descriptions, undocumented required params, overlapping tools, ambiguous verbs, name-description-mismatch, missing-examples
• Live discover — handshake a stdio or HTTP JSON-RPC MCP server (or load a catalog fixture)
• Eval runner — mcp-evals run with --model mock (CI), --live SERVER.yaml, or an OpenAI-compatible endpoint
• CLI + Docker + GitHub Action — lint and optional eval in CI without a cloud key (mock selector); PR comment or job summary with accuracy

Architecture

┌─────────────┐     ┌──────────────────┐     ┌─────────────────┐
│  CLI        │────▶│  Linter / Runner │────▶│  Models (Pydantic)│
│  mcp-evals  │     │  spec_loader     │     │  EvalSuite, etc.  │
└─────────────┘     └──────────────────┘     └─────────────────┘
                            │
                    (next) MCP client + LLM

Component	Technology	Why
Language	Python 3.11+	MCP SDK ecosystem, pytest integration
CLI	Click	Mature, composable commands
Schemas	Pydantic v2	Strict validation, good errors
Logging	structlog	Structured, JSON-capable
Config	pydantic-settings	Env-based, typed
Tests	pytest + ruff + mypy	Standard Python OSS stack

Installation

pip install mcp-tool-evals
# or from source:
pip install -e ".[dev]"

Local development

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
mcp-evals health
mcp-evals lint examples/tools.yaml
mcp-evals discover examples/server.yaml
mcp-evals lint --live examples/server.yaml
mcp-evals validate-spec examples/eval-suite.yaml
mcp-evals run examples/eval-suite.yaml --catalog examples/tools.yaml --model mock
pytest tests/ -v

Docker

# Health check
docker compose run --rm dev

# Run tests
docker compose run --rm test

# Lint a live stdio server (this example passes)
docker compose run --rm lint-example

# Discover tools from the example calc server
docker compose run --rm discover-example

# Mock eval run (no API key)
docker compose run --rm run-example

Configuration

Copy .env.example to .env:

Variable	Default	Description
MCP_EVALS_LOG_LEVEL	INFO	Log level
MCP_EVALS_LOG_JSON	false	JSON log output
MCP_EVALS_DEFAULT_MODEL	gpt-4o-mini	Model for eval runs (future)
MCP_EVALS_PASS_THRESHOLD	0.8	Minimum selection accuracy

Usage

Lint tool descriptions

mcp-evals lint examples/tools.yaml
mcp-evals lint --live examples/server.yaml
mcp-evals discover examples/server.yaml
mcp-evals lint examples/tools.yaml --format json
mcp-evals lint examples/tools.yaml --fail-on-warning

Run behavioral evals

mcp-evals run examples/eval-suite.yaml --catalog examples/tools.yaml --model mock
mcp-evals run examples/eval-suite.yaml --catalog examples/tools.yaml \
  --model llama3.2 --base-url http://localhost:11434/v1 --pass-threshold 0.8
# or discover tools from a live server that matches the suite:
# mcp-evals run suite.yaml --live server.yaml --model mock

Validate eval suite

mcp-evals validate-spec examples/eval-suite.yaml

Health check

mcp-evals health

GitHub Action

On pull_request, the composite action posts (or updates) a comment with selection accuracy and a per-case table. On other events it writes the same markdown to $GITHUB_STEP_SUMMARY. Set comment: false to skip PR comments.

permissions:
  contents: read
  pull-requests: write
steps:
  - uses: actions/checkout@v4
  - uses: yashshah9/mcp-evals@v0.6
    with:
      catalog: examples/tools.yaml
      suite: examples/eval-suite.yaml
      model: mock
      threshold: "0.8"
      # comment: false  # optional; default true

Example eval suite

See examples/eval-suite.yaml:

name: document-server-behavior
cases:
  - id: search-by-keyword
    request: Find documents about quarterly revenue
    expected_tool:
      name: search_documents
      arguments:
        query: quarterly revenue

Running tests

pytest tests/ -v
ruff check src tests
mypy src

Development workflow

1. Add linter rules in src/mcp_evals/linter/
2. Add runner logic in src/mcp_evals/runner/
3. Add tests in tests/
4. Update examples in examples/

Roadmap

• MCP client: stdio discover + lint --live
• LLM eval runner (mock + OpenAI-compatible)
• HTTP JSON-RPC MCP transport
• name-description-mismatch linter rule
• Streamable HTTP/SSE MCP transport
• GitHub Action PR comments (accuracy + per-case table; deltas later)

Known limitations (v0.4)

• Streamable HTTP/SSE MCP is not implemented — HTTP is JSON-RPC POST only
• Mock selector does not fill tool arguments (accuracy is tool-name only)
• Live Ollama/OpenAI evals need a reachable --base-url; CI uses --model mock
• examples/tools.yaml is intentionally dirty so mcp-evals lint can show findings

Contributing

See CONTRIBUTING.md.

License

MIT — see LICENSE.