mcp-data-check 0.5.0

Evaluate MCP server accuracy against known questions and answers

mcp-data-check

Evaluate MCP server accuracy against known questions and answers.

Installation

pip install mcp-data-check

Or install from source:

pip install -e .

Usage

Python API

Anthropic (default)

from mcp_data_check import run_evaluation

results = run_evaluation(
    questions_filepath="questions.csv",
    api_key="sk-ant-...",
    server_url="https://mcp.example.com/sse"
)

print(f"Pass rate: {results['summary']['pass_rate']:.1%}")
print(f"Passed: {results['summary']['passed']}/{results['summary']['total']}")

OpenAI

from mcp_data_check import run_evaluation

results = run_evaluation(
    questions_filepath="questions.csv",
    api_key="sk-...",
    server_url="https://mcp.example.com/sse",
    provider="openai",
    model="gpt-4o"
)

Command Line

Anthropic (default)

mcp-data-check https://mcp.example.com/sse -q questions.csv -k YOUR_API_KEY

OpenAI

mcp-data-check https://mcp.example.com/sse -q questions.csv -p openai -m gpt-4o -k YOUR_API_KEY

Options:

• -q, --questions: Path to questions CSV file (required)
• -p, --provider: LLM provider to use: anthropic (default) or openai
• -k, --api-key: API key for the chosen provider (defaults to ANTHROPIC_API_KEY or OPENAI_API_KEY env var)
• -o, --output: Output directory for results (default: ./results)
• -m, --model: Model to use for evaluation (default: claude-sonnet-4-20250514; use e.g. gpt-4o for OpenAI)
• -n, --server-name: Name for the MCP server (default: mcp-server)
• -v, --verbose: Print detailed progress

Questions CSV Format

The questions CSV file must have three columns:

Column	Description
question	The question to ask the MCP server
expected_answer	The expected answer to compare against
eval_type	Evaluation method: numeric, string, or llm_judge

Example:

question,expected_answer,eval_type
How many grants were awarded in 2023?,1234,numeric
What organization received the most funding?,NIH,string
Explain the grant distribution,Most grants went to research institutions...,llm_judge

Evaluation Types

• numeric: Extracts numbers from responses and compares with 5% tolerance
• string: Checks if expected string appears in response (case-insensitive)
• llm_judge: Uses the selected model to semantically evaluate if the response is correct

Return Value

The run_evaluation function returns a dictionary:

{
    "summary": {
        "total": 10,
        "passed": 8,
        "failed": 2,
        "pass_rate": 0.8,
        "by_eval_type": {
            "numeric": {"total": 5, "passed": 4},
            "string": {"total": 3, "passed": 3},
            "llm_judge": {"total": 2, "passed": 1}
        }
    },
    "results": [
        {
            "question": "...",
            "expected_answer": "...",
            "eval_type": "numeric",
            "model_response": "...",
            "passed": True,
            "details": {...},
            "error": None,
            "time_to_answer": 2.35,
            "tools_called": [
                {
                    "tool_name": "get_grants",
                    "server_name": "mcp-server",
                    "input": {"year": 2023}
                }
            ]
        },
        ...
    ],
    "metadata": {
        "server_url": "https://mcp.example.com/sse",
        "model": "claude-sonnet-4-20250514",
        "provider": "anthropic",
        "timestamp": "20250127_143022"
    }
}

Result Fields

Each result in the results array contains:

Field	Description
question	The original question asked
expected_answer	The expected answer from the CSV
eval_type	Evaluation method used
model_response	The model's full response text
passed	Whether the evaluation passed
details	Additional evaluation details
error	Error message if the evaluation failed
time_to_answer	Response time in seconds for the MCP server call
tools_called	List of MCP tools invoked during the response

The tools_called array contains objects with:

• tool_name: Name of the MCP tool called
• server_name: Name of the MCP server that provided the tool
• input: Parameters passed to the tool

Requirements

• Python 3.10+
• API key for your chosen provider (Anthropic or OpenAI)