mcp-guardeval

mcp-guardeval is a standalone, framework-agnostic evaluation harness for LLM agents utilizing the Model Context Protocol (MCP). It captures OpenTelemetry spans (gen_ai.* conventions), normalizes them into queryable SQLite records, and quantitatively scores both task performance and security resilience against catalogued SAFE-MCP adversarial techniques.

Installation

pip install mcp-guardeval

Key Capabilities

1. Task Success Scoring: Measures whether the agent called the expected tools, supplied correct arguments, and respected tool dependencies.
2. Adversarial Security Scoring: Evaluates agent behavior against red-team attack techniques (prompt injection, argument hijacking, data exfiltration, context planting, PII harvesting).
3. Telemetry & Trace Analysis: Normalizes hierarchical OpenTelemetry distributed traces into flat SQLite rows for high-speed SQL analytics.
4. Pytest Integration: Built-in pytest plugin enabling single-command evaluation:

   pytest tests/ --agenteval -v
   

Quick Start

1. Telemetry Ingestion & Storage

mcp-guardeval reads OpenTelemetry trace data persisted to SQLite:

from agenteval.storage import TraceStore

# Connect to the SQLite trace database
store = TraceStore("traces.db")

# Inspect completed agent runs
runs = store.get_runs()
for run in runs:
    print(f"Run {run.run_id}: {run.prompt[:50]}... | {run.latency_ms:.1f}ms")

2. Scoring Task Success

Evaluate whether an agent executed the intended workflow:

from agenteval.metrics.task_success import score_task
from agenteval.storage import TraceStore

store = TraceStore("traces.db")

# Score a customer lookup workflow
result = score_task(
    store=store,
    run_id="run_1042",
    expected_tools=["query_customer_db"],
    expected_args={"query_customer_db": {"customer_id": "4471"}},
    expected_sequence=["query_customer_db"],
)

print(f"Task Passed: {result.passed}")
print(f"Confidence Score: {result.score:.2f}")
print(f"Missing Tools: {result.missing_tools}")

3. Evaluating Guardrail Security

Assess whether guardrails intercepted adversarial SAFE-MCP attacks:

from agenteval.metrics.security import score_security_run
from agenteval.storage import TraceStore

store = TraceStore("traces.db")

# Evaluate an indirect prompt injection attack (SAFE-T1201)
verdict = score_security_run(
    store=store,
    run_id="attack_run_88",
    technique_id="SAFE-T1201",
    guardrail_log="reference_system/fixtures/guardrail.log",
)

# Verdict options: BLOCKED (secure), PASSED (exploited), or PARTIAL
print(f"Security Verdict: {verdict.status.name}")
print(f"Mitigation Reason: {verdict.reason}")

Pytest Plugin Usage

mcp-guardeval automatically registers with pytest when installed. Use the --agenteval CLI flag to activate trace evaluation and automated reporting during test execution:

# Run security test suite with AgentEval report summary
pytest tests/test_security.py --agenteval -v

Supported SAFE-MCP Attack Techniques

License

Distributed under the MIT License. See LICENSE for more information.