multivon-mcp

Docs · Website · PyPI · multivon-eval (engine) · Changelog

These 23 tools cover what an autonomous eval agent needs to do its job: discover its own capabilities (eval_discover), normalize traces from supported sources (eval_ingest_trace), and run evaluators against them, with calibration evidence specific to the tested task. We put the framework behind an MCP boundary because eval belongs in the agent's working loop, not behind a separate dashboard.

An MCP server that gives AI coding agents direct access to evaluation tools. Drop into Claude Desktop, Claude Code, Cursor, Cline, or any Model Context Protocol–compatible agent.

When the agent is helping you build an LLM product, it can:

• Score a RAG output for hallucination without you writing the scaffolding
• Generate an adversarial PDF on demand to test your document AI
• Run the full pdfhell mini-suite against a model and analyse the results
• Produce a self-verifying audit pack with a SHA-256 file manifest
• Discover the full evaluation capability catalog as JSON

No copy-paste, and no asking the agent to figure out the SDK calls from python -c "..." one-liners.

Current release: 0.4.0. Tested with MCP Python SDK 1.29.x, multivon-eval 0.18.0, and pdfhell 0.6.2. See the changelog.

Install

pip install "multivon-mcp==0.4.0"

The next release carries this compatibility bound itself. Installation pulls multivon-eval, pdfhell, and the MCP SDK. The provider SDKs (anthropic, openai, google-genai) come along too — bring your own API key in env.

Configure your agent

Claude Code

claude mcp add --transport stdio --scope user multivon -- multivon-mcp
claude mcp get multivon

Set provider keys in your shell or secure environment before starting Claude Code. To share the server configuration with a project instead, use --scope project; Claude Code writes .mcp.json and supports environment-variable expansion there. It does not read claude_desktop_config.json.

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "multivon": {
      "command": "multivon-mcp",
      "env": {
        "ANTHROPIC_API_KEY": "sk-ant-...",
        "OPENAI_API_KEY": "sk-proj-...",
        "GOOGLE_API_KEY": "AIza..."
      }
    }
  }
}

Restart Claude. The 23 tools become available; ask Claude "use multivon to evaluate this RAG output" and it figures out which tool to call.

Cursor

.cursor/mcp.json or via Settings → MCP:

{ "mcpServers": { "multivon": { "command": "multivon-mcp" } } }

Cline / OpenCode / any MCP-compatible agent

Same shape — point at the multivon-mcp console script.

Local dev / debugging

From a clone of this repo:

mcp dev multivon_mcp/server.py

From a pip install (the file lives in site-packages, so resolve it):

mcp dev "$(python -c 'import multivon_mcp.server as s; print(s.__file__)')"

Opens the MCP Inspector UI in your browser. You can call any tool by name, see the JSON schemas, and watch the requests/responses.

The 23 tools

Discovery & document AI

Tool	What it does	API key
eval_discover	Full machine-readable capability catalog (evaluators, traps, suites, calibration data, versions). Call first.	No
pdfhell_make	Generate one adversarial PDF + its answer key.	No
pdfhell_run	Run the pdfhell adversarial-PDF benchmark against a vision model. Returns pass rate, per-trap CIs, suite hash.	Yes (vision)
eval_audit_pack	Build a procurement-ready ZIP with a SHA-256 file manifest from a pdfhell run.	No

RAG generation & retrieval

Tool	What it does	API key
eval_faithfulness	QAG-graded faithfulness — is a RAG output grounded in the retrieved context?	Yes
eval_hallucination	QAG-graded hallucination — does the output contain content NOT in context?	Yes
eval_relevance	QAG-graded answer-vs-question relevance.	Yes
eval_answer_accuracy	QAG-graded semantic equivalence vs ground truth.	Yes
eval_context_precision	RAG retrieval quality — are the retrieved chunks on-topic?	Yes
eval_context_recall	RAG retrieval completeness — does context contain enough info to answer?	Yes

Safety, compliance, fairness

Tool	What it does	API key
eval_toxicity	QAG-graded toxicity / harmful-content detection.	Yes
eval_bias	QAG-graded bias across gender, race, politics, age, socioeconomic axes.	Yes
eval_pii_detection	Local-only regex scan for PII (GDPR / CCPA / PIPEDA / HIPAA / DPDP packs).	No
eval_schema_compliance	Validate an LLM output against a JSON Schema.	No

Agent & multimodal

Tool	What it does	API key
eval_tool_call_accuracy	Deterministic agent tool-call correctness. No LLM.	No
eval_vqa_faithfulness	Image-grounded visual-QA faithfulness.	Yes (vision)
eval_document_grounding	Multi-page document-grounded faithfulness for document-AI agents.	Yes (vision)

Agent traces. eval_tool_call_accuracy and the other agent-trace evaluators in multivon-eval (ToolArgumentAccuracy, ToolCallNecessity, TrajectoryEfficiency, AgentMemoryEval, PlanQuality, TaskCompletion, StepFaithfulness) take an agent_trace=[AgentStep(...)] plus expected_tool_calls=[...] on the case. Three-shape semantics matter: expected_tool_calls=None skips, [] asserts "no tools called", and [...] checks for the named calls. On repository main (shipping in the next release), the MCP tool supports the same trace mode: normalize trace JSON with eval_ingest_trace, then pass its agent_trace plus expected_tool_calls to eval_tool_call_accuracy. Set require_order=true when sequence matters or penalize_unexpected=true for a strict allow-list. See the multivon-eval agent integrations for the source-of-truth tracer code.

Flexible scoring

Tool	What it does	API key
eval_g_eval	G-Eval holistic 0.0-1.0 scoring against a plain-English criterion.	Yes
eval_custom_rubric	Score against your own list of yes/no quality checks.	Yes

Agent workflows (new in 0.3.0)

Tool	What it does	API key
eval_acceptance_report	Apply required-check, coverage and slice policy to a saved report; returns accept/reject/indeterminate.	No
eval_compare_runs	Diff two eval report JSONs — pass-rate delta, per-case regressions/improvements, McNemar p-value. Includes identity warnings; inconclusive comparisons cannot confirm improvement.	No
eval_generate_cases	Generate N eval cases (input / expected_output / context) from a chunk of source text. Eliminates the cold-start when building a new suite.	Yes (judge)
eval_ingest_trace	Convert a JSON agent trace (LangGraph / OpenAI Agents / manual) into an EvalCase payload. Use to score trajectories your agent just executed.	No

Example session

User: I just shipped a RAG endpoint. Can you check it for hallucinations?

Claude: I'll use multivon to evaluate it.
        [calls eval_discover to see what's available]
        [calls eval_faithfulness with your input/context/output]

→ score: 0.667 (passed: False), threshold: 0.9
  reason: 2/3 claims grounded
    ✓ "annual renewal" — supported by context
    ✓ "30-day notice" — supported by context
    ✗ "automatic upgrade" — NOT in context

Claude: Your RAG hallucinated the "automatic upgrade" detail. The context
        doesn't mention upgrades. I'd add a Hallucination evaluator to your CI
        gate, threshold ≥0.85, and re-prompt with explicit "only use facts
        from context" instructions.

Release decisions from saved evidence

Use eval_acceptance_report(report_json_path, policy_json_path) to apply the same multivon.policy/v1 contract used by the CLI and CI action. It makes no model calls. The result includes decision (accept, reject, indeterminate), exit_code, policy_digest, measurements and findings. Malformed files or policies produce an MCP tool error, not an accepted result. See the acceptance policy guide.

Comparison is diagnostic: eval_compare_runs retains identity_verified and identity_issues. Missing or incompatible evidence suppresses the paired p-value. A large p-value does not prove equivalence, and an unchanged failing application still needs an absolute release contract.

Evaluator responses include status, measured and original metadata. For skipped or errored measurements, score and passed are null. Do not coerce them into an ordinary pass or fail. Missing agent_trace means unknown; provide agent_trace: [] only for an observed empty trajectory. Trace ingestion also requires an explicit steps list.

In single-call accuracy mode, supplied expected arguments must match exactly: extra keys, absent null-valued keys and changed JSON types fail. Omitting expected_arguments explicitly selects tool-name-only comparison. Trace mode checks tool names/order, not argument correctness or external side effects.

These response semantics change in 0.4.0. Update clients that assume passed and score are always measured values. The stdio integration test exercises real initialization, discovery, acceptance, missing evidence and tool errors.

Why these 23 tools (not all 44)

eval_discover returns the full 44-evaluator catalog, so the agent can always introspect everything. The 23 tools we expose directly are the ones agents actually call mid-edit:

• RAG generation checks (faithfulness, hallucination, relevance, answer_accuracy)
• RAG retrieval checks (context_precision, context_recall)
• Safety / fairness guardrails (toxicity, bias)
• Compliance (pii_detection, schema_compliance) — local-only, no API egress
• Flexible scoring (g_eval, custom_rubric) for user-defined rubrics
• Multimodal (vqa_faithfulness, document_grounding) for vision agents
• Agent traces (tool_call_accuracy)
• Document AI (pdfhell_run, pdfhell_make) — for any RAG-on-PDFs flow
• Audit pack — when procurement is involved
• Discover — meta-capability for planning
• Agent workflows (compare_runs, generate_cases, ingest_trace) — the loop that turns one-shot scoring into iterative improvement

The three new 0.3.0 tools exist because evals pay off as a loop: generate a starting suite from your own docs (eval_generate_cases), run your agent over it, score the trace (eval_ingest_trace → eval_*), make a fix, then verify the fix improved things vs. the baseline (eval_compare_runs). Agents need that whole loop callable from within a conversation, or they fall back to ad-hoc judgment.

Exposing all 44 evaluators as MCP tools would bloat the agent's context window and overwhelm tool-selection. If you need an evaluator that's not directly exposed, the agent can still use multivon-eval as a library — eval_discover returns the import paths.

Dependencies

Tested runtime bounds (from pyproject.toml):

• mcp[cli] >= 1.29, < 2 — official MCP Python SDK and Inspector. MCP 2.0 has a different server API and is intentionally excluded until this server migrates.
• multivon-eval >= 0.18.0 — the 44-evaluator engine, current report schema, and reasoning-judge fix.
• pdfhell >= 0.6.2 — the 17-family mini-v4 registry, corrected trap renderings, and current audit-pack schema.

These bounds are on repository main and will ship in the next release. For released 0.3.2, use pip install "mcp<2" multivon-mcp so pip does not resolve the incompatible MCP 2.0 server API.

All Apache 2.0.

MCP server vs Claude Code skills vs eval-action — which one do I use?

multivon-eval ships three agent-facing surfaces. They overlap on what they call (the same evaluator catalog) but differ on where the agent lives.

Surface	Where the agent runs	Best for
multivon-mcp (this repo)	Any MCP-compatible client — Claude Desktop, Cursor, Cline, OpenCode, Claude Code	Mid-edit scoring inside an IDE or chat app. Agent calls eval_faithfulness / eval_hallucination / etc. directly as tools.
Claude Code skills — eval-bootstrap, eval-audit, eval-explain (bundled in multivon-eval >= 0.9.8; install with multivon-eval install-skills)	Claude Code only	Workflow-shaped tasks: scaffold an eval suite from a project description, pre-PR regression checks against a baseline, explaining why a particular evaluator was picked. The skills know how to call multivon-eval bootstrap / use compare_reports / etc. so the agent doesn't have to figure it out from docs.
eval-action	GitHub CI	Gate every PR on eval regressions automatically. Posts the Wilson-CI + McNemar verdict as a PR comment.

If you're building an LLM product and want the agent in your editor to score a RAG output without copy-pasting Python, use multivon-mcp. If you live in Claude Code and want the bootstrap → audit → explain loop wired up as native commands, use the bundled skills. For PR-time gating, use the GitHub Action. Most projects end up using more than one.

The Multivon ecosystem

Four public packages plus one closed early-access product, built around the same evaluation engine:

Repo	What it is
multivon-eval	Python SDK — 44 evaluators + bootstrap CLI + multivon_eval.auto. The engine multivon-mcp wraps.
pdfhell	Adversarial PDFs that break AI document readers — exposed here as pdfhell_run + pdfhell_make tools
multivon-mcp (you are here)	MCP server — 23 tools from multivon-eval + pdfhell
eval-action	GitHub Action — runs the same evals on every PR
multivon-guard (early access)	Local proxy that catches LLM coding agents leaking secrets / PII

License

Apache 2.0.

Citing

@software{multivon_mcp,
  title  = {multivon-mcp: MCP server exposing multivon-eval + pdfhell as agent-callable tools},
  author = {Multivon},
  year   = {2026},
  url    = {https://github.com/multivon-ai/multivon-mcp},
}