multivon-eval

Did your AI application get better—or did the measurement change?

multivon-eval is a Python library for testing LLM applications, RAG systems, and agents. Define task-specific cases, grade outputs, compare changes, and inspect quality failures separately from errors and missing evidence. Runs and reports stay local; LLM graders call the judge provider you configure. No hosted account is required.

Documentation · Examples · Benchmarks · Changelog

Current release: 0.18.0 — September 17, 2026. Python 3.10+, Apache 2.0. Migration notes.

Case manifests and trial evidence support safer comparisons and regrading. Use Hugging Face for dataset operations, Inspect for execution, and Multivon's acceptance policies for required checks and task slices. These features ship in 0.18.0. The implementation program tracks unfinished work; reuse decisions keep the integration boundaries explicit.

Start in 30 seconds

pip install multivon-eval

Run this complete example. It needs no API key:

from multivon_eval import EvalCase, EvalSuite, ExactMatch

suite = EvalSuite("capital lookup")
suite.add_cases([
    EvalCase(input="Capital of France?", expected_output="Paris"),
    EvalCase(input="Capital of Japan?", expected_output="Tokyo"),
])
suite.add_evaluators(ExactMatch())

# Replace this fixture with your application: a function from str to str.
answers = {"Capital of France?": "Paris", "Capital of Japan?": "Tokyo"}
report = suite.run(
    answers.__getitem__,
    fail_threshold=1.0,
    save_json="results.json",
    save_html="results.html",
    verbose=False,
)
print(f"{report.passed}/{report.evaluated} passed; {report.errors} errors")
# 2/2 passed; 0 errors

Open results.html to inspect each verdict. The example verifies a small lookup fixture; it does not establish the quality of a real AI application. Start your own suite by defining task success.

A real workflow example

The document-to-ledger study reuses CORD receipts, pdfhell generators and Inspect execution. Models write to SQLite; independent checks verify the saved state. Both models fail the frozen policy on 39 held-out source documents. The report separates wrong amounts, missing posts, transcription-only mismatches and integration errors, with raw-log hashes, cost accounting and a reproducible protocol. It is a sandbox study, not proof of production readiness or a new dataset.

Why use it

Pick your path

Bootstrap suggests evaluators and synthetic cases. Its p25 threshold suggestions are provisional score summaries, not calibration against human acceptance labels. Review them before using them as release criteria.

Add an LLM judge

Use deterministic checks for exact requirements and judges for qualities that need interpretation. Install the relevant provider SDK and set its API key; configure a local judge if you want to avoid hosted calls.

from multivon_eval import EvalCase, EvalSuite, Faithfulness, JudgeConfig, configure

configure(JudgeConfig(provider="anthropic", model="claude-haiku-4-5"))
suite = EvalSuite("policy answers")
suite.add_cases([EvalCase(
    input="What is the refund window?",
    context="Refunds are available within 30 days of purchase.",
)])
suite.add_evaluators(Faithfulness())

# your_app(prompt) must call your application, including its retrieval step.
report = suite.run(your_app, fail_threshold=0.90, save_json="policy-results.json")

Judge providers include Anthropic, OpenAI, Google, Ollama, and LiteLLM; OpenAI-compatible endpoints support local servers. See judge configuration for setup, historical threshold packs, and the current temperature-forwarding limitation. A judge score is an estimate to validate on your task, not ground truth.

Evaluators — 44 across 7 tiers

Browse the evaluator reference. For tool expectations, None means unspecified, [] means no calls expected, and require_order=True checks an ordered subsequence. A matching trace alone does not prove the task changed external state correctly.

Use it in CI

fail_threshold checks absolute quality. In 0.17.0, an active gate returns:

Set max_error_rate= explicitly to allow an error budget. Pass save_json=, save_html=, or save_junit_xml= into suite.run() so reports are written before a failing gate raises.

For a saved baseline/proposal comparison:

multivon-eval compare baseline.json proposal.json --fail-on-regression

This also blocks incomplete or unmatched comparisons. It flags case regressions without waiting for statistical significance; no detected regression is not proof of equivalence. See CI/CD and statistical assumptions.

Evidence and limitations

The repository publishes benchmark scripts and historical results. One cross-task measurement reported F1 0.830 on 60 HaluEval summarization outputs from 30 source examples, using a QA-selected threshold. It is a small, maintainer-run result with generated hallucination labels; it does not establish state-of-the-art accuracy. Historical live benchmarks have not been rerun after the 0.17.0 grader changes.

Confidence intervals do not correct biased labels, correlated cases, or grader mistakes. Validate your success criteria and inspect errors, skips, and important task slices. Optional compliance reports organize evidence; they do not certify legal compliance.

A case with one measured check and another skipped check still counts as evaluated. The built-in gate blocks wholly skipped cases; enforce per-check coverage separately when every evaluator is required.

Useful commands

multivon-eval validate eval.py                 # check reference outputs against graders
multivon-eval view --dir runs/                 # browse saved reports
multivon-eval bootstrap --product PRODUCT.md --traces TRACES.jsonl
multivon-eval generate --from docs/faq.md --n 20
multivon-eval assess traces.jsonl              # inspect input quality locally
multivon-eval staleness .                      # inspect prompt/case drift
multivon-eval doctor --no-ping --json          # check configuration offline

doctor exits 0 when clean, 2 when it finds warnings, and 1 when it finds an error. Use multivon-eval --help for all commands.

Current release — 0.18.0

• Reuse Hugging Face Datasets for loading; retain versioned case manifests and source groups.
• Retain individual outputs, grader results, retries and traces for inspection and regrading.
• Use Inspect for execution, native provider logs and crash recovery.
• Apply required-check, coverage and slice contracts with accept/reject/indeterminate decisions.
• Block paired significance when identity, retained trials or recorded grader settings are incompatible.
• Preserve missing trace observations and use exact small-sample McNemar tests.

See migration notes, the worked document study, and the changelog. This release does not complete provider accounting, opaque callback compatibility, customer validation or the remaining industrial program.

Related tools and contributing

Use pdfhell for adversarial document fixtures, multivon-mcp for MCP access, and eval-action for GitHub workflows. The library can sit alongside your existing tracing system.

Issues and pull requests are welcome. See CONTRIBUTING.md for setup and testing. Apache 2.0 — Multivon.