multivon-eval

Docs · Examples · Benchmarks · Changelog · Website

Know whether your AI product actually got better. multivon-eval is an open-source Python framework for evaluating LLM applications, RAG systems, agents, conversations, and document AI. It validates automated graders, quantifies uncertainty, and produces evidence your team can inspect locally.

Current release: 0.17.0 — September 17, 2026. Python 3.10–3.14, Apache 2.0, no hosted account and no telemetry. See what changed.

Start in 30 seconds

pip install multivon-eval
python -m multivon_eval

That runs a real deterministic evaluation without an API key. To scaffold an editable project:

multivon-eval init -t quickstart -d my-eval
cd my-eval
python eval.py

Or use the Python API directly:

from multivon_eval import Contains, EvalCase, EvalSuite, NotEmpty

suite = EvalSuite("support-smoke", purpose="regression")
suite.add_cases([
    EvalCase(input="What is 2+2?", expected_output="4"),
])
suite.add_evaluators(NotEmpty(), Contains(["4"]))

if __name__ == "__main__":
    report = suite.run(lambda prompt: "2+2 = 4", runs=5)
    print(report.pass_rate, report.pass_rate_ci())
    print(report.pass_hat_k(3))

Replace the lambda with your model function. The report includes confidence intervals, repeated-trial reliability, failure reasons, and exportable JSON, CSV, HTML, and JUnit artifacts.

Why teams choose it

Most evaluation libraries grade model output. multivon-eval also checks whether the measurement itself deserves trust.

The design rule is simple: an honest UNKNOWN is better than a confident wrong answer. Judge outages are reported as infrastructure errors, insufficient samples trigger power warnings, and unsupported statistical questions do not silently become scores.

Pick your path

Anthropic, OpenAI, Gemini, Ollama, LM Studio, vLLM, and other OpenAI-compatible judge endpoints are supported. Agent integrations include LangChain, LangGraph, LangSmith, the OpenAI Agents SDK, and manual traces.

How a run works

cases + reference answers
        ↓
your model, one or many trials per case
        ↓
deterministic checks + calibrated judges + trace evaluators
        ↓
validated report: verdicts + reasons + uncertainty + errors
        ↓
local HTML / JSON / CSV / JUnit / CI gate / audit package

Three objects make up the core API:

• EvalCase describes the input, context, reference output, agent trace, or conversation to evaluate.
• Evaluator grades one dimension and explains its verdict.
• EvalSuite runs cases, coordinates repeated trials, and produces an EvalReport.

from multivon_eval import EvalCase, EvalSuite, Faithfulness, Relevance

suite = EvalSuite("rag-regression", purpose="regression")
suite.add_evaluators(Faithfulness(), Relevance())
suite.add_cases([
    EvalCase(
        input="What is the renewal period?",
        context="The agreement renews annually unless terminated.",
    )
])

report = suite.run(my_rag_model, runs=5, fail_threshold=0.95)
report.save_html("report.html")
report.save_junit_xml("junit.xml")

Evaluators — 44 across 7 tiers

See the evaluator reference for signatures, inputs, costs, and examples. The two multimodal evaluators are experimental and are not yet covered by the text-evaluator calibration pipeline.

For agent cases, expected_tool_calls=None means no expectation was supplied, [] asserts that no tool should be called, and [...] lists the expected calls. require_order=True checks them as an ordered subsequence, so unrelated calls may still occur between expected ones.

Reliability and CI

Every report carries the context needed to interpret its headline score:

• Wilson 95% confidence interval for pass rate and bootstrap interval for the average score.
• Power warning and sample-size guidance when the suite cannot detect the change you care about.
• pass@k for capability and pass^k for repeated-use reliability.
• Explicit JUDGE_ERROR / EVALUATOR_ERROR accounting and an optional max_error_rate gate.
• Paired run comparison with McNemar tests and Benjamini–Hochberg correction.

report = suite.run(model_fn, runs=5, fail_threshold=0.90, max_error_rate=0.05)
report.assert_pass_hat_k(k=3, min_ci_low=0.80)

For the estimators, assumptions, and interpretation, read the statistical-rigor guide and reliability guide.

Evidence, not just claims

The repository includes the benchmark code, datasets, configurations, and raw results behind its published evaluator claims.

• Held-out hallucination evaluation: F1 0.830 [0.70–0.92] after calibrating on a different HaluEval split.
• In-distribution HaluEval QA: multivon-eval F1 0.804 [0.71–0.88] versus DeepEval 0.586 [0.48–0.68] under the disclosed configurations.
• When a prompt-drift experiment missed its own 50% determinacy gate and scored 20.9%, the failed result was published and the design was changed.

Inspect the benchmark methodology and raw artifacts, or read the project’s public track record.

Local-first compliance

PIIEvaluator performs local detection for GDPR, CCPA, HIPAA, DPDP (India), and PIPEDA patterns. ComplianceReporter writes a hash-linked NDJSON record; anchor the head hash externally when you need evidence against deliberate rewriting. Audit packages include the run, cases, calibration provenance, manifest, and offline verifier.

from multivon_eval import ComplianceReporter, EvalSuite

suite = EvalSuite.eu_ai_act_high_risk(jurisdiction="gdpr")
report = suite.run(model_fn, runs=5)
ComplianceReporter("./audit", framework="eu-ai-act").record(report)

See the compliance guide and audit-package format.

Useful commands

multivon-eval validate eval.py                 # grade the graders
multivon-eval view --dir runs/                 # browse and compare local reports
multivon-eval compare before.json after.json   # paired regression analysis
multivon-eval bootstrap --product PRODUCT.md --traces TRACES.jsonl
multivon-eval generate --from docs/faq.md --n 20
multivon-eval assess traces.jsonl              # free input-quality preflight
multivon-eval staleness .                      # detect prompt/case drift
multivon-eval doctor --no-ping --json          # offline configuration check

Run multivon-eval --help for the complete CLI. The documentation carries the exhaustive flag and API reference so this README can stay focused on orientation. doctor exits 0 when clean, 2 when it finds warnings, and 1 when it finds an error; warnings are useful diagnostics, not a failed installation.

Ecosystem

Four public packages plus one closed early-access product:

Use multivon-eval beside an observability platform when you need both production traces and release gates. For research-grade model capability studies, consider Inspect AI. The longer selection guide is in the docs.

Current release — 0.17.0

Released September 17, 2026:

• Skipped graders no longer count as successful measurements or inflate scores.
• Quality gates reject unmeasured cases and default to zero infrastructure errors; set max_error_rate= explicitly to permit an error budget.
• Apology prefixes cannot bypass factuality checks. Ambiguous verdicts stay UNKNOWN.
• Comparisons separate infrastructure recovery from model improvements.
• Judge timeouts reach the provider; reliability checks bypass cached verdicts.

See the migration notes for the stricter gate and skip semantics. Published benchmark numbers describe historical configurations; this correctness release does not claim new benchmark accuracy.

Read the complete changelog for release history and migration details.

Contributing

Issues and pull requests are welcome. For substantial new evaluators or API changes, open an issue first so the measurement contract can be reviewed.

git clone https://github.com/multivon-ai/multivon-eval
cd multivon-eval
pip install -e ".[dev]"
python -m pytest tests/ --ignore=tests/test_integrations_live.py

License

Apache 2.0 — built by Multivon