Verdict

Evaluation infrastructure for AI agents.

Demo

Install

pip install verdict-eval

Quickstart

# Run an evaluation against a built-in adapter
verdict eval --target simple_rag --num-per-category 5

# Compare two adapter versions
verdict diff --target-a simple_rag --target-b path/to/v2.py:MyAdapter --num 10

# Analyze flakiness across historical runs
verdict flakiness --target my-system --reports-dir ./reports

CLI reference

verdict eval

Run a full evaluation against a target adapter.

verdict diff

Compare two adapter versions against the same generated test suite.

verdict diff \
  --target-a simple_rag \
  --target-b path/to/v2.py:V2Adapter \
  --num 10

verdict flakiness

Analyze judge and target consistency across historical evaluation runs.

verdict flakiness --target my-system --min-runs 5 --reports-dir ./reports

verdict compliance

Map an eval report to HIPAA Security Rule and NIST AI RMF controls, producing machine-readable and human-readable compliance artifacts.

# After running verdict eval, point at the JSON report:
verdict compliance --report ./reports/eval_abc123.json --output-dir ./compliance

Outputs two files:

Frameworks covered: 5 HIPAA Security Rule controls (general, administrative, and technical safeguards) and 8 NIST AI RMF controls (MAP, MEASURE, MANAGE functions) — a curated subset that maps naturally to eval outcomes.

What makes this statistically grounded:

• Each control's evidence entry includes a per-source 95% bootstrap CI
• Controls aggregated from multiple categories carry a combined CI
• Flakiness-detected prompts reduce the confidence rating for affected controls
• Token/cost provenance is recorded in provenance for full audit traceability

Python API:

from verdict.compliance import generate_audit_artifact, save_artifacts
from verdict.models.schemas import EvalReport
import json

report = EvalReport(**json.loads(Path("eval_report.json").read_text()))
artifact = generate_audit_artifact(report)
json_path, md_path = save_artifacts(artifact, Path("./compliance"), report.run_id)

See examples/compliance_example.py for a runnable demo with synthetic data.

Adaptive mode

When --adaptive is enabled, Verdict runs a second pass of follow-up probes selected based on each initial response. Pattern selection is entirely rule-based — no LLM is used to generate new attacks. All probes are composed from the curated OWASP LLM Top 10 pattern library in verdict/evals/attack_patterns/patterns.json.

This design ensures Verdict remains a defensive evaluation tool. See CONTRIBUTING.md for the security boundary policy.

verdict eval --target simple_rag --adaptive

Writing a custom adapter

# my_adapter.py
from verdict.adapters.base import TargetAdapter
from verdict.models.schemas import ExecutionResult, TestPrompt

class MyAdapter(TargetAdapter):
    name = "my-system"
    version = "1.0.0"

    async def execute(self, prompt: TestPrompt) -> ExecutionResult:
        response = call_my_system(prompt.prompt)
        return self.make_result(prompt, response=response)

verdict eval --target my_adapter.py:MyAdapter

Test categories

Judge calibration

The Judge is validated against 22 hand-labeled examples covering all five test categories. Results are produced by running the live judge against known ground truth — no labels were derived from judge output.

Measured on claude-sonnet-4-6, 2026-05-22.

Run calibration locally (requires ANTHROPIC_API_KEY):

pytest tests/qa/test_judge_calibration.py -v -m llm

See docs/judge_calibration.md for full methodology.

License

MIT — see LICENSE.