Verdict

Regression testing for AI agent products. Verdict runs your app's real code path against a recorded world, with the database, external APIs, messages and clock contained at the edges. It grades what your app did, and tells you whether a code change made things better or worse, beyond noise.

uv sync --extra dev
cd examples/refunds
uv run verdict run -k 3 --label baseline
uv run verdict run -k 3 --app-root candidate --label candidate
uv run verdict compare <baseline-run> <candidate-run> --gate     # exit 1: the candidate regressed

uv run verdict generate --dry-run -n 5     # write cases from the generator spec (drop --dry-run to use claude -p)
(cd ../../web && npm install && npm run build) && uv run verdict serve   # the web app on :8765

• Your code, our edges. Nothing in your app is mocked except its boundaries.
• Outcomes over wording. Checks run on effects and final state.
• Honest statistics. The case is the unit: an A/A noise floor, Holm-adjusted per-case tests, a holdout split and a regression gate.
• Graders you can trust. Each case can carry a known-good and a known-bad result; a grader that passes the bad one or fails the good one stops the run.
• Traces. OpenTelemetry spans per trial as agent, model and tool steps, with the step behind every failed check; optionally exported to your own backend.
• Cases at scale. verdict generate: code decides each scenario's correct outcome, Claude Code writes the content, gates and a reviewer decide what is kept.

Status: early (0.1.0, unreleased), APIs may change. Licence: Apache-2.0.