AgentVerity

Your agent test passed. Would it pass again?

A Python library and CLI that reruns the decisions your agent makes and reports whether they are repeatable enough, and varied enough, to trust as a regression baseline: the reviewed run you save as expected behaviour and compare future releases against.

Keep using Promptfoo, DeepEval, or your own assertions to decide whether each answer is correct. AgentVerity reuses those same results to find unstable routes, missing decisions, and runs too small to support a reliable baseline. It can read runs you already collected, so you do not pay for the same calls twice.

The 60-second problem

Consider a routing workflow: a model classifies each payment dispute and directs it to one of six specialist queues. Promptfoo runs six cases 26 times. Its configured quality checks accept either fraud queue for one ambiguous card-security case. All 156/156 assertions pass.

Would you save that run as the expected behaviour for future releases?

AgentVerity reuses the same Promptfoo export and finds this:

4. STABILITY BY ROUTE
   route              cases  pairs  flips  95% CI            result
   card_security          1     13      8  [0.355, 0.823]    stochastic
   cash_withdrawal        1     13      0  [0.000, 0.228]    undecided
   duplicate_charge       1     13      0  [0.000, 0.228]    undecided
   ...
   flip pairs:
     card_security <-> merchant_dispute  x8

The contract check passes, but the decision switches between card_security and merchant_dispute in 8 of 13 paired reruns.

Why you should care

• Those labels send work to different queues, controls, and owners.
• A moving reference makes later regression failures noisy and hard to trust.
• One pooled score hides which route is moving.
• Zero observed changes do not prove a quiet route is stable when the sample is too small.

AgentVerity names the unstable route and leaves the five underpowered routes undecided. It will not freeze this run as a baseline.

Try it without model calls

The repository includes that recorded Promptfoo run:

git clone --depth 1 https://github.com/mrwersa/agentverity.git
cd agentverity
python -m pip install agentverity
agentverity assess \
  --promptfoo examples/promptfoo_bridge/results.json \
  --suite examples/payment_decisions.json

The last command performs arithmetic over saved decisions. It makes no model or provider calls.

Already using DeepEval? Pass the same precomputed LLMTestCase objects to evidence_from_deepeval. Any harness can use the small neutral evidence format.

Keep your existing evaluator for correctness and trajectory quality. AgentVerity checks whether the repeated results are stable and complete enough for a regression baseline.

Where to integrate it

AgentVerity is a test and release step, not serving-path middleware.

One real integration combines DeepEval quality, AgentVerity evidence, and Amazon AgentCore health before admitting a baseline:

Never repeat live customer requests. Use reviewed synthetic cases in CI, before release, or on a schedule.

Where it sits in the evaluation loop

1. Explore capability. Use challenging cases to learn what the agent can and cannot do.
2. Evaluate quality. Keep Promptfoo, DeepEval, or your current assertions for outcomes and trajectories.
3. Qualify the evidence. Import repeated decisions into AgentVerity without calling the model again.
4. Fix what is missing. Repair moving routes, add missing cases, or collect the reruns needed for an honest conclusion.
5. Promote a regression case. A mature capability case becomes a reusable baseline only after a human approves it and the evidence gate admits it.
6. Monitor and learn. Review canary failures and production incidents, then add suitable cases back to the offline dataset.

The import command diagnoses decisions already collected by another evaluator. The snapshot and check commands below provide the same admission policy when AgentVerity calls your agent directly.

Outcomes, trajectories, and decisions remain separate evidence:

An evaluation framework can grade the first two. AgentVerity qualifies the repeatability and declared coverage of the bounded decision before that result becomes a regression reference.

What it checks

It keeps three outcomes separate:

• stable enough for the declared tolerance
• unstable above that tolerance
• undecided because the run did not collect enough evidence

Is it for my agent?

Use it when:

• the component chooses from named routes, approvals, policies, tools, or hand-offs
• repeated runs can start from equivalent isolated state
• you can write deliberately varied cases for the decisions that matter

It fits the agent workflow patterns whose value is a named choice rather than open prose:

Concrete examples: support and payment routing, fraud triage, incident dispatch, approval flows, and bounded tool selection.

Use another evaluator for open-ended chat, RAG quality, generated content, or coding-agent output. If such a system also emits a reviewed route or approval, AgentVerity can qualify that decision layer.

Check applicability and exact limits.

Why rerun counts are harder than they look

Picking three or five reruns by convention is guesswork:

• 36 paired reruns with no changes only bound the change rate below 9.6%.
• A claim below 5% needs 73 zero-change pairs.
• A short quiet run is therefore undecided, not proven stable.

AgentVerity sizes the run from your tolerance, uses non-overlapping pairs, and keeps three answers: stable enough, unstable, or undecided. The default balanced setting uses a 5% tolerance.

A small pytest loop can collect calls. The library packages the harder policy: evidence sizing, route-specific targets, and one consistent decision across text, JSON, JUnit, telemetry, and snapshots.

Read the executable arithmetic and design.

The evidence gate

The evidence gate refuses to save a baseline until:

• calls complete
• decisions are stable enough
• the cases reach the required decisions
• a person approves the reference outputs as correct

The bundled payment-dispute example runs two test sets:

python examples/payment_dispute_gate.py

Both score 6/6. The narrow set is a valid unit test for one route, but it is not a system-wide baseline. The repaired set reaches all six required routes and can be admitted.

Before spending model calls, inspect the zero-change evidence budget:

agentverity plan --suite examples/route_stability_plan.json

Then create the reviewed baseline:

agentverity snapshot \
  --agent examples/payment_dispute_gate.py:build_agent \
  --suite examples/payment_decisions.json \
  --output baseline.json \
  --accept-reference

The same checks run before agentverity check reports differences as regressions. Snapshot files retain SHA-256 input fingerprints rather than raw prompts.

The contract can also declare stricter stability targets for critical routes and minimum case counts. Repeats support a stability claim. Distinct reviewed cases support breadth. AgentVerity keeps those two claims separate.

Measured production example

The optional production example combines a Strands routing agent on Amazon Bedrock, DeepEval route-quality checks, AgentVerity, AgentCore Runtime, and CloudWatch.

At its declared 10% canary tolerance, the London run recorded 6/6 correct routes, no changes across 36 repeat pairs, all six routes reached, and 78 successful cloud calls with no errors or throttles. An earlier run was stable but only 5/6 correct. The release policy therefore requires both quality and evidence rather than treating either tool as sufficient.

This is deployment proof, not an AWS requirement. The zero-dependency callable works with any stack.

Run the production example · Read the measured result

What it does not prove

TRUSTWORTHY means the supplied cases produced stable, non-collapsed evidence at the declared tolerance and satisfied any declared decision contract.

It does not prove:

• that each decision was correct or safe
• that every code branch or behavioural boundary was tested
• that several cases are semantically diverse
• that an open-ended answer is high quality

It also does not store traces, host a dashboard, or monitor production traffic. Static coverage, Promptfoo or DeepEval quality checks, and production observability remain separate parts of the stack.

Go deeper

• Which agents fit, and what the result does not prove
• Why arbitrary rerun counts fail
• How to read and budget per-route evidence
• Reuse Promptfoo, DeepEval, or generic evidence without duplicate calls
• Qualify a categorical LLM judge without confusing stability with validity
• Integrations and AgentCore validation
• API guide
• API stability and path to 1.0
• Security and data handling

Read the design story: Introducing AgentVerity: What Does a Green Agent Test Prove?

Development

pip install -e ".[dev]"
python -m pytest -q
ruff check .

CI covers Python 3.10 through 3.14, lint, package construction, and the generated README evidence. A coverage job enforces at least 90% statement coverage, and the branch-protection CI gate requires that job to pass.

Status and licence

Alpha. Pin a minor series for production use, for example agentverity~=0.13.0. Patch releases preserve the public API.

Apache-2.0. Contributions are welcome through the pull-request workflow.