AgentVerity

Your agent test passed. Would it pass again?

AgentVerity is an offline Python library and CLI that qualifies repeated, categorical AI-agent evidence before it becomes a regression reference for future releases. It finds routes whose repeatability is rejected, weak decision coverage, and runs too small to support a conclusion. It does not judge whether an answer is correct.

The 60-second problem

A payment router sends disputes to six specialist queues. Promptfoo runs six reviewed cases 26 times, and all 156/156 assertions pass. One ambiguous case allows either of two valid fraud queues.

AgentVerity reads that same export and finds:

route              cases  pairs  flips  95% CI            result
card_security          1     13      8  [0.355, 0.823]    stochastic
cash_withdrawal        1     13      0  [0.000, 0.228]    undecided
duplicate_charge       1     13      0  [0.000, 0.228]    undecided

flip pairs:
  card_security <-> merchant_dispute  x8

The quality policy accepts both answers, but a reference that switches queues will make later regression checks noisy. The changing route is stochastic; the five quiet routes are undecided because 13 pairs are too few to certify them separately. A flip means the two observations in a paired rerun differed. AgentVerity therefore refuses this evidence as a regression reference.

Try it without model calls

git clone --depth 1 https://github.com/mrwersa/agentverity.git
cd agentverity
python -m pip install .
agentverity assess \
  --promptfoo examples/promptfoo_bridge/results.json \
  --suite examples/payment_decisions.json

assess performs arithmetic over recorded decisions. It makes no model or provider calls. You can also reuse precomputed DeepEval LLMTestCase objects or any ordered JSONL log:

agentverity assess --jsonl runs.jsonl \
  --input-path probe.text --decision-path result.route

Order matters because observations are paired in collection order. See imported evidence before converting a log.

To call an agent directly, install only the framework adapter you need:

pip install "agentverity[strands]"
pip install "agentverity[langgraph]"

Plain Python callables need no extra dependency:

from agentverity import from_callable, run


def route(text):
    return "billing" if "charge" in text.lower() else "cash_withdrawal"


agent = from_callable(lambda text: {"verdict": route(text)})
result = run(agent, inputs=["duplicate charge", "cash withdrawal"])
print(result.summary())

What it decides

AgentVerity keeps three API outcomes separate and explains them in repeatability terms:

• deterministic: repeatability qualified; evidence supports the tolerance
• stochastic: repeatability rejected; changes exceed the tolerance
• undecided: inconclusive; the evidence supports neither direction

These strings remain the public machine contract. deterministic does not claim that the underlying agent has zero randomness.

It then checks whether the probe set collapsed onto one decision and, when a decision contract is supplied, whether every required route was intended and observed. Per-route results show where changes concentrate. Optional relations check reviewed input transformations and report no-op transforms as untested, not passed.

Once you have two evidence windows, agentverity compare-evidence before.json after.json reports changed route conclusions, gained or lost decisions, changed flip pairs, isolation, and provenance.

Where it fits

AgentVerity is a local test and release step, not serving-path middleware. Use it for named routes, approvals, policy outcomes, tool choices, hand-offs, or a reviewed finite tool path. Use another evaluator for open-ended chat, RAG quality, generated content, or coding-agent output. If those systems also emit a bounded route or approval, AgentVerity can qualify that decision layer.

Why rerun counts are harder than they look

Three or five reruns by convention do not state what variation they can rule out. With no observed changes:

• 36 independent pairs bound the change rate below about 9.6%
• a claim below 5% needs 73 pairs
• a short quiet run is therefore undecided, not repeatability qualified

AgentVerity sizes calls from the tolerance, uses non-overlapping pairs, and places a Wilson interval around the flip rate. Optional sequential collection uses checkpoints declared before collection; it does not repeatedly inspect a fixed-sample interval and stop when the result looks favourable.

For evidence already collected, best_case_admission_pairs tests whether an all-agree continuation could admit within a predeclared pair budget. It may justify stopping an impossible run early; it never creates an early admission. For live fixed-endpoint collection, opt into the same bound with --curtail. It reports the stopping pair and avoided calls but no final repeatability class; a path that could admit still pays the full fixed budget and is classified only there.

Use agentverity plan --suite examples/route_stability_plan.json before spending remote calls. The decision repeatability method guide explains the arithmetic, and the validation artifact records exact-boundary checks and dependence stress tests.

The evidence gate

snapshot refuses a regression reference until calls complete, the evidence supports the declared repeatability and coverage policy, and a person approves the reference as acceptable. The bundled offline example shows why correctness alone is not enough:

python examples/payment_dispute_gate.py

Both sets score 6/6. Only the repaired set reaches all six required routes.

Real-system evidence is also committed and reproducible without new calls:

• The AgentCore canary validates a production-shaped integration while explicitly stopping short of per-route certification.
• The AgentKit study records 4,380 calls across three models and shows that the most repeatable model can be less correct.

Never repeat live customer requests. Use reviewed synthetic cases in CI, before release, or on a schedule.

What it does not prove

TRUSTWORTHY means the supplied cases produced repeatable, non-collapsed evidence at the declared tolerance and satisfied any declared decision contract. It does not prove correctness, safety, semantic diversity, complete behavioural coverage, provider independence, or production reliability. AgentVerity also does not store traces, host a dashboard, monitor traffic, or score open-ended answers.

Documentation

• Start: concepts, applicability, and limits, runnable examples, and the API guide
• Use existing tools: imported evidence, integration placement, and the importer conformance contract
• Framework recipes: qualifying Inspect AI epoch runs and qualifying repeated BFCL function-call runs
• Understand the method: decision repeatability, per-route evidence, and categorical evaluator repeatability
• Operate safely: security, data-retention audit, and API stability
• Project direction: design, roadmap, and the agent-evaluation landscape
• Participate: contribute or join the design-partner pilot

Development

pip install -e ".[dev]"
python -m pytest -q
ruff check .

CI covers Python 3.10–3.14, package construction, and at least 90% statement coverage. See the contributing guide above before opening a pull request.

Status and licence

Alpha. Pin the current minor series for production use: agentverity~=0.21.0. Patch releases preserve the public API.

Apache-2.0.