agentverity 0.6.0

Measure-first testing for non-deterministic LLM agents

agentverity

Can you trust this green agent test?

AgentVerity is a pre-flight testing library for non-deterministic agents. It checks whether the verdict is stable across identical reruns and whether the probe set can move the decision at all. Then it tells you which test result is safe to trust.

It does not replace DeepEval, promptfoo, AgentCore Evaluations, or your quality metrics. It catches two conditions that can make their result misleading before you read the score.

This report is regenerated from the executable bugfix_pipeline.py example and tested against its current results.

Try it

pip install agentverity

from agentverity import from_callable, run

def route(ticket: str) -> dict:
    # Deliberate defect: the input is ignored.
    return {"text": "route: general", "verdict": "general"}

agent = from_callable(route)
result = run(agent, inputs=[
    "my card was charged twice",
    "the app crashes on login",
    "where is my refund",
    "the checkout button is the wrong colour",
])

print(result.headline)

NOT TRUSTWORTHY - the agent answered 'general' on 100% of the probes,
so a pass says more about the probe set than about the agent.

The default balanced precision sizes the repeat count automatically. A default run answers rather than leaving a deterministic function as undecided.

From a repository checkout, the same example is directly runnable:

python examples/support_router.py

agentverity run \
  --agent examples/support_router.py:build_agent \
  --inputs examples/support_tickets.txt

What it catches

Condition	Why the green result is unsafe	AgentVerity's action
Blind probes	One verdict dominates, so relations can pass without crossing a decision boundary	Repair the probe set
Stochastic verdict	Identical reruns disagree, so zero-tolerance checks confuse noise with regressions	Use repeated rates against measured noise
Stable, varied verdict	The decision is suitable for a reviewed reference	Prefer an evidence-gated snapshot or targeted relations
Unexercised relation	The transform returned the original input, so no test happened	Report n/a, never a false pass

The meter reports deterministic, stochastic, or undecided from a Wilson interval over disjoint repeat pairs. The skew scan separately checks whether varied inputs produce varied verdicts. Stability is not correctness, so snapshots still require explicit human approval.

Where it fits

agent or workflow
       |
       v
AgentVerity pre-flight ----> quality evaluator
       |                    DeepEval / promptfoo / AgentCore Evaluations
       |
       +---- text for people
       +---- JSON for code
       +---- JUnit XML for CI
       +---- OTEL span for CloudWatch / Phoenix / LangSmith

AgentVerity is deliberately small at this boundary. The core has no runtime dependencies, no hosted account, and no dashboard to adopt.

CI reporting

Write a report that Jenkins, GitLab, Azure DevOps, and JUnit collectors already understand:

agentverity run \
  --agent examples/support_router.py:build_agent \
  --inputs examples/support_tickets.txt \
  --format junit \
  --output agentverity.xml

Exit codes and JUnit carry the same interpretation:

• 0: interpretable evidence and no relation violation
• 1: blind or vacuous probes, a relation violation, or snapshot drift
• 2: incomplete, undecided, or unsupported evidence

Monitoring

Use the host application's OpenTelemetry pipeline:

pip install "agentverity[otel]"

from agentverity import record_otel_run

result = run(agent, inputs=canary_probes)
record_otel_run(result)

The summary span contains low-cardinality agentverity.* attributes. It excludes prompts, outputs, fingerprints, relation names, and exception messages. OTLP can carry the span into Amazon Bedrock AgentCore Observability and CloudWatch, Phoenix, LangSmith, or another collector.

Run this in CI, before deployment, or as a scheduled canary. AgentVerity is not an online evaluator and should not repeat every customer request.

Integration examples and the AgentCore validation plan

Evidence-gated snapshots

Create a baseline only after reviewing the outputs:

agentverity snapshot \
  --agent mymod:build_agent \
  --inputs seeds.txt \
  --output baseline.json \
  --accept-reference

AgentVerity refuses to freeze a reference if calls failed, the verdict remains undecided or stochastic, the probes are blind, or approval is absent. It also rejects a mathematically impossible k before spending model calls.

Check the same approved cases later:

agentverity check \
  --agent mymod:build_agent \
  --inputs seeds.txt \
  --snapshot baseline.json

Current evidence must pass the same admission checks before differences are reported as regressions. Snapshot files store SHA-256 input fingerprints rather than raw prompts.

Observation layers

Every call becomes one Observation:

from agentverity import Observation

Observation(
    text="Escalating this case",
    verdict="escalate",
    tools=("lookup_order", "create_case"),
)

Measure the layer the test protects:

• verdict for routing, policy, and categorical decisions
• tools for the ordered tool trajectory
• text when wording itself is the contract

Token variation does not have to become verdict instability, and a stable answer can still hide a changed tool path.

Configuration

Most runs need two knobs:

from agentverity import RunConfig

config = RunConfig(
    precision="balanced",  # cheap=10%, balanced=5%, strict=1%
    budget=None,           # optional cap on meter calls
    max_workers=4,         # distinct inputs only
    error_policy="record",
)

k= and epsilon= remain exact overrides. Repeated calls for one input stay sequential, while distinct inputs can run concurrently. Recorded provider failures mark evidence incomplete and never become passing verdicts.

Examples

• support_router.py: the smallest blind-agent example
• bugfix_pipeline.py: blind triage plus a stochastic supervisor
• strands_example.py: optional Strands adapter
• otel_monitoring.py: one diagnostic span through OTEL

For Strands:

pip install "agentverity[strands]"

Plain LangGraph, remote APIs, and other frameworks can use from_callable by returning an Observation from their invocation wrapper.

Scope

AgentVerity is:

• a pre-flight check for the evidence behind an agent test
• an evidence gate for reviewed snapshots
• a small CI and telemetry citizen

It is not:

• a correctness oracle or LLM judge
• a benchmark or leaderboard
• a trace store, dashboard, or production monitoring service
• a claim that metamorphic relations or Wilson intervals are new

The distinctive part is their ordering: measure verdict stability and probe coverage before interpreting the ordinary test result.

Documentation

• Integrations and AgentCore validation
• API guide
• Design decisions
• Security and data handling
• Contributing
• Release process

Development

pip install -e ".[dev]"
python -m pytest -q
ruff check .

CI covers Python 3.10 through 3.14, lint, package construction, and the generated README diagnostic.

Status and licence

Alpha. Public APIs may change before 1.0.

Apache-2.0. Contributions are welcome through the pull-request workflow.