agent-reliability-harness 0.2.1

Local-first policy and trajectory-regression harness for tool-using AI agents: deterministic trace validation, policy-as-code trajectory rules, baseline-vs-candidate regression gates, and CI-ready JSON/Markdown/JUnit/SARIF reports.

Agent Reliability Harness

A provider-neutral, local-first reliability and regression-testing harness for tool-using AI agents. It validates recorded agent execution traces against declarative policies (tool-call contracts, trajectory rules, budgets, unsafe patterns, grounding), produces deterministic findings with stable rule IDs, and compares candidate runs against a saved baseline so CI fails only on real regressions.

The core has zero runtime dependencies, makes no network calls, sends no telemetry, and produces byte-identical reports for identical input.

Why this exists

An agent can reach the correct final answer while still behaving unreliably. It may:

• select the wrong tool, or skip a required one;
• use correct tools in the wrong order;
• pass invalid or out-of-range arguments;
• repeat an irreversible action (double refund, duplicate email);
• ignore a tool failure and answer anyway;
• get stuck in a retry storm;
• stop before finishing, or fail to terminate;
• return facts without citations;
• leak secrets or propagate prompt-injection text;
• blow past latency, cost, or token budgets;
• regress silently after a model, prompt, or tool change.

Final-answer evals miss all of this. This harness evaluates the trajectory, and turns each of these concerns into a versioned policy rule with a stable ID.

60-second example

python -m pip install -e .   # not yet on PyPI; install from a checkout

arh validate --policy samples/policy_trajectory.json \
  samples/traces/refund_workflow_pass.json \
  samples/traces/refund_workflow_double_refund.json

The second trace ends with a perfectly plausible final answer — and still fails, because the trajectory was unacceptable:

[FAIL] refund-workflow-1002  (agent=refund-copilot, workflow=refund)
  score=84.5/100  policy=refund-workflow-v1  latency=1300ms  cost=$0.0080
  - [ERROR  ] [flow     ] ARH-FLW-003: duplicate side effect: tool 'issue_refund' was called again with identical arguments after already succeeding (step=s3)
  - [ERROR  ] [sequence ] ARH-SEQ-001: required tool 'check_refund_eligibility' was never called
  - [ERROR  ] [sequence ] ARH-SEQ-004: tool 'issue_refund' was called 2 times, above max_calls of 1
  - [ERROR  ] [sequence ] ARH-SEQ-003: call order violation: first call of 'lookup_order' must precede first call of 'issue_refund' (step=s1)

Exit code 0 when everything passes, 1 on failure, 2 on usage errors — usable as a CI gate as-is.

Regression gates

Save a baseline once, then gate future runs on changes, not absolutes:

arh validate --policy samples/policy_trajectory.json \
  samples/traces/refund_workflow_pass.json \
  --json-out baseline.json --quiet

arh compare --baseline baseline.json \
  --policy samples/policy_trajectory.json \
  samples/traces/refund_workflow_pass.json \
  samples/traces/refund_workflow_double_refund.json

GATE: FAIL
  - 4 new error finding(s): refund-workflow-1002:ARH-FLW-003, refund-workflow-1002:ARH-SEQ-001, refund-workflow-1002:ARH-SEQ-003, refund-workflow-1002:ARH-SEQ-004
  - added trace 'refund-workflow-1002' fails

The gate fails on new error findings, pass→fail transitions, and added failing traces — resolved findings and unchanged failures don't re-alarm. Gate modes: --fail-on regressions (default), failures, never; add --max-score-drop for score-based gating. See docs/regression-testing.md.

What it checks

Seven deterministic categories; every finding carries a stable rule ID (ARH-<CATEGORY>-<NNN>), an expected/observed pair, and remediation guidance:

Category	Rules	Examples
schema	ARH-SCH-001..010	unlisted tool, missing/mistyped args, enum/pattern/range violations
budget	ARH-BUD-001..007	step/total latency, cost, and token budgets, plus unverifiable-budget warnings
safety	ARH-SAF-001	prompt-injection phrases, secret-like strings (policy regexes)
grounding	ARH-GRD-001..003	missing citations, low coverage, malformed citation URLs
sequence	ARH-SEQ-001..005	required/forbidden tools, partial-order violations, call counts
flow	ARH-FLW-001..003	ignored tool errors, retry storms, duplicate side effects
completion	ARH-CMP-001..002	missing final response, failure to terminate

Full reference: docs/rules.md. Ordering rules are a deliberate partial order (constraints), not an exact golden-trajectory match — agents can legitimately reach a goal via different paths.

Trace formats

arh validate accepts (auto-detected, or forced with --format):

• arh — the canonical JSON trace format (TRACE-SPEC.md);
• openai-chat — OpenAI Chat Completions message lists with tool_calls;
• anthropic-messages — Anthropic Messages conversations with tool_use/tool_result blocks.

Adapters never guess: fields a transcript format cannot carry (latency, cost, tokens) are left unset, which marks the dependent checks not applicable instead of silently passing them. Unparseable input is recorded in the trace's metadata, not dropped. See docs/adapters.md.

Reports

Console, JSON (doubles as the regression baseline), Markdown (PR-comment ready), JUnit XML, and SARIF 2.1.0 (GitHub code-scanning annotations). All renderers are deterministic: identical input produces byte-identical output — no timestamps, no randomness, no clock reads.

CI

- uses: felmonon/agent-reliability-harness@main
  with:
    policy: policies/agent-policy.json
    traces: traces/*.json
    baseline: baselines/main.json   # optional: gate on regressions

The action writes JSON/Markdown/JUnit/SARIF reports, appends the Markdown summary to the workflow step summary, and fails according to the gate. It uses no secrets and is fork-safe. Full reference and SARIF/JUnit upload examples: docs/ci.md.

Design principles

• Zero runtime dependencies. pip install adds nothing to your application's dependency graph.
• Deterministic core. No model calls, no network, no telemetry, no clock reads. Semantic (model-graded) evaluation is a planned, clearly separated optional extra — never hidden inside deterministic scores (ROADMAP.md).
• Additive schema evolution. v0.1.x traces, policies, and baselines work unchanged; unknown major schema versions are rejected loudly (COMPATIBILITY.md).
• Evidence over claims. The benchmark suite contains 34 seeded cases (28 expected-fail, 6 expected-pass controls) and measures detection: currently 34/34 correct, precision 1.0, recall 1.0, 0 false positives/negatives, byte-identical repeat runs, ~0.03 ms per trace. Those numbers cover seeded, deterministically detectable failures only — scope and limits are documented in BENCHMARK-METHODOLOGY.md, results in BENCHMARK-RESULTS.md.

Documentation

Doc	What it covers
docs/quickstart.md	Five minutes from install to a regression gate
docs/concepts.md	Traces, policies, findings, scores, baselines
docs/policy-cookbook.md	Copy-paste recipes for every rule type
docs/adapters.md	OpenAI/Anthropic transcript ingestion, field mapping
docs/regression-testing.md	Baselines, fingerprints, gates
docs/ci.md	GitHub Action reference and workflows
docs/rules.md	Every rule ID with remediation
docs/troubleshooting.md	Error messages and fixes
TRACE-SPEC.md / POLICY-SPEC.md	Normative formats
ARCHITECTURE.md / DECISIONS.md	Design and ADRs

Compatibility

v0.1.x trace and policy files remain accepted and preserve verdicts and finding messages. Two review-driven detection fixes are documented exceptions: nested safety scanning changes one sample score from 70.0 to 70.83, and strict telemetry validation rejects malformed values that v0.1 accepted. New report fields are additive. Details: COMPATIBILITY.md.

Development

python -m venv .venv && source .venv/bin/activate
python -m pip install -e ".[dev]"
python -m unittest discover -s tests   # full suite (194 tests as of 0.2.0)
ruff check src tests benchmarks
mypy src                               # strict
python benchmarks/run.py               # thresholds enforced

CI runs the suite on Linux/macOS/Windows across Python 3.11-3.13, plus packaging checks. Contributions welcome — see CONTRIBUTING.md.

Author

Felmon Fekadu GitHub

License

MIT