ATB vNext (engine)

A verifiable, offline, multi-mode evaluation harness for payment agents. Built on the live ATB Keystone binding: every step, score, and verdict is a content-addressed reference (sha256: + SHA-256(RFC 8785 JCS)), so a result recomputes byte-for-byte and is Falcon-signable, exactly like the ATB Pass Certificate.

What is here (Phase 1, staged, local)

• atb_vnext/refs.py — the trace model: step_ref (alias execution_ref) -> journey_ref (alias trust_query_ref) -> run_ref (atb_run_v2, the signed/diffable unit) + score_ref.
• atb_vnext/schema.py — atb-fixture-1 strict fixture schema + dependency-free validator.
• atb_vnext/graders.py — the five scores: outcome, trace, trust (judge hook), cost_latency, and the headline severity_weighted (unsafe success penalised by blast radius).
• atb_vnext/runner.py — run_fixture(fixture, transcript) -> recomputable atb_run_v2.
• atb_vnext/packs/ — priority-one fixtures: tool-misuse (3) + aml-edgecase (1).
• tests/ — 16 conformance tests (schema, determinism, byte-for-byte recompute, tamper divergence, severity weighting, REFER-as-safe-block, judge hook). All pass.
• goldens/ — per-fixture golden run_ref baselines (nightly regression anchor).

Modes and scores

See DESIGN.md for the full matrix (unit / workflow / persona / redteam / runtime / regression), the six scenario packs, the fixture schema, and the pass/fail rules.

Run

PYTHONPATH=. python -m pytest tests/ -q

Full feature set (all built + tested; live on VM4)

• 6 modes: unit, workflow, persona, redteam, runtime, regression.
• 7 packs, 15 fixtures: tool-misuse, aml-edgecase, persona, long-context, multi-agent, data-protection, runtime. All 7 OWASP LLM Top-10 categories the bench maps are covered.
• 5 scores: outcome, trace, trust (deterministic RubricJudge, or plug an LLMJudge), cost_latency, and the headline severity_weighted.
• Agent-driven run loop (client.py): run_agent / run_and_report drive any AgentAdapter through the fixtures, threading conversation history so stateful agents are testable. Reference adapters in agents.py (HeuristicPolicyAgent, NaiveProceedAgent, LiteralInjectionAgent).
• Runtime kill-switch (runtime.py): scores in-flight, halts on a severity breach.
• Replay from production (replay.py): turn a real bench_events session into a fixture.
• Dynamic persona simulation (persona_sim.py): an adversary that escalates on the agent's reply.
• Mutation testing (mutate.py): obfuscate an attack (base64 / zero-width / leetspeak) to expose brittle detection.
• Suite analysis (analysis.py): coverage report, flakiness detector, regression diff, cross-cutting safety invariants, differential agent comparison.
• Configurable risk policy (RiskPolicy): per-deployment severity weights + tolerances.
• Signed corpus (corpus/, verify with verify_corpus.py): 15 positives + 15 negatives, Ed25519-signed under the same trust root as the main JCS corpus.
• Live API (/agent-trust-bench/vnext/): packs, coverage, fixture/{id}, score (returns a Falcon-1024-signed, recomputable atb_run_v2 with the trust score populated).