DFAH-Bench

DFAH is a replay harness for tool-using AI agents. It asks a practical question: when an agent reaches the same decision more than once, did it also take the same observable path?

The package records versioned replay groups, qualifies whether their required channels are comparable, and reports decision agreement (DAR), tool-path agreement (TAR), the paired DAR–TAR gap, and expected review load. It measures repeatability and observable execution fidelity. It does not establish that a decision is correct, that latent reasoning is faithful, or that a system is safe.

The package is alpha software. Start with synthetic cases and shadow replays.

Install

From PyPI:

python -m venv .venv-dfah
source .venv-dfah/bin/activate
python -m pip install dfah-bench

Add OpenTelemetry support when you need it:

python -m pip install "dfah-bench[otel]"

For package development:

git clone https://github.com/ibm-client-engineering/output-drift-financial-llms
cd output-drift-financial-llms
python -m venv .venv-dfah
source .venv-dfah/bin/activate
python -m pip install -e ".[dev,otel]"

Python 3.10 or newer is required.

A two-minute, no-network run

The demo agent ships inside the wheel, so these commands work outside a source checkout and do not require an API key:

dfah check-agent \
  --agent dfah.demo:toy_agent \
  --episode-timeout-s 5

dfah run \
  --agent dfah.demo:toy_agent \
  --replays 3 \
  --episode-timeout-s 5 \
  --out .dfah/runs/toy-local-01

dfah analyze .dfah/runs/toy-local-01 \
  --report .dfah/runs/toy-local-01/report.html

dfah inspect .dfah/runs/toy-local-01 --case CASE-001

The stable demo should finish with DAR=1.000, TARseq=1.000, gap=0.000, and flags/100=0.0. That is an integration check, not a claim that every agent or task should score 1.0.

Python API

from pathlib import Path

from dfah import Replay
from dfah.demo import toy_agent, toy_suite

report = Replay(
    suite=toy_suite,
    replays=3,
    seed=42,
    out=Path(".dfah/runs/python-quickstart"),
).run(toy_agent)

if report.metrics_available:
    assert report.tar is not None
    print(report.dar, report.tar.seq, report.gap)
else:
    print("metrics unavailable", report.ineligibility_reasons)

check_agent() is the first call to use with a real integration:

from dfah.testing import check_agent

conformance = check_agent(
    my_agent,
    max_cases=2,
    budget_usd=0.20,
    estimated_max_episode_cost_usd=0.05,
    episode_timeout_s=30.0,
    raise_on_error=True,
)

An integration implements the small Agent protocol and returns a typed AgentResult containing an observed trajectory, parse provenance, and a sanitized echo of the request that was actually sent. The package does not read ambient API keys, retry provider calls, or normalize requests behind the manifest.

What DFAH keeps explicit

• Missing or malformed required channels make a replay group ineligible. Unavailable aggregates render as — and serialize as null; they never become zero agreement or zero divergence.
• suite_version, fixture hashes, tool-schema hashes, request settings, and implementation provenance are part of the comparison contract.
• Empty observed paths remain different from missing paths.
• Strong trajectory identity can include canonical argument and result hashes without placing raw values in reports.
• Run plans are immutable, episode commits are append-only, and resumability does not resend an already committed episode.
• Cost admission is conservative after dispatch, and shadow sampling reports both estimated cost and expected flags per 100 cases.
• The optional OpenTelemetry integration emits GenAI spans without prompts, arguments, or tool results. Normalized decision labels and tool identities remain observable metadata and should be reviewed before export.
• The pytest plugin lets an existing test suite load a verified report and enforce project-specific replay gates.

Research artifact versus package

The repository contains two complementary layers:

• bench/ and the checked-in replay corpus reproduce the published DFAH-Bench preprint.
• src/dfah/ is the prospective package for new integrations and new replay captures.

The package does not rewrite historical logs or silently mix old and new studies. Its built-in suites validate integration plumbing; they are not financial-accuracy benchmarks.

Replay, review, and retest

Supply a policy when a run should enforce thresholds:

dfah run --agent package.module:agent \
  --mode blocking --policy gate.yaml --episode-timeout-s 30

Blocking mode without a policy is a configuration error. A failed gate keeps the report and exits unsuccessfully. In pytest, an explicit --dfah-policy is enforced before collection, even if no test uses a DFAH fixture.

The bounded replay-and-review example evaluates two explicitly versioned local candidates under one fixed policy: the first changes tool paths and fails; the corrected candidate passes. It keeps both evidence sets so the change can be reviewed and retested.

Export replay evidence

Version 0.1.2 includes a local exporter for the Every Eval Ever v0.2.2 interchange schema:

python -m pip install "dfah-bench==0.1.2"
dfah export .dfah/runs/MY-RUN \
  --format every-eval-ever \
  --out .dfah/exports/MY-RUN

The exporter writes local files only. It requires an artifact-verified run with at least one eligible replay group, hashes arbitrary request settings, and excludes prompts, raw tool arguments, raw results, and reasoning traces. Model/provider/adapter identifiers, normalized decision labels, tool names, and equality hashes remain metadata; review them before sharing an export.

The exporter works on all supported Python versions. Optional upstream validation requires Python 3.12 or newer and dfah-bench[eee]; the extra pins the upstream 0.2.3rc1 validator for schema 0.2.2. See the export guide.

Guides

• Quickstart
• Bring your own agent
• Production rollout
• Design decisions
• Maintainer release process
• Research paper

The recommended rollout is simple: qualify the adapter, run sampled shadow replays, inspect the review queue, and only then decide whether a gate should block promotion.