Dankaro Agent Quality Harness

The harness evaluates the result of an AI agent before that result is released. It treats the following as release failures that can be measured: claims without sources, comparisons without a declared baseline, evidence that cites itself, excessive complexity, results that change between runs, contradictions and missing deliverables.

The central design rule is that a filled field proves nothing on its own. A source field that contains text is not evidence that the text points to anything a reader can check.

Gates

By default the gates are conjunctive: every gate must pass on its own, or the release is blocked. The harness also reports a weighted overall score so that you can follow a project over time, but that score cannot block a release by itself. A release should only fail for a reason that a specific gate names. To let the overall score block as well, set conjunctive = false in thresholds.toml.

Install

pip install dankaro-agent-quality

The package has no dependencies beyond the Python standard library and needs Python 3.11 or later.

Command line

dankaro-quality --version
dankaro-quality init                        # write thresholds.toml to edit
dankaro-quality evaluate result.json        # full report in Markdown
dankaro-quality evaluate result.json --format json --out reports
dankaro-quality validate a.json b.json      # one line per file

evaluate and validate exit with 0 when every result passes and 2 when any result is blocked. The thresholds come from the file named with --settings; without that option, from ./thresholds.toml if it exists; otherwise from the defaults shipped with the package.

Working on the harness

python -m pip install -e .
python -m unittest discover -s tests -v
dankaro-quality evaluate fixtures/passing.json

What counts as a source

The evidence gate requires a locator, not just a string. A source such as x or trust me is blocked. The following forms are accepted:

https://example.com/report          measurement://prototype/workflow-v2
runs/2026-08-06/RUN_REPORT.md:24    docs/report.md
$ pytest -q tests/test_pipeline.py  Companies House 05662277
sheet 12                            a1b2c3d (commit)

A source that refers to the agent's own earlier output, such as "our earlier draft" or "as stated above", is rejected as circular by both the evidence gate and the provenance gate. A number in an earlier draft is not a source. It exists only because someone wrote it down before.

The baseline rule

A comparison must state what it was compared against, and at least one comparison must be against a naive baseline. Examples of naive baselines are uniform rotation, the base rate, the hand set prior, an empty field, the most recent item and the most popular item.

AgentResult(
    output="The scheduler outperforms the alternative.",
    comparisons=(
        Comparison(
            subject="thompson scheduler",
            baseline="uniform rotation",
            baseline_kind=BaselineKind.NAIVE,
            metric="fund multiple",
            effect="+3.94x",
            sample_size=40,
            tests=("paired t", "Wilcoxon signed-rank", "sign"),
        ),
    ),
)

This gate exists because of a measured case. In a simulation, a policy compared only against a plausible alternative won by 2.08 times (t = 2.08). When a third policy was added that read no input and rotated uniformly, the first policy never beat it at any coverage level tested: +0.66 (p = 0.23), −0.69 (p = 0.52) and +0.64 (p = 0.58). The original comparison was not wrong. It was incomplete, and the missing baseline reversed its meaning.

The claims contract

Every substantive statement in a result is a Claim.

• A fact needs a source that resolves, a confidence value and a verified flag.
• An inference, a judgment or a recommendation needs a rationale.

The gates also read these fields:

The verified flag separates "I checked this" from "this is how it usually works". Recalled facts are allowed, but they cannot make up most of the evidence, and they cannot be the only support for a recommendation.

Stability measures content, not wording

The stability gate compares keyed claims across repeated runs. The same key with a different value counts as drift and blocks the release. The same key and value in different words does not count as drift. A result that contains no keyed claims is reported as not comparable, rather than as stable, because a run that cannot be measured has not passed.

The claim ledger across runs

Checking for contradictions inside one result does not catch the failure that happens in practice: a claim disproved in March reappears in a June report, because the June run never saw the correction. To catch it, point the harness at a claim ledger that persists between runs:

[global]
claim_ledger = "memory/claims.jsonl"   # leave empty to search upwards for memory/claims.jsonl

The ledger holds one JSON object per line. Each object has a status of live or demoted, and may have a watch regular expression:

{"id":"CLM-1","text":"Thompson beats greedy by 2.08x","status":"demoted",
 "reason":"Never beats round-robin; the gain belongs to breadth",
 "demoted_by":"review/verify_baseline.py","watch":"2\\.08x"}

A run whose output matches the watch pattern of a demoted claim is blocked. If the new result names that claim in supersedes, it is allowed through, because an explicit replacement is legitimate. A missing or malformed ledger is treated as empty and never causes an error.

Simplicity limits

AgentResult(..., exposed_controls=5, required_decisions=3, configuration_steps=1)

The limits are set in thresholds.toml: by default at most 7 exposed controls, 5 required decisions and 3 configuration steps. Keeping them in a settings file makes simplicity an engineering constraint that the team agrees once, rather than a design opinion that is argued again in every sprint.

Hooks and adversaries

hooks = HookRegistry()

@hooks.add_pre_run
def require_owner(task):
    return [] if task.metadata.get("owner") else [Finding("OWNER", "Task owner missing")]

An error from a hook that runs before the agent prevents the agent from running. Hooks that run afterwards, and adversaries, contribute to the release decision. Model based review should stay secondary to the deterministic checks: a critic model may point out a problem, but it should not be the only authority on evidence, required deliverables or numeric thresholds.

Enforcing the gates with a decorator

@quality_gated(harness, repeats=3)
def product_agent(task):
    return AgentResult(...)

A blocked run raises RuntimeError. The full structured report is available on exception.quality_report.

Regression fixtures

Each file in fixtures/regression/ records a case that an earlier version of the harness passed when it should have blocked it. If one of these fixtures starts to pass, a gate has been weakened.

Repository map

src/dankaro_quality/
├── contracts.py      task, result, claim and comparison types
├── sources.py        rules for what counts as a source
├── ledger.py         the claim ledger that persists between runs
├── harness.py        orchestration and the release decision
├── hooks.py          hooks before and after a run, and adversaries
├── adversaries.py    deterministic critics
├── reporting.py      JSON and Markdown reports
├── serialization.py
├── cli.py
└── evaluators/       one module for each gate