VeriHarness -- Harness-of-Harness

VeriHarness — an evidence-bound Harness-of-Harness for verifiable long-horizon agent development. hoh is the command-line package, module, and CLI that implement the Harness-of-Harness pattern; VeriHarness is the project and repository built on that pattern, not a second name for the same thing. VeriHarness sits on top of Herdr, a terminal multiplexer for coding agents, and drives a plan → develop → independent QA → evidence → accept/reject → replan loop around agent-written changes, so that acceptance rests on receipts a deterministic runner produced, not on a model's own report that its work is done. docs/EVIDENCE_MODEL.md makes the full case for why that distinction matters; this file is the map of the rest.

What this actually is

A specification goes in. A planner role turns it into a development plan with machine-checkable acceptance criteria. A developer role implements against that plan, in an isolated worktree it is the only writer to. An independent QA role re-executes the criteria against a frozen, isolated copy of the result and reports verdicts -- verdicts that HoH itself re-checks against the runner's own receipts before trusting them. A candidate is accepted only if every criterion passes and at least one of them is proven to have been genuinely red on the state before the change. A rejected candidate goes back into planning with its evidence attached, not into the trash.

docs/ARCHITECTURE.md draws the exact lines: what Herdr owns (sessions, panes, worktrees, restore) and what HoH owns (the plan, the evidence, acceptance, replanning, and objectives that span multiple runs).

Running it: two commands, on purpose

hoh start --repo <worktree> --spec <spec.md> --run-id <id>
hoh run <id> --iterations 2 --planner claude --developer claude --qa claude

hoh start only creates the run record. hoh run is the command that actually dispatches roles and spends quota -- and it is a separate, deliberate invocation for a reason, not an accident of the CLI's design.

Automation is policy-controlled, not implicit

HoH does not switch itself on. There is no background service that intercepts agent work, watches for something worth planning, and starts spending quota on it. Nothing happens until something explicitly calls hoh start and then hoh run, naming a specific repository, a specific specification, and a specific run.

That is a statement about implicit automation, not about automation as such -- and the distinction matters, because this project's own development depended on it. There are two ways to operate the loop, and both are intended:

Manual CLI mode. A person types the two commands. This is what the examples document and the right way to start.

Orchestrated mode. A long-running agent session drives the loop: it writes and revises specifications, starts runs, reads the resulting verdicts and receipts, decides what to merge, executes the global gates, turns gate failures into new repair runs, and repeats until the whole set closes. The kernel cannot tell the difference -- it is the same CLI either way -- but the horizon is, because a session that can read a rejection can also plan the next run from it.

Implicit automation is refused while orchestrated automation is not, and the reason is consent rather than capability. HoH costs three role runs per iteration plus one control run per new acceptance criterion. Something that quietly spends money in the background is exactly the surprise the rest of this design guards against. An orchestrator spends the same money -- but it does so because someone delegated that authority for a scope they named. Herdr exposes event hooks one could wire a trigger to; whether a given piece of work is worth the recurring cost is a policy decision, and this design's position is that the decision must be somewhere explicit, not that it must be re-made by hand every time.

Which decisions an orchestrator may take on its own is itself policy, not architecture. In the campaign described below, a small number were reserved for a human, and irreversible external actions -- publishing to a public remote, for one -- still are. Nothing in the design requires that particular split. A different deployment can delegate more, or less.

Four layers, and what each one decides

The orchestrator is not hidden inside src/hoh/. It is a deployment layer that drives HoH through the same CLI and Herdr interfaces anyone else would. docs/ARCHITECTURE.md sets out what each layer may and may not do.

What has actually been exercised

VeriHarness was developed and released using VeriHarness, in orchestrated mode. Not as a demonstration assembled afterwards: the loop below produced this project's documentation, its position paper, its claims ledger, its export and release tooling, and the repair work that followed its own gate failures.

Measured on the campaign that produced this release, derived from the run records in the development repository. Those records are deliberately not part of this export -- they carry working material, not product -- so the figures are stated here rather than linked:

The findings are the point of that table more than the run count is. A campaign that produces 127 recorded findings against its own harness is not a campaign that went smoothly. The findings themselves live in the development repository's working ledger, which does not ship; what ships is the subset that became a standing limit, written out in full in docs/LIMITATIONS.md rather than summarised. Limits 8 and 9 there are both findings from this campaign that were never fully closed.

Who wrote what

Three different authorities wrote to this repository, and collapsing them into "the loop wrote itself" would be false in both directions -- it would overstate the kernel and erase the work that happened above it.

Measured over the non-merge commits reachable from the mainline of the development repository, at commit 30ff053 -- a merge commit is the taking of a run branch, not a direct write, so counting one as "written directly" would invert what it shows. The published repository is an export with its own, much shorter history; these counts do not describe it and are not reproducible there.

Naming that commit is not pedantry, and the first published version of this table got three of its five rows wrong for want of it. The table counts commits to paths the table itself lives in, so the commit that writes these numbers changes them, and a count taken "now" is stale the moment it is written down. A measurement of a tree has to name the tree.

The paper/ row deserves its own sentence, because it changed while this release was being prepared. Nine of its ten commits came from runs: the position paper was written by the loop, not by hand. The tenth is the correction that produced this release -- sections 1.1 and 1.2, written directly by the orchestrating session, outside any run. Until that commit the column read zero, and saying so now rather than leaving the older, tidier number in place is the same discipline the rest of this section is about.

The tools/ row is the one worth pausing on: all three of this project's release-gate tools -- check_claims.py, export_manifest.py and union_gate.py -- were created by runs, not written by hand, and so were the three test files that guard them (tests/test_claims_anchors.py, tests/test_export_manifest.py, tests/test_union_gate.py). The machinery that decides whether this release may ship was itself produced through the loop it gates.

The single src/hoh/ commit that arrived via a run branch added packaged policy data -- the guard pattern lists under src/hoh/policy/ -- and no Python. No run wrote HoH's own logic. A normal run's specification named the paths it was allowed to touch, and HoH's source was excluded from that set. Core fixes were therefore made deliberately outside the run boundary, at the orchestrator level. Where those fixes repaired a defect this campaign found in HoH itself -- eleven core defects and five guard gaps -- each was declared as an exception and carries a test; the remainder of the 48 direct commits is ordinary development that predates or sits outside that accounting, and this README does not claim otherwise. All of it is a considerably weaker claim than "the system repaired itself", and that is the point of stating it this way.

Measured against a plain agent: the matched-budget benchmark

For most of this project's life docs/LIMITATIONS.md limit 2 said there was no baseline of any kind, so every claim about what the harness was worth was a claim about its own internals. There are now three campaigns, and none was edited in the light of another. The full protocol is docs/BENCHMARK_PROTOCOL.md, frozen before the first cell ran; the results are docs/BENCHMARK_RESULTS.md, _v2.md and _v3.md.

Five small tasks, three arms -- A a plain agent with no harness, B one hoh run, C the full control plane with global gates and repair nodes -- a hidden test suite per task that decides the verdict and that no arm sees, and a budget matched on the scarce resource, role dispatches. The headline metric is the false accept: the arm's own checks green, the hidden suite red.

Campaign v3 was pre-registered in full -- 5 tasks x 3 arms x 3 repetitions, unconditionally -- and the registration's exact bytes were bound in a git commit 12.8 seconds before the first cell started. That is established from git objects in docs/benchmarks/v3/PREREGISTRATION_PROVENANCE.json and re-derived by tests/test_prereg_provenance.py on every suite run, rather than asserted.

Read this the unflattering way, because it is the correct way. On these five tasks the plain agent was cheaper and at least as correct. Arm C's zero false accepts is not a correctness result: a false accept requires an arm to claim it is finished and be wrong, and arm C never claimed it -- it spent its nine dispatches, met a red global gate, spawned a repair node, and had the repair refused for want of budget, fifteen times out of fifteen. In v2 the same arm closed five of five, on eighteen dispatches per cell against a declared nine; the budget is now enforced per cell and shared with the repair nodes, and under that rule this control plane does not close on these tasks.

Arm B's one false accept is the finding the benchmark exists to produce -- and it reproduced v2's: the same task, the same arm, a normalisation that dropped a character. Three repetitions carry no rate; what they carry is that it was not a one-off.

benchmark_v3 = PASS on the readiness board means the campaign was pre-registered, commit-bound, complete at 45/45, matched-budget-valid, and reproducible from the raw cell files by a second, independently written aggregation that never imports the reporter. It does not mean the harness performed better.

The positive path is measured separately, because a ceiling that refuses too early is indistinguishable from one that refuses correctly if you only ever measure the case where it binds. At a budget taken from what v2's arm C actually spent -- eighteen, decided before the run, never raised afterwards -- the control plane reaches a real CLOSED fixpoint through a full repair cycle: 18 of 18 declared dispatches, 9 primary and 9 repair, one repair node, zero human decisions. The first attempt at that test failed with zero provider calls, and that artifact is kept beside the result.

What it demonstrably does

The benchmark says what the harness did not do. This is what it did, each item bound to something you can re-run.

It makes "green" mean something, and proves it by failing on purpose. Five release-critical metrics carry a falsifier -- a deliberately broken build the instrument must catch, or the metric stays NOT_RUN. The dispatch-budget instrument runs eleven controls at three ceilings ([9, 8, 4], at least one of which must force the refusal inside an iteration, or the tool refuses to run) plus two falsifiers that delete the enforcement. That rule exists because an earlier version of the same instrument passed seven of eight controls against a build with the enforcement removed: five of its controls were reading the test fixture's own loop guard rather than the product's refusal. Two reviewers tasked to refute it found that before campaign v3 was frozen.

It refuses states that are conventionally rounded up to a pass. NOT_RUN, NOT_DETERMINABLE, UNSUPPORTED_ENVIRONMENT and INVALIDATED are separate states, and none of them collapses into a pass. The public CI's sandbox job is green with its relevant step skipped, because GitHub's runners cannot create the namespace -- reported as UNSUPPORTED_ENVIRONMENT, never derived as a pass from the green job. A fresh clone of this repository reports its passes and its environment-gap skips as two separate numbers that are never summed, each skip naming the withheld artifact and why. At the v0.1.0 tag that is 1220 passed and 30 skipped in a --depth 1 clone, which is what CI makes, and 1222 passed and 28 skipped in a full one -- two of the skips exist only when the history is shallow, so the number is quoted with the clone it was measured in rather than on its own. A skip is authorised only when EXPORT_MANIFEST.json declares that path excluded for the reason the test expects; missing-and-included fails, missing-and- unclassified fails, missing-manifest fails, wrong-reason fails. Five negative controls hold that rule in place, because the first version of it would have turned a lost public file green by skipping instead of red.

It binds acceptance to receipts it did not write. A candidate is accepted only when a receipt a deterministic runner produced supports every criterion, and only when at least one criterion is proven to have been red on the state before the change. The planner's capability boundary is enforced by digest rather than requested in a prompt, and the instrument that measures it plants seven violations into throwaway copies and must catch each one -- 7 of 7.

Every number in the public documents is anchored. CLAIMS.json binds each number-bearing sentence in README.md and docs/** to a claim with evidence, or to an explicit not_claims entry with a reason. python3 tools/check_claims.py check all fails on an unbound sentence, on an anchor whose content changed, and on two claims that share one resolution sentence. This paragraph is itself subject to it.

And it found these things in itself. The list is the product more than the software is: a benchmark whose reported cost was a constant written beside the result rather than a measurement; an enforced ceiling off by one, where max_dispatches=9 bought eight provider calls because the counter incremented before the budget was consulted, with an existing test that passed against it because it only checked that something raised; a dispatch log counted one line per call, wrong in both directions; a budget a restart refunded; a capability witness that could be made to absorb a hostile write during a retry; a published claim that told readers how to verify it and did not survive that verification (167 of 329 failures in an export tree carried no environment-gap marker where the document promised none would); and three release gates that had no state in which they could fail -- one of them asking a remote the internal tree deliberately does not have, another comparing a tree against itself.

Two more were found while writing this release, and they are here for the same reason the others are. A sentence in the position paper carried a correct count with a command beside it that computed something else -- lines rather than distinct ids -- so the figure could not be reproduced the way the paper told a reader to reproduce it. And the export's reference checker, which is supposed to catch a published document pointing at a file the export does not carry, harvests paths only from backticked spans and markdown links: sixteen new table rows naming internal evidence trees in bare form passed it green, and the one exemption that existed before them had only ever been caught because the same path happened to appear backticked elsewhere in its row. The first is fixed; the second is fixed for the references at hand and its detection gap is recorded as a tracked defect rather than patched in the hour before a tag.

The full board of what is verified, what is advisory and what is UNSUPPORTED_ENVIRONMENT, with the exact command that re-derives each row, is docs/READINESS.md.

The four real hurdles to using this

Adopting HoH is not free, and pretending otherwise would undercut the whole point of an evidence-first project. Four things stand between "I have an agent" and "I have HoH running":

Honestly stated: HoH is currently a tool for someone who already wants this discipline -- specification-first, evidence-bound acceptance -- and is willing to pay its running cost, not one that makes that discipline easy or cheap to adopt. For a research preview, that is the right thing to say about it. Claiming general, low-friction usability would not be.

See it work -- including a rejection

examples/minimal/ is a complete, small specification you can run through the real loop yourself, plus a recorded rejection copied unmodified from this project's own dogfood evidence: a real candidate, a real failing preservation check, and the receipts that show why it was rejected rather than accepted. Read examples/minimal/README.md first if you only have a few minutes -- watching the loop refuse a candidate, with the evidence to back that refusal up, explains more about what HoH is for than a paragraph of prose can.

The documents

Read docs/LIMITATIONS.md before deciding this is production-ready for anything you care about. It names, among other things, that the guard around acceptance-check commands is a tripwire against accidents rather than a security boundary, and that multi-day orchestrated operation is demonstrated while long-duration operation without any human intervention is not.

Existing internal documents

This project's original internal working documents -- planning and operational notes written before this repository took its public form, in German -- remain exactly as they are and stay on record. docs/OPERATIONS.md is their English successor for operational purposes, not a replacement -- both stay on record.