Earlier agent-auditing benchmarks include R-Judge, which evaluates safety-risk awareness from agent interaction records, and Agent Security Bench, which evaluates attacks and defenses for LLM-based agents. CatchBench differs in the lifecycle organization and shared interface, in the spirit of ADBench for tabular anomaly detection and BOND for graph anomaly detection.

The Full Task List

Each task is one auditor question at one information state, with the label that answers it, not a list chosen for coverage. Implemented in the current build:

• Fault localization (POST, which step): rank the steps of a failed run. Top-1 / Top-3 / MRR against Who&When human attribution.
• Failure detection (POST, did it fail): predict run failure. ROC-AUC against SWE-Gym and tau-bench resolved / unresolved outcomes.
• Cause attribution (POST, what kind): tell stale-state from dropped-grounding on a paired Gold injection (the same run injected both ways). ROC-AUC.
• Gold fault localization (POST, injection experiment): plant a known fault in a real run and localize it. Top-1 / Top-3 / MRR against injection-site labels.
• Streaming early warning (LIVE, can you tell early): flag a failing run from a growing prefix. Prefix-AUC and time to detection.
• Online stale-state detection (LIVE, catch it live): detect the Gold stale-state injection online. True-positive rate at a fixed false-positive budget.
• Over-privilege audit (PRE, is the declared harness safe): flag granted capabilities the task does not need. Compare an OWASP / CWE static-scanner set and a held-out LLM judge against the four constructed label processes. Precision / recall / F1 over 1187 configs from six corpora.

Planned:

• Missing-guardrail plan audit (PRE, is the plan safe): flag removed or weakened guardrails in a declared plan.

Fault Localization on Who&When

Rank the steps of a failed run by how likely each is the fault, scored against the human mistake_step. 126 failed runs, 1099 steps, 11% of steps are faults.

How to read it. The direct LLM control shows a failed trace to a model and asks it to name the decisive step. The 11-model panel uses one all-at-once prompt per run, following Who&When's protocol; predictions are cached and committed, so scoring the board makes no API call. GPT-5.5 has the highest Top-1 score here at 0.452. The panel spans 0.127 to 0.452, and eight of the eleven models sit in one band from 0.333 up, scored on 126 runs. Mistral-Small and Nova-Micro fall below the position prior on point estimate. Among methods that use no LLM, the supervised execution-feature ranker scores above the prior on Top-1 and Top-3 without an API call. auditable's blast share ties the prior at displayed precision because this corpus assumes every step depends on all prior steps. PyGOD DOMINANT has the only displayed Top-1 point estimate below chance, 0.048 against the 0.119 random floor. The benchmark uses separate structural methods in LIVE settings, where a full-trace judge cannot run. Separating a dependency signal from raw position requires traces in which long-range dependencies diverge from the corpus's full-context assumption.

Failure Detection on SWE-Gym and tau-bench

Predict whether a run failed, scored by ROC-AUC. The comparison asks whether the dependency structure of a run predicts failure beyond its raw size and event counts. Compare auditable (size+deps) against size (flat).

SWE-Gym, 376 runs (188 failed, 188 resolved):

tau-bench, 660 runs (363 failed, 297 resolved):

How to read it. The size-normalized dependency block scores above the size-and-counts baseline on both corpora (+0.142 on SWE-Gym, +0.046 on tau-bench). These increments use unrounded scores and come from the linear specification. The controlled audit in tools/detection_audit_results.json addresses sensitivity to the model specification: the dependency increment stays positive across all three tested specifications on tau-bench and changes sign on SWE-Gym. On tau-bench the structural block and full-feature reference tie at the displayed precision (0.665), so the full vector shows no displayed gain there. On SWE-Gym, PyOD ECOD exceeds the linear size model (0.765 over 0.663), and the dependency-structure method scores higher again at 0.804.

CatchBench runs a wider unsupervised arena behind the headline table: the PyOD tabular family (Isolation Forest, KNN, LOF, COPOD, HBOS) and the PyGOD graph family (DOMINANT, CONAD, AnomalyDAE, GAAN). On SWE-Gym the tabular detectors span ROC-AUC 0.319 to 0.625, all below the 0.663 size baseline. The graph family spans 0.547 to 0.850, and two of its members clear that baseline: CONAD at 0.750 and GAAN at 0.850. On tau-bench both families stay below the 0.619 size baseline, 0.504 to 0.593 for the tabular set and 0.490 to 0.552 for the graph set. GUARDIAN, the agent-specific reconstruction autoencoder, scores 0.767 next to ECOD at 0.765.

Read that SWE-Gym graph maximum carefully. GAAN's 0.850 is a single-seed number, and its five-seed range overlaps the supervised references. Ranking only within runs of exactly equal node count leaves it no advantage beyond run size on the matchable subset (tools/pygod_seed_stability.py). The same family fares worse on the other boards. DOMINANT lands under the random floor on Who&When localization, and every PyGOD entry stays below the size baseline on tau-bench. No off-the-shelf detector establishes a task-relevant board lead. The task-aware structural method sits close to the better ones on SWE-Gym: 0.804 against 0.765 for ECOD and 0.767 for GUARDIAN. G-Safeguard is the supervised graph comparator, with the highest supervised SWE-Gym point estimate at 0.828 and 0.824 +/- 0.007 over five cross-validation seeds.

Baselines and Lineage

The graph-AD baselines are ports of published methods onto the dependency graph, in the ADBench / BOND tradition of running a method on the benchmark's representation rather than gesturing at it. PyGOD's DOMINANT (Ding et al., 2019, Deep Anomaly Detection on Attributed Networks, SDM) is an unsupervised graph autoencoder that scores nodes by reconstruction error. GUARDIAN (Zhou et al., 2025, arXiv:2505.19234), which safeguards multi-agent collaborations with a reconstruction-error temporal graph autoencoder, is implemented here as a directed-GCN attribute-reconstruction autoencoder over the per-run graph; the explicit adjacency-reconstruction term and the information-bottleneck compression are simplified, as the code documents. G-Safeguard (Wang et al., 2025, arXiv:2502.11127) uses a GNN to detect anomalies on a multi-agent utterance graph; here it is implemented as a supervised graph-classification GNN over the dependency graph. Its 0.828 is the highest displayed value on the SWE-Gym table, above the 0.819 full-feature reference.

Gold: Injected Faults on Real Runs

The boards above borrow labels (human attribution, run outcomes). Gold plants a fault in a fetched clean run and asks whether a method points to it. There are 188 clean SWE-Gym runs with one injected fault each: 82 stale-state and 106 dropped-grounding. A run affords a stale-state fault only when it has an earlier superseded same-file event to redirect to. The label records the injection site by construction; it does not establish that the alteration is a realistic fault, and the file-level substrate has the construction artifact documented below. The numbers below are one representative seed, with mean and standard deviation across five injection seeds printed by run.py. Read the board per fault kind because the two injected mechanisms behave differently and the aggregate hides that difference.

The injector can target only steps that meet the precondition for the selected fault kind, so the full-pool table mixes fault localization with target eligibility. The eligible-pool control re-ranks each method only within the steps the injector could have picked for that run's fault kind, with a mean of 7.4 candidates per run. It matches eligibility, not exact degree. The random floor rises accordingly, and ties are averaged so a constant-score baseline lands on the floor instead of winning on sort order.

How to read it. In the full pool, max-span scores 0.703 on stale-state, above a 0.029 analytic floor, and has-dep scores 0.005 on dropped grounding against a 0.035 floor. Within the eligible pool, has-dep displays 0.350 on stale-state, degree displays 0.394, and max-span displays 0.805 against the displayed 0.350 floor. For dropped grounding, position displays 0.321 against 0.277. PyGOD displays 0.622 on stale-state, 0.236 on dropped grounding, and 0.404 overall, against max-span's 0.394 overall. These cells are point estimates.

The Injection: Where It Holds Up, and Where It Does Not Yet

The injection checks include both the measured signal and its known limitations:

• One measured localization mechanism. Stale-state max-span scores 0.703 against its 0.029 analytic floor for this seed, and 0.653 +/- 0.028 across five injection seeds. The dropped-grounding column carries no comparable margin: has-dep displays 0.005 against a 0.035 floor, and the highest cell there is the graph detector's 0.094.
• Leakage check, two levels. In the full pool, position, degree, and has-dep display 0.000, 0.045, and 0.078 overall against the 0.032 random floor. The latter two point estimates are above the floor. Ranking within the exact eligible pool controls target selection. On stale-state, has-dep equals the matched floor at 0.350, degree scores 0.394, and the dependency-span detector scores 0.805, with 0.795 +/- 0.020 across seeds. That controls selection, not construction: the clean substrate wires every file event to its immediate same-file predecessor, both injections break exactly that invariant at the injected step, and a broken-predecessor baseline (tools/gold_artifact_diagnostic.py) uniquely ranks all 82 stale-state and all 106 dropped-grounding targets Top-1 while flagging 0 of 188 clean runs, across five injection seeds. Both fault kinds therefore fail the no-artifact-leakage bar on the file-level substrate; read the Gold boards as mechanism diagnostics. Artifact-controlled evidence is not yet available.
• Distribution shifts. Stale-state preserves the valid dependency-edge count (mean 9.2, unchanged); dropped-grounding removes exactly one valid dependency edge (mean 7.9 to 6.9), so treat edge count as a reported run-level shift for the dropped half, not as matched. Stale-state also lengthens the run-level max dependency span by construction. The board's span line is not split by fault kind: across all 188 paired runs the mean rises from 8.6 to 9.4 and 53 increase, and only 82 of those runs carry a stale-state injection. Localization still requires finding the step.
• Constructed labels. The label records the step modified by the injector and is independent of detector output. It identifies the programmed modification, not a human-verified natural fault.
• Characterized caveat. SWE-Gym dependencies are inferred, not gold value-flow, so a redirected edge is a dependency-misattribution proxy for a true stale read. An artifact-controlled test requires a named-value corpus where writes and reads are explicit, plus a human-audited validation slice.

Cause Attribution and the LIVE Boards

Three more boards complete the POST and LIVE pillars; run.py prints their full tables.

Cause attribution (POST, what kind). Given a faulty run, is it stale-state or dropped-grounding? The substrate is paired, with the same run injected both ways, so run identity and eligibility are held fixed. The label records which injection was applied; it does not show that these two categories cover naturally occurring causes. On 166 paired runs the two injections leave opposite traces: a stale read lengthens the max dependency span (ROC-AUC 0.675, and 0.671 +/- 0.005 across seeds), dropped grounding removes an edge (edge-count 0.566), against a 0.498 floor. Each feature is keyed to one mechanism, so this measures discrimination within the injection design, not general cause attribution.

LIVE streaming early warning (can you tell early). Can a method separate failing from resolved runs from a growing prefix? On SWE-Gym at the 25% prefix the dependency-structure block scores 0.74 above the flat size-and-counts baseline's 0.63 point estimate. The 20 SWE-Gym cells read against the 0.70 bar were added after these scores were examined, so the paper reports them as exploratory and prints each cell's own interval. Random is a label-independent reference. The raw per-run span point estimate is 0.36 at 25% and is length-confounded. On tau-bench, no reported method's point estimate reaches the 0.70 time-to-detection threshold at any prefix.

The dashed line is the 0.70 warning bar, and each curve is one entrant's board ROC-AUC at the four prefixes. The panels carry point estimates; the paper prints each cell's own interval beside them. Random retains its x reference marker. The panels show a descriptive domain split, and no comparison across the two corpora is computed. The online per-run span is the lowest curve at the shortest prefix on SWE-Gym, and on tau-bench it sits just above the random reference there; it is the only setting that is genuinely online. Drawn by figure-src/board_live_prefix.py from tests/golden/board.txt.

LIVE online stale-state detection (catch it live). The same Gold stale-state injection is detected online at a fixed false-positive rate instead of localized post-hoc. At realized false-positive rates of 6.1% and 11.0%, the prefix-only span z-score displays true-positive rates of 0.061 and 0.110; raw span displays 0.122 and 0.159. Across five injection seeds, the corresponding means are 0.054 and 0.124 for the z-score and 0.098 and 0.151 for raw span. At the displayed 5% target, the dependency- count control and z-score each display 0.061, while raw span displays 0.122. These cells are point estimates. They support no claim about the effect of per-run normalization. These are displayed cells from 82 paired runs. The same clean runs calibrate and report each empirical threshold, and the five injection seeds reuse those runs rather than supplying 410 independent observations. The Gold 0.703 value scores post-hoc within-run localization, a different decision, so it is context rather than a cross-state effect estimate.

Over-Privilege Audit on Declared Harnesses (PRE)

Before a run, the available evidence comes from the declared task or role and the capabilities its harness grants. The PRE board asks whether a method can flag granted capabilities that the task does not need. It runs over 1187 configurations from six corpora (crewai, n8n, mcp, injecagent, sweagent, and a synthetic set), each declared capability labeled needed or excess, scored by precision, recall, and F1 over the flagged-excess set. The distributed PRE records contain declared-capability metadata, labels, and a spec_tokens list derived from the task or role, not the upstream prose itself.

All PRE labels are constructed. The crewai, n8n, and mcp labels are the intersection of two LLM judges' excess decisions, with overall Cohen's kappa 0.666; they remain subjective judgments with imperfect agreement. The injecagent roster relabel treats its designated user tool as needed and its attacker-tool roster as excess, so it evaluates those source roles rather than an independent need annotation. The sweagent declared-minus-used heuristic treats every capability not exercised in the observed trajectory as excess, although non-use does not prove lack of need. The synthetic labels are authored injections, so they measure planted cases rather than a natural deployment distribution.

The static baseline maps each rule to an over-privilege category from OWASP LLM06:2025 Excessive Agency, the OWASP Agentic Security Initiative (ASI), and the CWE privilege family:

owasp_asi_combined is their union. Three standard concerns are named but stay out of static single-config scope, and the board says so rather than overclaiming: a full excessive-autonomy check needs an approval-gate field the schema does not carry, so unrequested_high_impact is an approximation (a high-impact action the task never asks for); full ASI Tool Misuse needs declared operation, scope, and allowlist controls the schema does not express, so sensitive_access is a narrower LLM02 exposure heuristic; and a deprecated or duplicate extension needs deployment history. The board scores each rule and the union, so coverage is visible rather than asserted:

Coverage is the share of the 1187 configurations a method actually judged. Every rule answers all of them; the held-out judge abstains on 5, so its precision, recall, and F1 are computed over 1182. Rows on different denominators are not comparable cell for cell. The per-source section below says why the judge abstains and where.

How to read it. The combined OWASP/CWE scanner has the highest displayed rule-based F1 (0.654, 0.910 recall), against 0.695 F1 for the held-out LLM judge on the 1182 configs it answered. Its predictions include unnecessary read and unknown capabilities through the excessive-functionality rule, not just risky permission levels. The three rules make 37 privilege-escalation, 59 sensitive-access, and 676 unrequested-high-impact predictions over the full 1187-config board. Their low overall recall shows that each identifies a limited slice of the aggregate excess set. The corpus labels one undifferentiated excess set with no per-category annotation, so it cannot say how prevalent each standard category is, and these rules' precision is measured over those small-to-moderate samples rather than a category-level ground truth. Three rules display a higher precision cell than the judge's 0.594: owasp_privilege_escalation at 0.811, sensitive_access at 0.763, and unrequested_high_impact at 0.633. The board computes no comparison between a rule's precision and the judge's, and the judge abstains so it is scored on a different set of configurations, which is the same reason the coverage column exists. Read those three as printed cells, never as a ranking against the judge. The crewai, n8n, and mcp labels were made by two other judges (GPT-5.5 and Claude), so the Llama-3.3-70B baseline did not create its own evaluation labels. The scanners are keyword-based and language-limited: a task spec in a language outside the keyword lists falls back to the read-only floor and over-flags.

Read the board per source, because the four label processes differ and a pooled F1 hides it (the per-rule per-source numbers print from run.py):

Each row prints one source's best method beside that same source's floor, which is the only comparison the labels support. The floor itself moves by nearly a factor of five across the six sources, so a pooled F1 says little about whether a method earned its false alarms anywhere. Drawn by figure-src/board_pre_source.py from tests/golden/board.txt.

Label origin per column: crewai, n8n, and mcp carry cross-vendor LLM-judge labels (Cohen's kappa 0.666, data/pre/LABEL_QUALITY.md); injecagent is a roster relabel; sweagent is declared-minus-used; the synthetic set is injection. The held-out judge lands near the top on injecagent (0.990) and synthetic (0.972), where the labels are constructed, and lands between 0.36 and 0.74 on the judge-labeled corpora.

The injecagent column carries a shortcut, and it is worth stating plainly rather than leaving for a reader to find. The harvester wrote each configuration's legitimate tool first and the attacker tools after it, and the released records preserve that order. So on all 340 injecagent configurations the first declared capability is exactly the minimum and the rest are exactly the excess. A rule that reads nothing but list position, keeping the first capability and flagging the tail, scores 1.000 F1 there: 510 true positives, no false positives, no false negatives. That is above the 0.990 held-out judge and above every other method in the table. The same rule scores 0.446 on crewai and 0.106 on n8n, which is how we know the shortcut lives in that corpus rather than in the rule. tools/statistical_tests_results.json records it under reported_quantities.pre.declaration_order_leak. Read the injecagent column as a property of its construction, not as a difficulty ranking.

Using a different scoring judge avoids direct label reuse: either label-making judge would inherit perfect recall on the intersection labels and an inflated F1 by construction. This separation does not turn the constructed labels into human ground truth. In n8n, where the mean excess ratio is 0.084, the non-oracle methods in the table score from 0.095 to 0.411 F1.

The held-out judge answered all 1187 configs. Five of its replies named a capability whose spelling did not match the declared roster. The parser that built the committed cache is all-or-nothing, so one unmatched name throws out the whole judgment, and those five never reached the cache. The judge abstains on them. They leave the denominator rather than counting as silent negatives. A reply nobody could read is a fact about the parser, and scoring it as an empty prediction would charge the method for it. The judge's scores therefore cover 1182 of 1187 configs, the coverage 0.996 on the board above. The abstentions land in two columns: n8n is scored on 215 of 219 and mcp on 143 of 144. One of them is large, a 622-capability MCP server carrying 337 of the corpus's 2893 excess labels. A head-to-head claim between the judge and a rule therefore needs a common evaluable set.

Every board, including POST, LIVE, and Gold. Budget about nine minutes and 320 MB on the first run. Later runs reuse the revision-keyed cache, but every run still re-resolves the three dataset heads against the Hub and refuses to score if any has moved, so the full board needs network each time.

git clone https://github.com/yzhao062/catchbench.git
git clone https://github.com/yzhao062/grade.git
git -C grade checkout 3839a57ac165d58a807fce0a3ff38346732ee936   # the pinned commit CI uses
cd catchbench
python -m pip install -e "../grade[experiments]"

# Pinned torch first, then its matching compiled backend. See the note below for why the order
# matters; installing ".[full]" on its own resolves an unpinned torch that pyg-lib has no wheel for.
python -m pip install "torch==2.12.1" --index-url https://download.pytorch.org/whl/cpu
python -m pip install pyg_lib -f https://data.pyg.org/whl/torch-2.12.1+cpu.html

python -m pip install -e ".[full]"
catchbench

The runner reuses GRADE's verified Who&When and SWE-Gym / tau-bench loaders and evaluations for the reference methods, and computes the auditable baseline through auditable's own public kernel (SessionGraph plus downstream_reach). SWE-Gym and tau-bench download from the Hugging Face Hub on first run. GRADE is not on PyPI, and its experiment modules are not included in its wheel, so keep its checkout next to this repository as shown above. Alternatively, set GRADE_DIR to its checkout.

Nearly every number the manuscript reports is regenerated from this repository, so the code is checkable against the write-up. The board-derived tables come from run.py through tools/emit_boards_table.py; the contrast table from tools/statistical_tests.py through tools/emit_stats_table.py --contrasts; the transfer table from tools/emit_transfer_table.py; the LIVE prefix tables from tools/emit_live_prefix_table.py; and the remaining reported quantities from tools/whoandwhen_split_report.py, tools/pre_merge_judges.py, tools/namedvalue_admissibility.py, tools/namedvalue_control_power.py, tools/pygod_seed_stability.py, and tools/gold_artifact_diagnostic.py, and tools/pre_label_maker_diagnostic.py. The four emit_* tools take a --check flag that exits non-zero and prints the delta when the manuscript is stale; the other seven print their quantities and are compared by reading. Two of those seven, the Who&When split report and the PRE label-maker diagnostic, additionally exit non-zero when their own printed values leave what the paper states. One group of reported values falls outside that coverage, and the paper's own source table lists it: the Who&When Pro trace count and the AuthBench task count come from those papers and carry their citations in place. All of it runs locally rather than in CI, for the reason given under The Full Board.

run.py computes the boards from the inputs available to a checkout. A repository commit fixes the benchmark code, committed PRE artifacts, and cached LLM-judge predictions. It also records immutable Hugging Face commits for Who&When, SWE-Gym, and tau-bench in catchbench.corpora. Before scoring, the runner verifies that each dataset head still equals its recorded commit, forces GRADE's Hub calls through that full revision, and verifies the observed fetch or Who&When snapshot metadata. The printed board header records all three commits. Direct execution of GRADE outside this runner is not covered by the CatchBench-side pin. The documented setup and both workflows pin the GRADE commit, but the runner does not verify the sibling checkout's revision at run time, the auditable revision is not pinned at all, and the remaining Python dependencies are not locked.

This repository implements POST localization, detection, cause attribution, and Gold injection; LIVE streaming early warning and online stale-state detection; and the PRE over-privilege audit across six config corpora.

The repository ships benchmark code, cached LLM-judge predictions, and a PRE derived feature with labels. It does not re-host the raw upstream trace corpora or the upstream PRE task and role prose. The loaders obtain Who&When, SWE-Gym, and tau-bench during setup or the first benchmark run. Raw PRE prose is excluded because its licences have not all been verified and it can contain personal data. The 31 published Who&When judge caches, the 12 addendum Who&When judge caches, and four PRE judge-vote artifacts retain raw or raw_response model output. Some outputs quote or restate source traces, roles, workflows, or tool descriptions; see the generated asset manifest and third-party terms. Consequently, replay of boards that use upstream corpora also depends on continued access to those sources.

For Gold, the repository ships the injector and evaluation code, not a static copy of the SWE-Gym runs. The board is generated from fetched clean runs and uses the exact eligible-pool ranking as its target-selection control. A broken-predecessor baseline still separates both fault kinds on the file-level substrate (tools/gold_artifact_diagnostic.py), so the Gold boards are mechanism diagnostics. An artifact-controlled version requires a named-value corpus and a human-audited validation slice.

Data and Generated-Artifact Licensing

The repository's MIT LICENSE covers CatchBench-authored code and the 56 authored synthetic PRE records. It is not a blanket licence for derived third-party records or cached model output. Of the 1187 committed PRE records, 663 carry an established licence: 626 MIT, 24 Apache-2.0, 9 GPL-3.0, 2 CC-BY-4.0, 1 AGPL-3.0, and 1 BUSL-1.1. The remaining 524 declare none.

Those 524 were checked rather than left unexamined, on 2026-08-21. Every n8n and SWE-agent record is among them: SWE-bench/experiments carries no licence file and states no submitter terms, and the n8n gallery terms grant other users a licence to use and adapt a template rather than to redistribute it. A re-check of the CrewAI and MCP records recorded as NOASSERTION found that 106 of them do carry a licence their upstream states in a README or package manifest, which is why the counts above are higher than an earlier reading of the same data. All 524 are recorded as NOASSERTION and marked unfinished in THIRD_PARTY_LICENSES.md.

The GPL-3.0, AGPL-3.0, CC-BY-4.0, and BUSL-1.1 records stay in the release. The first three ship their licence texts in third_party/licenses/; the BUSL-1.1 record's notice is reproduced inline in THIRD_PARTY_LICENSES.md instead, because BUSL-1.1 is a template whose Parameters block each licensor fills in, so one shared copy would state somebody else's terms. THIRD_PARTY_LICENSES.md sets out exactly what those records carry and why, so a reader can judge the position rather than take it. Every licence value the committed records declare is either carried as local text or recorded as a non-declaration, and tools/emit_third_party_notices.py --check enumerates the declared values rather than a fixed list, so a source declaring something unanticipated fails the check instead of passing it.

The associated Who&When code repository is MIT, but the pinned Who&When dataset card does not declare a dataset licence. The pinned tau-bench trajectory card likewise does not declare a licence; neither project's code licence is presented here as licensing separately hosted generated trajectories. Exact artifact paths, source identifiers, and declared distributions are in ASSET_MANIFEST.json. Reproduced notices, local third-party licence texts, source links, and clearly marked unresolved blocks are in NOTICE and THIRD_PARTY_LICENSES.md.

Posture: Branded Name, Neutral Content

Because the benchmark shares a name with one entrant, the relationship is stated explicitly:

• The borrowed POST labels come from source corpora. Localization scores against Who&When's human-verified attribution; detection scores against SWE-Gym and tau-bench outcome labels. Gold injection-site labels and PRE labels are constructed within this benchmark and carry the limitations stated below.
• auditable is one baseline, not the referee. It sits on the board next to a random floor, a size-and-counts baseline, PyOD on flattened features, a supervised reference, and a full-feature reference. Its scores use the same interface as the other entries, and it is not required to lead.

The Auditable Ecosystem