AN2B Labs Technical Report #020

Zero-Callers for Cognition: an Ablation Auditor for Agent Systems Is Certified, the Popular Defaults Come Up Clean, and the Cheap Shortcut Is Refuted

J. DeVere Cooley, AN2B Labs Status: v1.0, published 2026-08-28; ledger at an2b.com/labs Pre-registration: commit 7b7262d, 2026-08-24, github.com/jcools1977/an2b-labs

Abstract

Agent systems can contain components that run, produce output, and burn tokens without ever causally affecting a final answer. TR-020 built an automated ablation auditor for this failure mode and asked three frozen questions: can one-at-a-time masking recover known-dead components, what is the dead fraction in popular real configurations, and can a cheap textual-influence tracer substitute for full ablation. The answers: the auditor is certified (29/29 components class-exact against a sealed ground truth it could not read, precision and recall 1.000, replicated across disjoint probe halves); the canonical defaults are clean (0 of 8 components dead or redundant across LangChain, LlamaIndex, CrewAI, and AutoGen tutorial configurations, 95% CI [0, 0.32]); and the shortcut is refuted (surrogate kappa 0.29 against a frozen gate of 0.7, with thresholds committed before any trace existed). The refutation's mechanism is the report's sharpest finding: dead components in real systems talk about the same things as their systems, so four of five planted dead components scored textual influence of 0.99-1.00, statistically level with the twenty live ones. Topicality is not causality; only intervention tells them apart. The protocol verdict is the pre-registered SPLIT: ablation certified, surrogate not validated. Everything ran on one 16 GB Mac mini at $0, and the audit's final verify carries exactly two red legs, each red for a reason pre-registered before its data existed.

1. Question and hypothesis

H1 (three conjuncts, frozen): systematic component ablation identifies dead modules with high agreement against ground truth in seeded systems; real popular agent configurations contain a nontrivial dead fraction (point estimate reported, any value a finding); and a cheap textual-influence surrogate predicts full-ablation results well enough for continuous monitoring (kappa >= 0.7).

H0: causal influence is too entangled for one-at-a-time ablation, or every component in reasonable systems genuinely matters.

The verdict, ruled in advance for this exact configuration of results (DECISIONS D23): conjunct one holds, conjunct two is answered with a zero, conjunct three fails. Split: the auditor stands, the shortcut falls.

2. Method

Pre-registered thresholds, kill criteria, and negative controls (root commit 7b7262d); every implementation judgment in tr020/DECISIONS.md (D1-D23), each timestamped before the numbers it could have bent toward.

Taxonomy (D1). Three verdicts, forced by the benchmark's own duplicate-tool plant before any system was built: dead (no effect masked alone or jointly), redundant (no effect alone, effect when masked jointly with a partner), live (effect alone). The pairwise sample separates dead from redundant.

Criterion (frozen). A component has no effect if masking it changes under 5% of canonicalized answers AND the judged-quality delta CI includes zero, over 150 probes per system.

Seeded benchmark. Seven systems, 29 components, covering the planted archetypes: appended-but-never-read, produced-and-ignored, irrelevant-only retrieval, duplicate tools implemented as consumer robustness (a reconciler that takes the first parseable value), and an all-live control. Ground truth is sealed by hash; the auditor package provably never reads it (grep-enforced). Every trace is byte- reproducible in isolation (per-item seeding, sabotage-tested), so any verdict can be replayed by a skeptic.

Measurement layer, gated red-then-green. The canonicalizer had to match 80 human-ratified change/no-change pairs 100%; the judge (Llama 3.1 8B, greedy, disjoint from the actor) had to see three classes of planted damage by at least 1.0 point on its 10-point scale (measured separations 2.7-5.3). Ratification is hash-bound: the gate's first exercise caught three labels that were code artifacts, corrected under cross-instance adversarial review with PI override (D13, D17, D18).

Models. Actor Qwen3-1.7B-4bit, judge Llama-3.1-8B-Instruct-4bit, both pinned by HF snapshot commit; greedy everywhere; one 16 GB Apple M4 Mac mini.

3. Seeded results: the auditor is certified

Recovery: 29/29 class-exact. Precision 1.000, recall 1.000. All five dead plants found dead, including the irrelevant-retrieval plant (masking useless-but-present context left span answers stable, the one question no stub could pre-answer). Both redundant pairs recovered through the joint-effect signature. Zero flags on the all-live system. Three-class verdicts identical on both probe halves for every component. The interaction KILL (more than 10% of live components falsely flagged) never fired; joint ablations behaved additively where sampled.

The placebo control is red, and the red is a finding (D19, ratified). Paraphrasing a component's own output was inert on span and number systems (0.0-2.3% answer change) and structurally non-inert on text systems (38-96%), with quality deltas that moved in BOTH directions when a stronger model paraphrased a weaker system's intermediates. Two methodology results follow: (1) placebo controls for agent audits must split wording-inertness from quality-inertness, because greedy generative cascades make the former unsatisfiable for live components (any upstream rewording changes downstream wording); (2) paraphrase provenance moves quality: an 8B paraphrasing a 1.7B system's intermediates improved some pipelines and degraded others, so a placebo paraphraser must be capability-matched or the control measures the paraphraser. The failure is directionally asymmetric: cascades can only push away from invariance, so they can inflate change-rates on live components but cannot manufacture the false deads that would taint recovery. That is why recovery was perfect through six wording-brittle systems.

4. Wild phase: the canonical defaults are clean

Four configurations, named against pre-committed criteria (popularity, local-endpoint runnable, distinct architecture families, defaults as shipped) before any was audited: LangChain ReAct tool agent (145k stars), LlamaIndex starter RAG (52k), CrewAI researcher-writer crew (58k), AutoGen planner-executor (61k). Each adapter passed a bite-proof before its audit counted: masking a component the answer demonstrably depends on must visibly change output, the check must fail under deliberate interception sabotage, and traces must be byte-identical across runs. All four passed on their first attempt.

Dead fraction: 0/8, 95% CI [0, 0.32]. Every retriever, tool, researcher, writer, planner, and executor is causally live. No system tripped the unauditability wire (placebo quality arm with a capability-matched paraphraser and the certified 1.0-point resolution floor: zero false fires on the seven known-good seeded systems, zero fires in the wild).

The nuance worth more than the headline: AutoGen's planner showed answer-change 1.00 under masking with a quality CI including zero. Total textual influence, quality contribution below certified resolution. Both arms are therefore reported per component; and the generalization boundary is stated plainly: shipped defaults are curated; the folklore about bloated agent stacks describes accretion, the tools and stages teams bolt on after the tutorial. Auditing accreted production systems is the tool's actual job; this census says the starting points are lean.

5. The shortcut, refuted: topicality is not causality

The surrogate (span matching plus embedding similarity over persisted traces) ran with every threshold frozen before any trace existed (D23: 3-gram containment >= 0.2, substring for short outputs, bge-small cosine >= 0.60 as rescue, dead below 5% influence), behind a determinism certificate (two trace passes byte-identical; one full masked condition per system recomputed and matching its persisted change rate exactly, seven of seven).

Kappa: 0.29 primary (gate 0.7), 0.15 secondary.

Figure 1. Textual influence rate for all 29 seeded components by ablation verdict. Four of five dead components sit at 0.99-1.00, level with the live cluster; only the vocabulary-disjoint plant is separable. No threshold on this axis distinguishes dead from live.

The mechanism: the tracer caught exactly one dead component, the recipes retriever in a geography system, the only plant whose vocabulary is disjoint from its system's topic. The unread critic critiques a draft of the same brief; the ignored plan contains the very words the executor sorts; the unread sentiment score quotes the ticket the reply answers. Dead components in real systems talk about the same things as their systems, so content similarity cannot separate provenance from topicality. Because the thresholds were frozen first, no tuning could have escaped this, and the overlap is complete (three dead components at exactly 1.000), so no other threshold exists either. The miss is the measurement's verdict on the method, not on its calibration. The wild phase had already flagged the same dissociation from the other side: a planner with total textual influence and below-resolution quality contribution.

The invitation: the seeded benchmark (seven systems, sealed ground truth, committed probes, byte-replayable traces) is public. Any future tracer that believes it can predict causal deadness from content now has a sealed exam to take. The negative result is released as infrastructure.

6. Verdict

Per the pre-registered reading (D23): SPLIT. The ablation auditor is certified for use, its seeded and wild findings stand, and the cheap continuous-monitoring surrogate is not validated, published here as its own negative sub-finding. The final verify runs every instrument leg green and exactly two legs red, each red for a reason committed before its data existed: the placebo change-arm (D19) and the surrogate kappa (D23). Nothing was moved to make anything green.

7. Limitations

• One actor model class (1.7B) and one judge (8B, 1.0-point certified resolution); effects below that floor are invisible by design.
• The wild census covers canonical tutorial defaults (n=8 components, interval stated), not accreted production systems, and its generalization boundary is exactly there.
• Any content tracer is structurally blind to redundancy (both duplicates' content propagates); the secondary kappa quantifies it.
• Wild adapters recompose two frameworks' sequential flows with one added interception seam each, logged as plumbing; bite-proofs bound the risk.
• LangChain's shipped tutorial tools are cloud SaaS and were substituted with local equivalents preserving the tool-calling architecture.

8. The tool and reproducibility

The harness ships as deadwood-audit (CLI deadwood): the trace runtime, structure-preserving masking, the three-verdict criterion, pairwise redundancy separation, and the placebo and bite-proof machinery, pointed at any system that can expose components through the trace interface. Public repository: github.com/jcools1977/an2b-labs, tr020/. Pre-registration is the root commit; the decision log runs D1-D23; every threshold predates every number; per-component predictions, traces certificates, the sealed benchmark, and verify.sh are committed. Hardware: one Apple M4 Mac mini, 16 GB. Incremental cost: $0. Wall-clock: roughly three attended days, including one overnight seeded run and one morning of wild audits.