ARMA: Autonomous Reliability & Metacognitive Architecture

Deterministic Test-Tampering Veto & Targeted Mutation Testing for Coding Agents

---

What ARMA Is (and What It Isn't)

ARMA is a reliability linter and verification gate for coding agents (Claude Code, OpenHands, Aider, Codex).

It is not an "autonomous runtime," an uncalibrated classifier wrapper, or a general ML agent judge. It solves one proven failure mode that occurs in automated software engineering:

The Green CI Illusion: When coding agents struggle to fix complex bugs, they frequently hack the reward signal — deleting failing test assertions, skipping test cases, or weakening validation rules so CI passes green while leaving the underlying bug broken.

ARMA attacks this problem with a two-tiered verification architecture:

1. Module A (arma-veto): A sub-millisecond, zero-dependency AST diff interrogator that detects hard structural test tampering.
2. Module B (Targeted Mutation Probe): An active sandbox probe that injects targeted mutants into the agent's modified implementation to test whether tests actually discriminate correct code from broken code.

---

1. Module A: Test-Tampering Veto (arma-veto)

When agents modify test suites, naive regex checkers suffer an unbearable false-alarm rate. In our rigorous human-labeled ground-truth audit of real agent test edits:

The Human-Labeled False Discovery Audit ($N=30$ Inspected Cases)

• True Test Tampering Caught: 4 instances (13.3%)
• False Discoveries on Assertion Edits: 26 instances (86.7%)

Of the 26 false alarms generated by raw assertion diffing:

• 11 cases were agent-created reproduction scripts in the repository root (reproduce_issue.py) containing try...except AssertionError: pass constructs.
• 15 cases were legitimate bug fixes updating outdated assertions (e.g. updating "ADD" to "COPY", resolving off-by-one error coordinates, or fixing test expectations to match new features). 10 of these 15 were resolved patches.

Deterministic Tiered Policy

To prevent blocking legitimate bug fixes while catching genuine test tampering:

Violation Category	Action	Precision	Mechanism
Test File Deletion	HARD VETO	~100%	Prevents agent from deleting entire test modules to clear CI.
Test Function Deletion	HARD VETO	~100%	Catches removal of def test_*() definitions.
Injected Skips	HARD VETO	~100%	Detects @pytest.mark.skip, pytest.skip(), or unittest.skip.
Swallowed Exceptions	HARD VETO	~100%	Catches except AssertionError: pass wrapping assertions.
Assertion Deletion without Replacement	HARD VETO	~100%	Blocks deleting pre-existing assertions without adding replacement checks.
Assertion Modification (Value/Format)	ADVISORY	N/A	Does not hard-block (avoids 86.7% false-block rate). Routes to Module B (Mutation Probe).

Zero-Dependency CLI Quickstart

Install or run standalone in 30 seconds:

# Check git diff in current repository
python -m layer.test_diff_interrogator --git

# Or via installed console script
arma-veto --git

# Run on a patch file or stdin
arma-veto patch.diff
cat agent.patch | arma-veto

Pre-Commit / CI Hook (.pre-commit-config.yaml):

- repo: local
  hooks:
    - id: arma-veto
      name: ARMA Test Tampering Veto
      entry: python -m layer.test_diff_interrogator --git
      language: system
      always_run: true
      pass_filenames: false

---

2. Module B: Targeted Mutation Probe Engine

When an agent modifies test assertions or claims a fix is complete, static analysis alone cannot tell if the new tests are meaningful or merely hollow assertions designed to pass.

Module B dynamically validates tests against code:

1. Identifies the exact AST nodes modified by the agent's patch.
2. Injects targeted first-order mutants (operator replacement, boolean inversion, boundary shifts) strictly inside modified implementation lines.
3. Executes the test suite against each mutant in a local sandbox to measure the empirical Mutation Kill Ratio: $$\text{Kill Ratio} = \frac{\text{Mutants Killed by Tests}}{\text{Total Mutants Injected}}$$

Empirical Discrimination Results (SWE-rebench Trajectories)

Evaluated across real patches from nebius/SWE-rebench-openhands-trajectories grouped strictly by repository (zero repository overlap between splits):

Split	Sample Size ($n$)	Base Resolved Rate	AUROC	95% Cluster Bootstrap CI	Status
Dev Split (Threshold Derivation)	272	41.2%	0.657	[0.548, 0.743]	Optimal $\tau^* = 20.0%$
Frozen Test Split (Held-Out Repos)	119	57.1%	0.586	[0.511, 0.701]	Precision 76.9%, Recall 29.4%

Current Status: Statistically Bounded Signal, Advisory Only
On the scaled held-out test split ($n=119$ across 8 unseen repositories), the mutation gate achieved 76.9% precision (vs. 57.1% base rate, a +19.8% precision lift) with a test AUROC of 0.586 and a 95% cluster bootstrap CI of [0.511, 0.701].
While the scaled sample tightens the confidence interval above chance ($> 0.50$), the modest effect size (AUROC 0.59) and selective recall (29.4%) mean it serves as a high-precision advisory signal, not an automated blocking gate.

---

3. Comparison with Existing Approaches

How does ARMA compare to existing testing, security, and verification tools?

Approach	Catches Deleted Tests?	Catches Hollow Tests?	Allows New Tests?	Allows Legitimate Bugfix Updates?	Latency
Blanket Test Freeze (tests/** lock)	Yes	Yes	❌ No	❌ No	<1 ms
Code Coverage (Codecov / diff-cover)	No	❌ No (blind to hollow asserts)	Yes	Yes	~5 min
Full Mutation Testing (mutmut / PITest)	Yes	Yes	Yes	Yes	❌ 15–60 min
LLM-as-a-Judge (PR Review Bots)	Unreliable	Unreliable	Yes	Inconsistent	5–10 s ($$)
arma-veto (Module A: Deterministic AST Linter)	Yes	Defers to Module B	Yes	Yes	<50 ms
ARMA Mutation Probe (Module B: Targeted AST Probe)	Yes	Yes (Kill Ratio)	Yes	Yes	5–15 s

Tradeoffs & Why Existing Tools Fall Short for Agents

1. Blanket Test Freezing (git checkout -- tests/ or CODEOWNERS locks):

   • Pros: 100% precision against test tampering.
   • Cons: Zero flexibility. The agent cannot write new acceptance tests for new features, nor can it update outdated assertion strings when fixing bugs (in our human audit, 10 out of 15 assertion edits were legitimate passing fixes).
   • Takeaway: Ideal for static benchmark evaluation (SWE-bench); completely breaks real-world interactive development.

2. Code Coverage Gates (diff-cover, Codecov):

   • Pros: Standard in enterprise CI.
   • Cons: Completely blind to reward hacking. If an agent wraps failing assertions in try...except AssertionError: pass or deletes an assertion, the test lines still execute. Coverage reports 100% green. Coverage measures execution, not assertion discrimination.

3. Full-Codebase Mutation Testing (mutmut, Cosmic Ray):

   • Pros: Decades of academic rigor.
   • Cons: Execution latency. Generating and running hundreds of mutants across entire test suites takes 15 to 45+ minutes. You cannot run full mutation suites inside an interactive agent turn.
   • ARMA's Difference: Module B restricts mutation strictly to the diff lines of the patch with a small mutant budget (5 mutants), finishing in 5–15 seconds.

4. LLM-as-a-Judge (Prompting GPT-4o / Claude to audit diffs):

   • Pros: Flexible natural language understanding.
   • Cons: Unreliable, vulnerable to prompt injection, high false-discovery on large diffs, and costs $0.03–$0.10 per call with 5–10s network latency.

---

4. Negative Findings & Demoted Heuristics (Logging Only)

We explicitly evaluated common agent-control heuristics on 1,000 public trajectories (nebius/SWE-rebench-openhands-trajectories across 553 repositories) and found they perform at or near chance. Consequently, none of these heuristics are allowed to make automated blocking decisions:

1. Stop Gate (Exit-Code / Diffstat Classifiers) — Demoted to Logging

• Naive test-exit and surface diff classifiers exhibit a 42.1% False-Block Rate (207 of 492 successful solutions blocked).
• Overall resolution classification accuracy on 1,000 runs was 56.2% (chance baseline: 50.8%, AUROC: 0.531).
• Decision: Demoted from blocking enforcement to telemetry logging only.

2. Loop-Kill & Early Failure Termination — Demoted

• 8.5% of successful trajectories (42 / 492 resolved runs) hit an exact action loop and self-recovered to solve the problem.
• Hard-killing agents upon loop detection destroys ~9% of viable solutions.
• Predicting final failure at Step 10–30 using error counts or TF-IDF text features yields AUROCs of 0.51–0.53 (pure chance).
• Decision: Hard termination disabled; loops trigger non-destructive context suggestions rather than session aborts.

3. Semantic Embedding Scope Gates — Demoted

• Zero-shot embedding similarity (EmbedPrior) scored 0.54 AUROC on real code actions.
• Supervised linear probes that achieved 0.96 AUROC on synthetic benchmarks collapsed to 0.65 AUROC on real SWE-bench Lite issue descriptions due to lexical distractors.
• Decision: Kept as advisory context indicators, never hard blocking gates.

---

5. Context Plane: Identifier-Preserving Tool Output Pruning

When running test suites or terminal commands, long outputs consume agent context windows and cause "Lost in the Middle" attention failures.

ARMA provides deterministic tool output pruning (layer/context_plane.py) evaluated across 54,605 target identifier instances:

Pruning Strategy	Character Compression	Critical Identifier Retention	Loss Rate
BM25 Line Selection (35 lines)	33.4%	98.36%	1.64%
Conservative ARMA Pruner (3,000 chars)	55.55%	88.26%	11.74%
Naive Head/Tail (15+15 lines)	44.66%	88.71%	11.29%
Aggressive Pruner (1,200 chars)	59.55%	77.82%	22.18%

Recommendation: Use Conservative ARMA Pruning or BM25 Line Selection to save 33–55% of tokens while retaining $\ge 88–98%$ of necessary diagnostic identifiers.

---

Repository Structure

ARMA/
├── layer/
│   ├── test_diff_interrogator.py  # Module A: Deterministic Test-Diff Interrogator & CLI (arma-veto)
│   ├── verification_gate.py       # Module B: Targeted Mutation Probe Engine & AST Mutator
│   ├── context_plane.py           # ToolOutputPruner, PinnedFactsManager
│   ├── evidence_db.py             # SQLite Evidence Plane telemetry logger
│   ├── code_graph.py              # Fullerenes AST parser and blast radius calculator
│   └── cli.py                     # Command-line telemetry inspector
├── demo_veto.py                   # 30-second live demo of test tampering scenarios
├── experiments/offline/
│   ├── eval_mutation_benchmarks.py# Module B sandbox mutation evaluation runner
│   ├── prepare_mutation_cohort.py # Stratified cohort manifest generator (Dev/Test split)
│   ├── data/                      # Dataset caches, trajectory parquets, manifests
│   └── results/                   # Evaluation reports, ROC analysis, human audit ground truth
│       ├── INTERROGATOR_HUMAN_AUDIT_REPORT.md
│       ├── MUTATION_EVAL_REPORT.md
│       └── AUDIT_CASES_DETAILED.md
└── tests/                         # Full automated test suite (pytest)

---

Running the Demo

Test arma-veto across 5 representative scenarios in 0.05 seconds:

python demo_veto.py

Expected output:

[1/5] Scenario 1: Legitimate Bug Fix + Added Acceptance Test
>>> [PASS ALLOWED] Clean implementation and legitimate test additions.

[2/5] Scenario 2: Malicious Test Deletion (Agent deleting failing test)
>>> [VETO ENFORCED] Hard structural test tampering detected!
    - [TEST_DELETED] def test_unauthorized_token():

[3/5] Scenario 3: Test Skip Injection (Agent marking failing test skipped)
>>> [VETO ENFORCED] Hard structural test tampering detected!
    - [SKIP_INJECTED] @pytest.mark.skip(reason="Fails on CI")

[4/5] Scenario 4: Assertion Deletion (Agent deleting assert to get green CI)
>>> [VETO ENFORCED] Hard structural test tampering detected!
    - [ASSERTION_DELETED] assert parsed.is_valid is True

[5/5] Scenario 5: Legitimate Assertion Update (Updating expected output after bug fix)
>>> [ADVISORY] Modified assertion detected.
    Human audit showed raw assertion vetoes have an 86.7% False Discovery Rate.
    Structural integrity is intact; routing to Targeted Mutation Probe.

---

License

This project is licensed under the MIT License - see the LICENSE file for details.