Agent Security Harness

Even if an agent is properly authenticated and authorized, can it still be manipulated into unsafe or policy-violating behavior?

A test that did not reach the target is not a passing security test.

This harness sends adversarial traffic at a live endpoint, records what the target actually serviced, and reports PASS, FAIL or INCONCLUSIVE. It will not convert a request the target never answered — or a capability the target never exposed, or an answer that establishes nothing either way — into evidence that a control held. That distinction is enforced by testing/test_serviced_guard.py and testing/test_x402_capability_controls.py rather than asserted here, and the repairs that put it there are in CHANGELOG.md.

How that is checked: five target shapes

A verdict is only worth what it does when the target changes. Four scripts point every suite at a sentinel that differs only in how it answers, and four state files under testing/ pin what each one claims:

python3 scripts/dead_host_sweep.py        # a closed port: nothing answers
python3 scripts/permissive_host_sweep.py  # HTTP 200, grants everything
python3 scripts/refusing_host_sweep.py    # HTTP 403, refuses everything
python3 scripts/empty_answer_sweep.py     # HTTP 200 `{}`: answers, does nothing

The fourth transport shape is the one the other three cannot see. A target answering 200 with an empty body satisfies every serviced-request guard -- the answer is real -- while supplying no refusal, no decision, no inventory and no prose. The fourth external review found 47 rows passing against it, including two AIUC-1 controls; the remainder and the reading of each row is in testing/test_empty_answer_is_not_a_control.py.

The fifth shape is a live agent rather than a transport, and it is asserted in testing/test_refusal_establishes_a_pass.py rather than run as a script: one agent that complies with every request without using an indicator word, and one that declines in plain prose. A prose-graded module must pass nothing against the first and something against the second. Those two assertions are what v4.17.0 repaired eight modules against.

The first two ask whether a verdict can be wrong: a PASS against either is a control reported as holding when it was never exercised. The third asks whether a verdict can be right, which the other two cannot see — a suite that cannot pass a target which refuses every attack scores zero against all three, and that is what a healthy module looks like from the first two poles. The fourth asks whether a verdict needs the target to have done anything at all.

These are diagnostic sentinels, not conformant implementations. A non-zero row is a reading list rather than a defect count: it may be correct by construction (over_refusal_harness asks whether legitimate requests are wrongly blocked, so it should pass a target that blocks nothing), it may need a marker the generic sentinel does not emit, or it may be a real inversion. Only reading the test separates them.

Sentinels are the floor. The first run against a real MCP implementation found two defect classes all three had structurally missed, because every sentinel spoke the response idiom the harness already expected.

$ agent-security test mcp --url http://localhost:8080/mcp
Running MCP Protocol Security Tests v4.17.0...
 MCP-001: Tool List Integrity Check [PASS] (0.234s)
 MCP-002: Tool Registration via Call Injection [PASS] (0.412s)
 MCP-003: Capability Escalation via Initialize [FAIL] (0.156s)
...
Results: 8/10 passed (80% pass rate) - see report.json

Illustrative output. A target the harness cannot reach, or that answers without servicing the request, reports INCONCLUSIVE — never PASS. See v4.13.1 for why that distinction is enforced rather than assumed.

640 executable security tests across 48 test-bearing modules on main (verified 2026-09-21 via scripts/count_tests.py; the v4.22.0 release carries the same 640). MCP + A2A + L402 + x402 wire-protocol testing, plus UCP/ACP merchant-journey, AP2 mandate-chain, Fireblocks x402 hardening, Visa TAP / Mastercard Agentic Token funding-instrument, and denial-of-settlement finality conformance across the full agentic-payments stack. Decision-layer attack scenarios. One pip install away.

If this evidence discipline is useful in your agent-security work, star this repository to follow releases.

Evidence before coverage

Every claim in this project is bounded by the E1-E5 Evidence Class Taxonomy: observation, runtime characterization, enforcement, persistence/replay resistance, and isolation. A second axis, I0-I2, states who produced the oracle: self-authored, independently reimplemented, or an independent sensor the target does not control. Strength and independence are different properties, so both are cited. A result is not promoted beyond what its retained artifact and execution record demonstrate. Author-performed mappings and test runs are not independent certification.

The AIUC-1 Evidence Field Guide is a plain-language companion that applies this same taxonomy. It adds one distinction the ladder above does not encode: whether evidence is mapped (a documented requirement relationship), executed (a recorded run against a stated target and pinned revision), or independently reviewed (assessed by a qualified outside party). Those describe the status of evidence and are orthogonal to E1-E5, which describes its strength. A mapping alone is E1-level material regardless of how many requirements it covers. The taxonomy in this repository is canonical; the field guide is hosted outside it and is not version-pinned.

OWASP Agentic AI v1.1 Threat Coverage Report — commit-pinned mapping from the full T1–T17 taxonomy to executable tests: 13 direct, 4 partial, 0 not evidenced, across 96 mapped tests and 66 named OWASP scenarios. Mitigation-control validation is tracked separately from threat coverage (11 validated, 10 partial, 1 guidance-only), and every gap, evidence class and reproduction command is in the report. (T1–T15 submission view · canonical mapping)

Adapted from OWASP Agentic AI — Threats and Mitigations v1.1 under CC BY-SA 4.0. A test-capability report — not a certification, conformance claim, or OWASP endorsement. The adjudication is author-performed and is not independent review.

Quick Start

pipx install agent-security-harness

pipx is the recommended install because this is a command-line tool. It builds an isolated environment and puts agent-security on your PATH, with no sudo and no PATH edits. Get it with brew install pipx, apt install pipx, or python3 -m pip install --user pipx.

python3 -m venv .venv && source .venv/bin/activate
pip install agent-security-harness

error: externally-managed-environment

pipx upgrade agent-security-harness

# Confirm which build you're on:
agent-security --version

# See it work immediately — no server needed:
agent-security test mcp --simulate

# Then test your real MCP server:
agent-security test mcp --url http://localhost:8080/mcp

# Test an x402 payment endpoint
agent-security test x402 --url https://your-x402-endpoint.com

# Human-oversight surface: reviewer exposure (T10) and agent-to-human
# manipulation (T15). An unreachable target reports INCONCLUSIVE, never PASS.
agent-security test hitl --url http://localhost:8080

# Which tests back a given OWASP threat, scenario or mitigation control
python scripts/owasp_agentic_select.py --threat T16
python scripts/owasp_agentic_select.py --control P5-REA-001

See docs/QUICKSTART.md for mock server setup, rate limiting, MCP server mode, and CI/CD integration.

Three Layers of Agent Decision Security

How This Differs From Other Projects

Star counts verified 2026-08-02 via the GitHub API. Invariant Labs' mcp-scan now redirects to snyk/agent-scan and is listed once rather than as two separate projects. Competitor rows describe what each tool does by design; a dash means the capability is outside its stated scope, not a defect.

What the table does not measure. Every row above is coverage: what gets tested, and how much. Coverage says nothing about what a result licenses you to claim. That is what the E1-E5 Evidence Class Taxonomy is for, and it is applied to this project's own output first: the AIUC-1 crosswalk is E1 material regardless of covering 19 of 20 requirements, and this harness holds no I2 evidence at all, because it reads protocol responses the target itself emits. No claim here is deliberately made about how other projects handle this.

One external check. The Research backing row says self-authored because it is. The one exception is #304, where an outside party reimplemented the receipt-claim oracle from the published contract and replayed a pinned corpus. One reproduction, of one corpus, by one party, is not independent review of the harness as a whole, and this project still does not have that.

Use both. Scan with Snyk Agent Scan or Cisco MCP Scanner for static analysis. Test with this framework for active exploitation. They're complementary layers.

Research

Seven public preprints and notes deposited on Zenodo (not represented as peer-reviewed publications) and three NIST submissions underpin the methodology. Every DOI below was re-verified on 2026-08-02 by content negotiation against doi.org — title and authorship confirmed:

On citation counts. These records carry internal citation lineage across later work in this portfolio. An OpenAlex cited_by audit on 2026-08-02 found 30 citation edges and 0 qualifying independent citations — every edge is a self-citation, collapsing to nine duplicate/version records across three title families. Nothing here should be read as third-party validation or scholarly adoption. The one external check this project has received is an independent reproduction, listed under Used By.

Correction (2026-08-02). Two entries previously in this table cited DOIs belonging to other researchers — 10.5281/zenodo.15105866 (a MALDI mass-spectrometry dataset by Ranes et al.) and 10.5281/zenodo.15106553 (an e-learning article by Toshtemirov). They were attributed here to "Normalization of Deviance in Autonomous Agent Systems" and "Cognitive Style Governance for Multi-Agent Deployments". No Zenodo record under those titles by this author was located, so both rows were removed rather than re-pointed. The surviving Detecting Normalization of Deviance in Multi-Agent Systems record (19195516) is unaffected and was already listed separately.

Related Projects

Constitutional Governance (WHY layer)

The constitutional-agent package provides the governance gates and hard constraints that complement this test harness. Six gates, 12 hard constraints, amendment protocol — enforced in code, not YAML policy files. pip install constitutional-agent.

Architecture context (why this layer exists)

This harness tests whether a control holds. It does not say which controls an agent workforce should have in the first place — that is an architecture question, and it is written up as Enterprise Agent Architecture: the case for treating autonomous agents as a fifth architecture domain alongside Business, Information, Application and Technology.

cognitivethoughtengine.com/eaa — the four-layer model (Agent/Workforce, Capability/Tool, Control Plane, Governance), a maturity model, a standards landscape, and the live reference environment the telemetry above comes from.

Documentation

Roadmap

Current: v4.17.0 (2026-08-30). Recent shipped work — v4.5 skill supply chain and governance modification · v4.6–v4.9 payment-stack depth (AP2 mandate chain, UCP/ACP merchant journey, card-network agentic tokens, settlement finality, Fireblocks x402) · v4.10 benchmark integrity · v4.11–v4.12 decision-governance corpus currency and provenance repair · v4.13 OWASP Agentic v1.1 T1–T17 coverage mapping and the human-in-the-loop harness · v4.13.1 a correctness fix to that harness · v4.14.0 endpoint provenance · v4.16.0 three target shapes: a verdict must be able to be wrong AND to be right · v4.17.0 eight modules could not tell a refusal from a compliance · v4.18.0 INCONCLUSIVE became a field, and the read-list emptied · v4.19.0 a correctness disclosure: verdicts moved in both directions · v4.20.0 the last absence-graded verdicts got a positive control · v4.21.0 the instrument was the thing under test: 7 of 9 defects were in the measurement apparatus · v4.21.1 the published wheel reported 17/17 against a host that did not exist · v4.21.2 a fourth review asked what the repaired layer assumes: 47 rows called a control held on a reply that said nothing · v4.21.3 a fifth review asked whether the checks can fail: the workflow contract tests passed with every step disabled, and nine of twelve register denominators could be replaced by a constant · v4.22.0 scripts shipped as a namespace portion and the console printed INCONCLUSIVE as FAIL in twenty-nine harnesses; four new families took the release from 623 to 640 · v4.15.0 unserviced requests are no longer recorded as passes (see CHANGELOG.md).

The release carries 640 tests; main is at 640. They agree as of v4.22.0, and the two numbers stay separately stated because that is temporary: the next merge to main reopens the gap. They did not agree at v4.21.3, which carried 623 while main stood at 640: the difference was MEM-013..MEM-016, the memory revocation-enforcement family, WT-001..WT-004, the workspace-trust ingestion family, HI-001..HI-004, the hidden-instruction family, and CR-001..CR-005, the capability-residue family, all of which landed after that tag. The same divergence held at v4.15.0, where the release carried 603 while main was at 608. Both numbers were correct for their own ref, and agreement now is a property of this release being fresh rather than a guarantee: the next merge to main reopens the gap. That is why scripts/check_public_metadata.py compares public claims against the release rather than against main, and why docs/release-claims.json pins the release count to a commit where re-running the command reproduces it.

Next — Standards & Evidence. Reproducible settlement-time payment evidence, a methodology paper, and the attestation/evidence schema submitted to a standards venue. Coverage breadth and test count are explicitly not goals. Full detail and the anti-goals in ROADMAP.md.

Used By

Independent reproduction

The only external check this project has received. @VrtxOmega, in veritas-agent-trust-lab, wrote a separate Node verifier for the portable receipt-claim oracle fixtures and ran it against a pinned commit and fixture hash, matching all 11 RCL results including both acceptance controls — so it did not pass by rejecting everything. Reported in #304; named here with their consent, granted 2026-08-31. The result is written up in docs/RCL-CROSS-IMPLEMENTATION-INTEROP.md, with their own report and verifier runner linked as the primary artifacts.

The exchange also produced two corrections to this repository: signature_algorithm now names the actual encoding and states plainly that it is not RFC 8785 JCS, and coverage_gaps now declares that freshness is exercised only in the stale direction. Both are in fixtures/rcl/.

It is a reproduction of a pinned artifact by one external party, submitted as a report and explicitly not as a contribution, endorsement, certification, or adoption. It is not a substitute for independent review of the harness as a whole, which this project still does not have.

"A third independent implementation is worth more to this corpus than any additional vector." That is fixtures/rcl/CONFORMANCE.md §7, verbatim, and it is not modesty. Two implementations agreeing establishes that they share a reading; only a third can tell you whether that reading is the contract or a coincidence.

The bar is lower than it sounds. The corpus is eleven vectors, one fixture file, and a pinned hash; the run above was a single Node script written against the published format. docs/REPRODUCING.md has the pin, the canonical byte basis that is the most common reason a correct implementation computes a different digest, and the table of what does and does not count as independent. §7 of CONFORMANCE.md adds the one rule that decides whether your result counts at all: write the verifier from the published contract, not from protocol_tests/receipt_claim_harness.py. Reading our implementation first turns an independent result into a port, and the value is entirely in the independence. File it with the reproduction report template.

A result that contradicts ours gets published too. #304 is the precedent for that as well: it matched 11 of 11 and still surfaced two defects in our own contract description, both corrected above. Disagreement is the more useful outcome, not the awkward one — a reproduction that only agrees may just mean two implementations share the same wrong assumption.

Using the harness? Open a PR to add yourself, or tag us in your project.

Contributing

See CONTRIBUTING.md for guidelines, SECURITY_POLICY.md for security policy, and CONTRIBUTION_REVIEW_CHECKLIST.md for the PR checklist.

Citation

If you cite this work in research:

Saleme, M. K. (2026). Agent Security Harness — multi-protocol agent security testing framework. ORCID: 0009-0003-6736-1900. https://github.com/msaleme/red-team-blue-team-agent-fabric

Related Zenodo preprints are listed in full under Research; that table is the single source of truth for DOIs in this repository. Do not cite a DOI for this work that does not appear there.

License

Apache License 2.0 -- see LICENSE.