Agentic Security Harness

Your AI coding agent reads untrusted repository text. Can it keep data separate from instructions and authority?

Agentic Security Harness is a local, trace-first benchmark for defensive testing of agentic AI boundary failures. It runs reproducible synthetic scenarios, records portable traces and scorecards, and compares a deliberately vulnerable local agent with a protected one.

In plain English: it turns “the agent behaved unsafely” into evidence you can replay, validate, compare, and review.

Quickstart

Install the stable release from PyPI:

python -m pip install agentic-security-harness==1.2.0
ash quickstart --out reports/quickstart
ash agent-host-quickstart --out reports/agent-host-quickstart

For source development:

git clone https://github.com/krivonosoff161/agentic-security-harness.git
cd agentic-security-harness
python -m pip install .
ash quickstart --out reports/quickstart
ash agent-host-quickstart --out reports/agent-host-quickstart

ash quickstart is local and no-network. It runs the same stable 24-pattern corpus against both demo targets, validates the generated artifacts, and renders a self-contained HTML report.

ash agent-host-quickstart is the shipped provider-neutral integration path in v1.1.0. It runs a built-in owned synthetic workflow through the instrumented collector, canonical recordings, deterministic evaluator, atomic bundle publication, and shared validator. It retains digest-only public evidence and makes no provider call.

Runtime Gateway synthetic contour

The source tree now also contains a runnable local Runtime Gateway increment. It applies a closed policy before synthetic tool dispatch, exposes bounded OpenAI-compatible and MCP 2026-07-28 stateless development endpoints, and maintains a privacy-minimized append-only audit chain:

ash gateway-init --out gateway.toml
ash gateway-check --config gateway.toml
ash gateway-serve --config gateway.toml

Open http://127.0.0.1:8787/dashboard after startup, or use the hardened Docker Compose profile in Runtime Gateway synthetic contour. This is a credential-free synthetic integration surface. Offline provider-neutral tool-call adapters normalize retained OpenAI Responses, Anthropic Messages, Google Interactions, and MCP payloads through the same policy without SDKs or credentials. Live provider transport and a production firewall are still not shipped. The gateway exposes its exact closed policy and non-executable approval-request digests; it intentionally has no approval-grant endpoint yet.

Ecosystem map

Harness is the released core and the public contract owner for a modular security ecosystem. Transfer verification, handoff safety, playbooks, routing, filtering, private Runtime Guard research, and the public profile keep their own source-owned component facts. The Harness generates only the cross-project roadmap and compatibility view:

• Ecosystem roadmap
• Components and current integration status
• Documentation crosswalk
• component.yaml and ecosystem/roadmap.yaml

Today most companion repositories are standalone or contract_only. The source tree now contains the first closed Extension SDK V1 candidate for validated observation-to-finding dataflow. It does not auto-load installed packages and is not part of the published 1.2.0 surface; selected checks become installable only after their own conformance and release gates.

The stacked Corpus Pack SDK V1 candidate adds a separate, canonical registry for optional namespaced boundary-invariant metadata. It preserves the frozen corpus 1.0.0, loads no package code, and treats complete evidence as readiness for later rule evaluation rather than a security verdict.

The stacked companion adapter candidate now exercises exact Transfer Verifier reports, Handoff metadata and Playbooks guidance through that SDK on Linux and Windows. This closes a concrete producer-to-consumer dataflow gap; it does not make those repositories installable extensions or production enforcement.

The stacked Security Intelligence candidate adds a provider-neutral offline weekly public-source review contract with digest-only evidence, explicit coverage gaps, and no live fetching or model authority.

The next stacked receipt auditor candidate independently checks exact-pinned Router invocation and Cheap Filter triage accounting receipts from caller-supplied canonical bytes. Valid accounting remains inconclusive, missing evidence remains inconclusive, and drift becomes a finding; the auditors never emit pass, import or invoke the companion packages, or lower a security decision.

The following Extension Distribution Discovery V1 candidate inspects one explicitly named local distribution without importing it. It verifies its RECORD, closed entry point, canonical manifest, implementation bytes and caller-supplied configuration digest, then requires exact reinspection before issuing an authority-free approval receipt. Harness still does not load package code: the operator supplies an already constructed object, and the binder checks it against the approved manifest pins.

The stacked Extension Operator Lifecycle V1 candidate exposes that metadata-only inspection and exact-reinspection approval through safe CLI commands, adds canonical disable and non-executable rollback-plan receipts, and lists only explicitly supplied receipt state. It never imports, downloads, starts, stops, or rolls back extension code; the embedding application must construct and bind an object and enforce any accepted disable artifact.

Above that stack, the controlled local adapter candidate connects only to an operator-started literal-loopback /v1/responses endpoint and passes canonical tool calls through the existing closed Runtime Gateway policy. It supports local model names as opaque identifiers—including Qwen and DeepSeek-style names—without vendor claims. It has no DNS, proxy, redirect, credential, external-provider, arbitrary-tool, or upstream-MCP path; receipts are digest-only and operational authority remains none.

The stacked Policy Pack extension candidate independently parses one exact-pinned data-only Playbooks pack and evaluates caller-supplied content-free signals bound to canonical observations. A missing pack is inconclusive; production Harness does not import or execute Playbooks code, discover packages, call a network, or grant allow/enforcement authority.

The deterministic result is 24 modeled findings for the vulnerable fixture and 0 modeled findings for the protected fixture. This is synthetic conformance evidence, not a production safety claim.

Inspect the committed before/after example in examples/comparison-report/ or validate every public example locally:

ash validate examples/
ash validate docs/evidence-status-registry.json

What this is / is not

The benchmark focuses on agent operating-environment boundaries, not just standalone model answers. Built-in targets are deterministic and offline. The experimental external adapter is explicit opt-in, prompt-only, and does not execute tools. See docs/benchmark-semantics.md and docs/authorized-testing-paths.md.

Visual evidence snapshot

The public evidence map separates deterministic executable specifications, sanitized local observations, historical material, and independently reviewed evidence:

• Evidence map
• Evidence classes
• Machine-readable evidence registry
• Evidence pack format
• Private/public evidence boundary

Public artifacts may include scenario identifiers, aggregate counts, response hashes, and validator results. They do not include raw private prompts, raw responses, synthetic canaries, or local paths. Keep external raw material under .internal/external-demo/latest. Do not commit raw_responses/ or other private runtime evidence.

Current stable surface

• Trace schema 1.0, with a bounded legacy 0.1 read/migration window.
• Corpus 1.0.0, freezing 24 ordered synthetic pattern identifiers.
• Local targets for vulnerable/protected agent, RAG, tool, function, and multi-agent handoff comparisons, including toy-multi-agent and protected-toy-multi-agent.
• JSON traces, scorecards, run manifests, remediation, Markdown reports, and self-contained HTML reports.
• Deterministic validators for artifact integrity and declared benchmark semantics.
• Linux/Python 3.11-3.13 as the primary installed-package contour, with Windows 3.11 compatibility coverage.
• Reproducible wheel/sdist builds, checksums, GitHub attestations, and exact-subject CycloneDX SBOMs for public releases from v1.0.0 onward.
• A shipped provider-neutral Agent Host V1 contour for canonical, authority-free record/replay, deterministic evaluation, explicit Python instrumentation, and a validated 48-case no-network quickstart. The CLI does not execute arbitrary hosts or tools and does not authenticate its producer or certify an external system; see Agent Host Adapter SDK.
• A shipped local Runtime Gateway synthetic contour with closed pre-dispatch policy, bounded OpenAI-compatible and stateless MCP endpoints, deterministic synthetic tools, privacy-minimized hash-chain audit, dashboard, and hardened source-build Docker Compose.
• Credential-free retained-envelope normalization for OpenAI Responses, Anthropic Messages, Google Interactions, and MCP tool calls. These adapters do not make provider calls or grant tool-execution authority.

The exact shipped, experimental, planned, and historical surfaces live in docs/current-state.md and docs/capability-matrix.md. Technical v1 gates and honest non-claims are in docs/v1-readiness.md.

If you only have one minute

• Run the no-network demo above.
• Read the committed comparison.
• Browse the showcase and scenario matrix.
• See weak spots and findings.
• Check current state and the project tracker.
• Bring your own local or OpenAI-compatible model through Run your model.

Use your own model or runtime

The shortest cross-platform operator path is docs/run-your-model.md. It covers:

1. a no-model deterministic demo;
2. one explicitly authorized OpenAI-compatible model;
3. a deterministic local swarm comparison;
4. a bounded local-model mini-swarm campaign.

Connection details and scenario selection are documented in docs/connect-models.md and docs/test-your-model.md. External runs are prompt-only self-report checks unless a separately reviewed host/tool adapter exists. A coherent answer is not evidence of safe tool execution.

Benchmark and evidence documentation

Start with the curated documentation map if you are not sure which contract, operator guide, or research page applies to your task.

The README is the front door; deeper contracts live in docs/:

Specialized reviewer paths:

• Local Prometheus workflow
• Local model profiles
• Multi-agent handoff toy topology
• Security audit causal map
• R5 sanitized research status
• Project governance

Standards and portfolio boundaries

The project publishes conservative mappings to OWASP LLM, NIST, and a direct-fit MITRE ATLAS subset in docs/standards-mapping.md. These are maintainer-reviewed mappings, not certification or independent standards validation. Independent review remains public follow-up work.

The related public contract includes:

• Threat ontology: 26 provider-neutral failure families.
• Scenario adjudication ledger: 127 bounded source units from 13 enumerated builders.
• Unified event envelope: separation of observations, data, authority, advisories, decisions, and effects.

The 127 units are not 127 canonical attacks and not a repository-wide total. The contract grants no executor, provider, deployment, enforcement, or production authority.

Public security stack

Agentic Security Harness is the benchmark/evidence layer in a small public defensive stack:

llm-safety-playbooks -> ai-agent-handoff -> agentic-transfer-verifier -> agentic-security-harness

• llm-safety-playbooks documents practical boundary rules.
• ai-agent-handoff represents task briefs and handoff state as reviewable files.
• agentic-transfer-verifier checks provenance, authority, approval, and audit evidence around transfers.
• This repository measures modeled failures and produces validated benchmark artifacts.

The repositories are related but not interchangeable. A playbook is not a runtime control, a handoff file is not a sandbox, and a passing benchmark is not a production certificate. The review-only ecosystem integration candidate builds the optional Transfer and Handoff extension wheels and exercises their explicit approval lifecycle on Ubuntu and Windows; it does not bundle or auto-install them.

Release and package status

The stable v1.2.0 package is published on PyPI and as an attested GitHub Release. The source tree is prepared as a source-only v1.3.0 release candidate containing the integrated ecosystem contracts above. It is not a tag, GitHub Release, TestPyPI/PyPI publication, deployment, or production-enforcement claim. Those remain separate, auditable owner gates.

• Release checklist
• PyPI release process
• Changelog
• Security policy

Independent standards review and a durable second GitHub reviewer are transparent post-v1 credibility tasks. Their absence does not change deterministic test results, but independent review is not claimed.

Development

python -m pip install -e ".[dev]"
python -m pytest
python -m ruff check .
python -m mypy src tests tools
ash validate examples/

Development follows idea -> issue -> branch -> implementation -> tests/artifacts -> PR -> GitHub checks -> review gate. See docs/git-evidence-workflow.md, CONTRIBUTING.md, and docs/agent-operating-guide.md.

Responsible use

Use only synthetic, local, owned, or explicitly authorized targets. Do not use this project for credential theft, persistence, evasion, destructive payloads, or unauthorized systems. See SECURITY.md and docs/authorized-testing-paths.md.

Citation, contributing, and license

• Citation metadata: CITATION.cff
• Governance: GOVERNANCE.md
• Contributing: CONTRIBUTING.md
• Support: SUPPORT.md
• License: Apache-2.0, see LICENSE and NOTICE