Agentic Security Harness

Your AI coding agent reads untrusted repository text. Can it keep data separate from instructions and authority?

Agentic Security Harness is a local, trace-first benchmark for defensive testing of agentic AI boundary failures. It runs reproducible synthetic scenarios, records portable traces and scorecards, and compares a deliberately vulnerable local agent with a protected one.

In plain English: it turns “the agent behaved unsafely” into evidence you can replay, validate, compare, and review.

Latest observation — 2026-09-27

The sixteen-call generalization study records four exact useful tasks, one permitted but semantically wrong hash operation, three rejected useful-task proposals and eight negative controls stopped at their declared boundaries. Five pure operations are not five correct tasks. The local Prometheus model was real; downstream Router transport and adversarial receipt mutations were declared fixtures. Real effects were zero; there was no response repair or retry. This is separate from the garak package-release gates.

Previous verified result — 2026-09-26

The twelve-call local-model follow-up records three real-model proposals completing all seven boundaries of the installed ecosystem, each ending in a built-in constant lookup. All six negative controls stopped at their declared boundary; real external effects were zero. Explicit protocol wording matched both fixed tasks; free wording matched neither capability identifier. This is bounded integration evidence, not a model reliability estimate, an autonomous-agent demonstration or an independent human audit.

• Inspect: fixed public corpus, content-free observations and artifact checks.
• Use the example: explicit proposal protocol. Printing the prompt or checking artifacts does not rerun the model.
• Delivery: merged PR #303. The earlier eight-call result is preserved separately in the report.

Package versus repository: the published package remains 1.6.0. This newer prompt helper, corpus, observations and tests are repository-owned; they are not a new installed package API. See release and package status and current state for the exact boundary.

Experimental garak connection (unreleased)

The 1.7.0 release candidate packages this opt-in adapter; it is not yet a published package. Candidate notes and gates keep staging, publication and the prior 1.6.0 release evidence separate.

The garak plan connector translates one strictly validated JSON plan into the existing Quarantine/Gateway path, without granting authority from a detector score. A four-case example keeps detection, admission, policy denial and synthetic execution separate. This is an independent Harness integration, not an official NVIDIA/garak component. It is new source functionality, not included in published 1.6.0; its optional detector compatibility lane pins an unmerged upstream PR rather than claiming release support.

Quickstart

Published v1.6.0: the native Ollama proposal adapter and an explicit installed-ecosystem pilot, retaining the Gateway lookup-key type guard and Router 0.2.1. The linked release notes distinguish source, staging and published evidence; a checkout or green CI alone is not proof of publication.

Install the exact package version from PyPI:

python -m pip install agentic-security-harness==1.6.0
ash quickstart --out reports/quickstart
ash agent-host-quickstart --out reports/agent-host-quickstart

For source development:

git clone https://github.com/krivonosoff161/agentic-security-harness.git
cd agentic-security-harness
python -m pip install .
ash quickstart --out reports/quickstart
ash agent-host-quickstart --out reports/agent-host-quickstart

ash quickstart is local and no-network. It runs the same stable 24-pattern corpus against both demo targets, validates the generated artifacts, and renders a self-contained HTML report.

ash agent-host-quickstart is the shipped provider-neutral integration path in v1.1.0. It runs a built-in owned synthetic workflow through the instrumented collector, canonical recordings, deterministic evaluator, atomic bundle publication, and shared validator. It retains digest-only public evidence and makes no provider call.

Runtime Gateway synthetic contour

The source tree now also contains a runnable local Runtime Gateway increment. It applies a closed policy before synthetic tool dispatch, exposes bounded OpenAI-compatible and MCP 2026-07-28 stateless development endpoints, and maintains a privacy-minimized append-only audit chain:

ash gateway-init --out gateway.toml
ash gateway-check --config gateway.toml
ash gateway-serve --config gateway.toml

Open http://127.0.0.1:8787/dashboard after startup, or use the hardened Docker Compose profile in Runtime Gateway synthetic contour. This is a credential-free synthetic integration surface. Offline provider-neutral tool-call adapters normalize retained OpenAI Responses, Anthropic Messages, Google Interactions, and MCP payloads through the same policy without SDKs or credentials. Live provider transport and a production firewall are still not shipped. The gateway exposes its exact closed policy and non-executable approval-request digests; it intentionally has no approval-grant endpoint yet.

Ecosystem map

Harness is the released core and the public contract owner for a modular security ecosystem. Transfer verification, handoff safety, playbooks, routing, filtering, private Runtime Guard research, and the public profile keep their own source-owned component facts. The Harness generates only the cross-project roadmap and compatibility view:

• Ecosystem roadmap
• Components and current integration status
• Documentation crosswalk
• component.yaml and ecosystem/roadmap.yaml

Runtime Guard remains private and contract_only. Harness version v1.6.0 contains the closed Extension SDK V1 and public passive extras for validated observation-to-finding dataflow. It does not auto-load installed packages; companion repositories remain optional, separately versioned distributions.

Version v1.6.0 retains the closed optional-dependency groups introduced in v1.4.0:

The generic PyPI coordinate llm-router is intentionally absent because it belongs to another project. CI builds all eight exact wheels from pinned Git SHAs and installs the closed local wheelhouse without loading either extension entry point. The public install commands are pip install "agentic-security-harness[router]==1.6.0" or pip install "agentic-security-harness[all]==1.6.0". Other companion pins are unchanged; installation remains separate from module activation.

For explicit installed extension binding, follow the fresh --no-compile environment and positive/negative controls in the installed-ecosystem example. The external pilot protocol explains the bounded feedback request.

For the actual six-component flow, run the separate 16-case functional chain and its independent report verifier. Router and Filter APIs execute here; provider transport is an explicit in-memory double. The expert-readiness contract separates this reproducible technical baseline from outstanding human review.

Published Router 0.2.1 includes the lazy HTTP-client import repair; Harness binds its reviewed receipt source to that exact release commit. Installing the package does not call a provider or make the receipt an action grant.

The published Corpus Pack SDK V1 adds a separate, canonical registry for optional namespaced boundary-invariant metadata. It preserves the frozen corpus 1.0.0, loads no package code, and treats complete evidence as readiness for later rule evaluation rather than a security verdict.

The published v1.3.0 companion adapter contracts exercise exact Transfer Verifier reports, Handoff metadata and Playbooks guidance through that SDK on Linux and Windows. This closes a concrete producer-to-consumer dataflow gap; it does not auto-load those installed distributions or make them production enforcement.

The published v1.3.0 Security Intelligence contour adds a provider-neutral offline weekly public-source review contract with digest-only evidence, explicit coverage gaps, and no live fetching or model authority.

The published v1.3.0 receipt auditors independently checks exact-pinned Router invocation and Cheap Filter triage accounting receipts from caller-supplied canonical bytes. Valid accounting remains inconclusive, missing evidence remains inconclusive, and drift becomes a finding; the auditors never emit pass, import or invoke the companion packages, or lower a security decision.

The published Extension Distribution Discovery V1 inspects one explicitly named local distribution without importing it. It verifies its RECORD, closed entry point, canonical manifest, implementation bytes and caller-supplied configuration digest, then requires exact reinspection before issuing an authority-free approval receipt. Harness still does not load package code: the operator supplies an already constructed object, and the binder checks it against the approved manifest pins.

The published Extension Operator Lifecycle V1 exposes that metadata-only inspection and exact-reinspection approval through safe CLI commands, adds canonical disable and non-executable rollback-plan receipts, and lists only explicitly supplied receipt state. It never imports, downloads, starts, stops, or rolls back extension code; the embedding application must construct and bind an object and enforce any accepted disable artifact.

The published controlled local adapter connects only to an operator-started literal-loopback /v1/responses endpoint and passes canonical tool calls through the existing closed Runtime Gateway policy. It supports local model names as opaque identifiers—including Qwen and DeepSeek-style names—without vendor claims. It has no DNS, proxy, redirect, credential, external-provider, arbitrary-tool, or upstream-MCP path; receipts are digest-only and operational authority remains none.

The published Policy Pack extension independently parses one exact-pinned data-only Playbooks pack and evaluates caller-supplied content-free signals bound to canonical observations. A missing pack is inconclusive; production Harness does not import or execute Playbooks code, discover packages, call a network, or grant allow/enforcement authority.

The deterministic result is 24 modeled findings for the vulnerable fixture and 0 modeled findings for the protected fixture. This is synthetic conformance evidence, not a production safety claim.

Inspect the committed before/after example in examples/comparison-report/ or validate every public example locally:

ash validate examples/
ash validate docs/evidence-status-registry.json

What this is / is not

The benchmark focuses on agent operating-environment boundaries, not just standalone model answers. Built-in targets are deterministic and offline. The experimental external adapter is explicit opt-in, prompt-only, and does not execute tools. See docs/benchmark-semantics.md and docs/authorized-testing-paths.md.

Visual evidence snapshot

The public evidence map separates deterministic executable specifications, sanitized local observations, historical material, and independently reviewed evidence:

• Evidence map
• Evidence classes
• Machine-readable evidence registry
• Evidence pack format
• Private/public evidence boundary

Public artifacts may include scenario identifiers, aggregate counts, response hashes, and validator results. They do not include raw private prompts, raw responses, synthetic canaries, or local paths. Keep external raw material under .internal/external-demo/latest. Do not commit raw_responses/ or other private runtime evidence.

Current stable surface

• Trace schema 1.0, with a bounded legacy 0.1 read/migration window.
• Corpus 1.0.0, freezing 24 ordered synthetic pattern identifiers.
• Local targets for vulnerable/protected agent, RAG, tool, function, and multi-agent handoff comparisons, including toy-multi-agent and protected-toy-multi-agent.
• JSON traces, scorecards, run manifests, remediation, Markdown reports, and self-contained HTML reports.
• Deterministic validators for artifact integrity and declared benchmark semantics.
• Linux/Python 3.11-3.13 as the primary installed-package contour, with Windows 3.11 compatibility coverage.
• Reproducible wheel/sdist builds, checksums, GitHub attestations, and exact-subject CycloneDX SBOMs for public releases from v1.0.0 onward.
• A shipped provider-neutral Agent Host V1 contour for canonical, authority-free record/replay, deterministic evaluation, explicit Python instrumentation, and a validated 48-case no-network quickstart. The CLI does not execute arbitrary hosts or tools and does not authenticate its producer or certify an external system; see Agent Host Adapter SDK.
• A shipped local Runtime Gateway synthetic contour with closed pre-dispatch policy, bounded OpenAI-compatible and stateless MCP endpoints, deterministic synthetic tools, privacy-minimized hash-chain audit, dashboard, and hardened source-build Docker Compose.
• Credential-free retained-envelope normalization for OpenAI Responses, Anthropic Messages, Google Interactions, and MCP tool calls. These adapters do not make provider calls or grant tool-execution authority.

The exact shipped, experimental, planned, and historical surfaces live in docs/current-state.md and docs/capability-matrix.md. Technical v1 gates and honest non-claims are in docs/v1-readiness.md.

If you only have one minute

• Run the no-network demo above.
• Read the committed comparison.
• Browse the showcase and scenario matrix.
• See weak spots and findings.
• Check current state and the project tracker.
• Bring your own local or OpenAI-compatible model through Run your model.

Use your own model or runtime

The shortest cross-platform operator path is docs/run-your-model.md. It covers:

1. a no-model deterministic demo;
2. one explicitly authorized OpenAI-compatible model;
3. a deterministic local swarm comparison;
4. a bounded local-model mini-swarm campaign.

Connection details and scenario selection are documented in docs/connect-models.md and docs/test-your-model.md. External runs are prompt-only self-report checks unless a separately reviewed host/tool adapter exists. A coherent answer is not evidence of safe tool execution.

Benchmark and evidence documentation

Start with the curated documentation map if you are not sure which contract, operator guide, or research page applies to your task.

The README is the front door; deeper contracts live in docs/:

Specialized reviewer paths:

• Local Prometheus workflow
• Local model profiles
• Multi-agent handoff toy topology
• Security audit causal map
• R5 sanitized research status
• Project governance

Standards and portfolio boundaries

The project publishes conservative mappings to OWASP LLM, NIST, and a direct-fit MITRE ATLAS subset in docs/standards-mapping.md. These are maintainer-reviewed mappings, not certification or independent standards validation. Independent review remains public follow-up work.

The related public contract includes:

• Threat ontology: 26 provider-neutral failure families.
• Scenario adjudication ledger: 127 bounded source units from 13 enumerated builders.
• Unified event envelope: separation of observations, data, authority, advisories, decisions, and effects.

The 127 units are not 127 canonical attacks and not a repository-wide total. The contract grants no executor, provider, deployment, enforcement, or production authority.

Public security stack

Agentic Security Harness is the benchmark/evidence layer in a small public defensive stack:

llm-safety-playbooks -> ai-agent-handoff -> agentic-transfer-verifier -> agentic-security-harness

• llm-safety-playbooks documents practical boundary rules.
• ai-agent-handoff represents task briefs and handoff state as reviewable files.
• agentic-transfer-verifier checks provenance, authority, approval, and audit evidence around transfers.
• This repository measures modeled failures and produces validated benchmark artifacts.

The repositories are related but not interchangeable. A playbook is not a runtime control, a handoff file is not a sandbox, and a passing benchmark is not a production certificate. The review-only ecosystem integration candidate builds the optional Transfer and Handoff extension wheels and exercises their explicit approval lifecycle on Ubuntu and Windows; it does not bundle or auto-install them.

Release and package status

Release v1.6.0 is published on PyPI and GitHub Releases. Its native adapter is now in the installed wheel; the installed-ecosystem example provides a hash-locked public-package route with positive and negative controls.

The current repository adds a 16-case functional chain, caller-input checks and the twelve-call proposal-contract evidence above. These examples and documentation are newer than the immutable release; use the current checkout to inspect or run them. pip install agentic-security-harness==1.6.0 installs the released runtime, not these repository files. No packaged runtime, dependency or public API changed in this follow-up, so it does not require a new wheel. The existing release artifacts and their embedded documentation remain unchanged; current guidance lives here.

The following prior release evidence is retained without rewriting its outcomes.

Prior release v1.5.1 is published on PyPI and GitHub Releases. Published-release verification passed exact-subject provenance/index checks and clean Linux/Python 3.11-3.13 plus Windows/Python 3.11 installation. Publication makes the bounded core and selected passive distributions installable; it is not automatic activation, production deployment, enforcement, provider authority, or security certification.

The v1.5.0 release adds explicit Quarantine and advisory-to-Gateway composition plus exact-pinned external Playbooks receipt ingress. These pure APIs stop at a Gateway decision: they do not dispatch, activate companions, authenticate producers or infer safe model intent. The v1.5.1 notes retain the initial TestPyPI and Windows PyPI index-lookup failures and successful read-only recovery; no published package was rebuilt or uploaded again. Historical v1.5.0 verification 35507083204 and its release artifacts remain unchanged.

• Release checklist
• PyPI release process
• Changelog
• Security policy

Independent standards review and a durable second GitHub reviewer are transparent post-v1 credibility tasks. Their absence does not change deterministic test results, but independent review is not claimed.

Development

python -m pip install -e ".[dev]"
python -m pytest
python -m ruff check .
python -m mypy src tests tools
ash validate examples/

Development follows idea -> issue -> branch -> implementation -> tests/artifacts -> PR -> GitHub checks -> review gate. See docs/git-evidence-workflow.md, CONTRIBUTING.md, and docs/agent-operating-guide.md.

Responsible use

Use only synthetic, local, owned, or explicitly authorized targets. Do not use this project for credential theft, persistence, evasion, destructive payloads, or unauthorized systems. See SECURITY.md and docs/authorized-testing-paths.md.

Citation, contributing, and license

• Citation metadata: CITATION.cff
• Governance: GOVERNANCE.md
• Contributing: CONTRIBUTING.md
• Support: SUPPORT.md
• License: Apache-2.0, see LICENSE and NOTICE