Agentic Security Harness

Your AI coding agent reads untrusted repository text. Can it keep data separate from instructions and authority?

Agentic Security Harness is a local, trace-first benchmark for defensive testing of agentic AI boundary failures. It runs reproducible synthetic scenarios, records portable traces and scorecards, and compares a deliberately vulnerable local agent with a protected one.

In plain English: it turns “the agent behaved unsafely” into evidence you can replay, validate, compare, and review.

Quickstart

Install the stable release from PyPI:

python -m pip install agentic-security-harness==1.2.0
ash quickstart --out reports/quickstart
ash agent-host-quickstart --out reports/agent-host-quickstart

For source development:

git clone https://github.com/krivonosoff161/agentic-security-harness.git
cd agentic-security-harness
python -m pip install .
ash quickstart --out reports/quickstart
ash agent-host-quickstart --out reports/agent-host-quickstart

ash quickstart is local and no-network. It runs the same stable 24-pattern corpus against both demo targets, validates the generated artifacts, and renders a self-contained HTML report.

ash agent-host-quickstart is the shipped provider-neutral integration path in v1.1.0. It runs a built-in owned synthetic workflow through the instrumented collector, canonical recordings, deterministic evaluator, atomic bundle publication, and shared validator. It retains digest-only public evidence and makes no provider call.

Runtime Gateway synthetic contour

The source tree now also contains a runnable local Runtime Gateway increment. It applies a closed policy before synthetic tool dispatch, exposes bounded OpenAI-compatible and MCP 2026-07-28 stateless development endpoints, and maintains a privacy-minimized append-only audit chain:

ash gateway-init --out gateway.toml
ash gateway-check --config gateway.toml
ash gateway-serve --config gateway.toml

Open http://127.0.0.1:8787/dashboard after startup, or use the hardened Docker Compose profile in Runtime Gateway synthetic contour. This is a credential-free synthetic integration surface. Offline provider-neutral tool-call adapters normalize retained OpenAI Responses, Anthropic Messages, Google Interactions, and MCP payloads through the same policy without SDKs or credentials. Live provider transport and a production firewall are still not shipped. The gateway exposes its exact closed policy and non-executable approval-request digests; it intentionally has no approval-grant endpoint yet.

The deterministic result is 24 modeled findings for the vulnerable fixture and 0 modeled findings for the protected fixture. This is synthetic conformance evidence, not a production safety claim.

Inspect the committed before/after example in examples/comparison-report/ or validate every public example locally:

ash validate examples/
ash validate docs/evidence-status-registry.json

What this is / is not

The benchmark focuses on agent operating-environment boundaries, not just standalone model answers. Built-in targets are deterministic and offline. The experimental external adapter is explicit opt-in, prompt-only, and does not execute tools. See docs/benchmark-semantics.md and docs/authorized-testing-paths.md.

Visual evidence snapshot

The public evidence map separates deterministic executable specifications, sanitized local observations, historical material, and independently reviewed evidence:

• Evidence map
• Evidence classes
• Machine-readable evidence registry
• Evidence pack format
• Private/public evidence boundary

Public artifacts may include scenario identifiers, aggregate counts, response hashes, and validator results. They do not include raw private prompts, raw responses, synthetic canaries, or local paths. Keep external raw material under .internal/external-demo/latest. Do not commit raw_responses/ or other private runtime evidence.

Current stable surface

• Trace schema 1.0, with a bounded legacy 0.1 read/migration window.
• Corpus 1.0.0, freezing 24 ordered synthetic pattern identifiers.
• Local targets for vulnerable/protected agent, RAG, tool, function, and multi-agent handoff comparisons, including toy-multi-agent and protected-toy-multi-agent.
• JSON traces, scorecards, run manifests, remediation, Markdown reports, and self-contained HTML reports.
• Deterministic validators for artifact integrity and declared benchmark semantics.
• Linux/Python 3.11-3.13 as the primary installed-package contour, with Windows 3.11 compatibility coverage.
• Reproducible wheel/sdist builds, checksums, GitHub attestations, and exact-subject CycloneDX SBOMs for public releases from v1.0.0 onward.
• A shipped provider-neutral Agent Host V1 contour for canonical, authority-free record/replay, deterministic evaluation, explicit Python instrumentation, and a validated 48-case no-network quickstart. The CLI does not execute arbitrary hosts or tools and does not authenticate its producer or certify an external system; see Agent Host Adapter SDK.
• A shipped local Runtime Gateway synthetic contour with closed pre-dispatch policy, bounded OpenAI-compatible and stateless MCP endpoints, deterministic synthetic tools, privacy-minimized hash-chain audit, dashboard, and hardened source-build Docker Compose.
• Credential-free retained-envelope normalization for OpenAI Responses, Anthropic Messages, Google Interactions, and MCP tool calls. These adapters do not make provider calls or grant tool-execution authority.

The exact shipped, experimental, planned, and historical surfaces live in docs/current-state.md and docs/capability-matrix.md. Technical v1 gates and honest non-claims are in docs/v1-readiness.md.

If you only have one minute

• Run the no-network demo above.
• Read the committed comparison.
• Browse the showcase and scenario matrix.
• See weak spots and findings.
• Check current state and the project tracker.
• Bring your own local or OpenAI-compatible model through Run your model.

Use your own model or runtime

The shortest cross-platform operator path is docs/run-your-model.md. It covers:

1. a no-model deterministic demo;
2. one explicitly authorized OpenAI-compatible model;
3. a deterministic local swarm comparison;
4. a bounded local-model mini-swarm campaign.

Connection details and scenario selection are documented in docs/connect-models.md and docs/test-your-model.md. External runs are prompt-only self-report checks unless a separately reviewed host/tool adapter exists. A coherent answer is not evidence of safe tool execution.

Benchmark and evidence documentation

Start with the curated documentation map if you are not sure which contract, operator guide, or research page applies to your task.

The README is the front door; deeper contracts live in docs/:

Specialized reviewer paths:

• Local Prometheus workflow
• Local model profiles
• Multi-agent handoff toy topology
• Security audit causal map
• R5 sanitized research status
• Project governance

Standards and portfolio boundaries

The project publishes conservative mappings to OWASP LLM, NIST, and a direct-fit MITRE ATLAS subset in docs/standards-mapping.md. These are maintainer-reviewed mappings, not certification or independent standards validation. Independent review remains public follow-up work.

The related public contract includes:

• Threat ontology: 26 provider-neutral failure families.
• Scenario adjudication ledger: 127 bounded source units from 13 enumerated builders.
• Unified event envelope: separation of observations, data, authority, advisories, decisions, and effects.

The 127 units are not 127 canonical attacks and not a repository-wide total. The contract grants no executor, provider, deployment, enforcement, or production authority.

Public security stack

Agentic Security Harness is the benchmark/evidence layer in a small public defensive stack:

llm-safety-playbooks -> ai-agent-handoff -> agentic-transfer-verifier -> agentic-security-harness

• llm-safety-playbooks documents practical boundary rules.
• ai-agent-handoff represents task briefs and handoff state as reviewable files.
• agentic-transfer-verifier checks provenance, authority, approval, and audit evidence around transfers.
• This repository measures modeled failures and produces validated benchmark artifacts.

The repositories are related but not interchangeable. A playbook is not a runtime control, a handoff file is not a sandbox, and a passing benchmark is not a production certificate.

Release and package status

The stable v1.2.0 package is published on PyPI and as an attested GitHub Release. Future releases remain separate, auditable owner gates.

• Release checklist
• PyPI release process
• Changelog
• Security policy

Independent standards review and a durable second GitHub reviewer are transparent post-v1 credibility tasks. Their absence does not change deterministic test results, but independent review is not claimed.

Development

python -m pip install -e ".[dev]"
python -m pytest
python -m ruff check .
python -m mypy src tests tools
ash validate examples/

Development follows idea -> issue -> branch -> implementation -> tests/artifacts -> PR -> GitHub checks -> review gate. See docs/git-evidence-workflow.md, CONTRIBUTING.md, and docs/agent-operating-guide.md.

Responsible use

Use only synthetic, local, owned, or explicitly authorized targets. Do not use this project for credential theft, persistence, evasion, destructive payloads, or unauthorized systems. See SECURITY.md and docs/authorized-testing-paths.md.

Citation, contributing, and license

• Citation metadata: CITATION.cff
• Governance: GOVERNANCE.md
• Contributing: CONTRIBUTING.md
• Support: SUPPORT.md
• License: Apache-2.0, see LICENSE and NOTICE