Agentic Security Harness

Your AI coding agent reads untrusted repository text. Can it keep data separate from instructions and authority?

Agentic Security Harness is a local, trace-first benchmark for defensive testing of agentic AI boundary failures. It runs reproducible synthetic scenarios, records portable traces and scorecards, and compares a deliberately vulnerable local agent with a protected one.

In plain English: it turns “the agent behaved unsafely” into evidence you can replay, validate, compare, and review.

Quickstart

Install the current source checkout:

git clone https://github.com/krivonosoff161/agentic-security-harness.git
cd agentic-security-harness
python -m pip install .
ash quickstart --out reports/quickstart

After the separately gated PyPI publication, the primary install command becomes:

python -m pip install agentic-security-harness

ash quickstart is local and no-network. It runs the same stable 24-pattern corpus against both demo targets, validates the generated artifacts, and renders a self-contained HTML report.

The deterministic result is 24 modeled findings for the vulnerable fixture and 0 modeled findings for the protected fixture. This is synthetic conformance evidence, not a production safety claim.

Inspect the committed before/after example in examples/comparison-report/ or validate every public example locally:

ash validate examples/
ash validate docs/evidence-status-registry.json

What this is / is not

The benchmark focuses on agent operating-environment boundaries, not just standalone model answers. Built-in targets are deterministic and offline. The experimental external adapter is explicit opt-in, prompt-only, and does not execute tools. See docs/benchmark-semantics.md and docs/authorized-testing-paths.md.

Visual evidence snapshot

The public evidence map separates deterministic executable specifications, sanitized local observations, historical material, and independently reviewed evidence:

• Evidence map
• Evidence classes
• Machine-readable evidence registry
• Evidence pack format
• Private/public evidence boundary

Public artifacts may include scenario identifiers, aggregate counts, response hashes, and validator results. They do not include raw private prompts, raw responses, synthetic canaries, or local paths. Keep external raw material under .internal/external-demo/latest. Do not commit raw_responses/ or other private runtime evidence.

Current stable surface

• Trace schema 1.0, with a bounded legacy 0.1 read/migration window.
• Corpus 1.0.0, freezing 24 ordered synthetic pattern identifiers.
• Local targets for vulnerable/protected agent, RAG, tool, function, and multi-agent handoff comparisons, including toy-multi-agent and protected-toy-multi-agent.
• JSON traces, scorecards, run manifests, remediation, Markdown reports, and self-contained HTML reports.
• Deterministic validators for artifact integrity and declared benchmark semantics.
• Linux/Python 3.11-3.13 as the primary installed-package contour, with Windows 3.11 compatibility coverage.
• Reproducible wheel/sdist builds, checksums, GitHub attestations, and an exact-subject CycloneDX SBOM in the next authorized tag workflow.

The exact shipped, experimental, planned, and historical surfaces live in docs/current-state.md and docs/capability-matrix.md. Technical v1 gates and honest non-claims are in docs/v1-readiness.md.

If you only have one minute

• Run the no-network demo above.
• Read the committed comparison.
• Browse the showcase and scenario matrix.
• See weak spots and findings.
• Check current state and the project tracker.
• Bring your own local or OpenAI-compatible model through Run your model.

Use your own model or runtime

The shortest cross-platform operator path is docs/run-your-model.md. It covers:

1. a no-model deterministic demo;
2. one explicitly authorized OpenAI-compatible model;
3. a deterministic local swarm comparison;
4. a bounded local-model mini-swarm campaign.

Connection details and scenario selection are documented in docs/connect-models.md and docs/test-your-model.md. External runs are prompt-only self-report checks unless a separately reviewed host/tool adapter exists. A coherent answer is not evidence of safe tool execution.

Benchmark and evidence documentation

The README is the front door; deeper contracts live in docs/:

Specialized reviewer paths:

• Local Prometheus workflow
• Local model profiles
• Multi-agent handoff toy topology
• Security audit causal map
• R5 sanitized research status
• Project governance

Standards and portfolio boundaries

The project publishes conservative mappings to OWASP LLM, NIST, and a direct-fit MITRE ATLAS subset in docs/standards-mapping.md. These are maintainer-reviewed mappings, not certification or independent standards validation. Independent review remains public follow-up work.

The related public contract includes:

• Threat ontology: 26 provider-neutral failure families.
• Scenario adjudication ledger: 127 bounded source units from 13 enumerated builders.
• Unified event envelope: separation of observations, data, authority, advisories, decisions, and effects.

The 127 units are not 127 canonical attacks and not a repository-wide total. The contract grants no executor, provider, deployment, enforcement, or production authority.

Public security stack

Agentic Security Harness is the benchmark/evidence layer in a small public defensive stack:

llm-safety-playbooks -> ai-agent-handoff -> agentic-transfer-verifier -> agentic-security-harness

• llm-safety-playbooks documents practical boundary rules.
• ai-agent-handoff represents task briefs and handoff state as reviewable files.
• agentic-transfer-verifier checks provenance, authority, approval, and audit evidence around transfers.
• This repository measures modeled failures and produces validated benchmark artifacts.

The repositories are related but not interchangeable. A playbook is not a runtime control, a handoff file is not a sandbox, and a passing benchmark is not a production certificate.

Release and package status

The source tree is prepared for the stable v1 contract. Tagging, GitHub Release creation, TestPyPI upload, and PyPI publication remain separate, auditable owner gates. Until the package-index promotion succeeds, source and GitHub Release installation are the supported paths.

• Release checklist
• PyPI release process
• Changelog
• Security policy

Independent standards review and a durable second GitHub reviewer are transparent post-v1 credibility tasks. Their absence does not change deterministic test results, but independent review is not claimed.

Development

python -m pip install -e ".[dev]"
python -m pytest
python -m ruff check .
python -m mypy src tests tools
ash validate examples/

Development follows idea -> issue -> branch -> implementation -> tests/artifacts -> PR -> GitHub checks -> review gate. See docs/git-evidence-workflow.md, CONTRIBUTING.md, and docs/agent-operating-guide.md.

Responsible use

Use only synthetic, local, owned, or explicitly authorized targets. Do not use this project for credential theft, persistence, evasion, destructive payloads, or unauthorized systems. See SECURITY.md and docs/authorized-testing-paths.md.

Citation, contributing, and license

• Citation metadata: CITATION.cff
• Governance: GOVERNANCE.md
• Contributing: CONTRIBUTING.md
• Support: SUPPORT.md
• License: Apache-2.0, see LICENSE and NOTICE