Agentic Security Harness

Distribution version: 1.10.1. Python 3.11 or newer. Apache-2.0.

A local defensive benchmark and toolkit for testing agent boundary failures: untrusted input, authority confusion, retained history and controlled file actions. It produces replayable traces, scorecards and separately verifiable evidence.

This description belongs to the distribution you are viewing. For live publication status, exact artifact hashes and verification runs, see the release records and current project state.

Install this version

python -m pip install agentic-security-harness==1.10.1
ash quickstart --out reports/quickstart

Use a fresh output directory. The quickstart runs fixed offline scenarios against vulnerable and protected fixtures; it does not call a model or an external target.

Check a controlled file boundary

ash controlled-file-workflow --out reports/file-boundary
ash controlled-file-verify --out reports/file-boundary

This opt-in workflow performs actual writes to newly created synthetic files. Report-only Runtime Guard authority permits the report file and denies the protected file. A separate read-only verifier checks disk bytes and captured history. Model output cannot supply a filesystem path or executable code. The default run is offline. An existing local Ollama model can be selected explicitly using the documented options; installing this package starts no model.

Versioned instructions and limits explain the guarded/ablated fixed control separately from model observations. Permission enforcement and correctness of the generated report are different measurements. This is not an arbitrary tool executor or a hostile-code sandbox.

This version also includes ControlledFileSession, a narrow fresh-fixture hook for trusted applications to submit proposal bytes through the same boundary. An optional repository example connects that hook to a Pydantic AI tool callback; Pydantic AI is not a base dependency. The application owns context, labels and files, and the model supplies only proposal text. Storage failures retain an incomplete directory for inspection; they do not undo earlier completed writes.

Other included surfaces

• Portable trace validation, scorecards and deterministic comparison reports.
• Retained ancestry admission and the closed synthetic Runtime Gateway.
• An experimental single-plan garak adapter, not an official NVIDIA integration or support for every garak detector.
• Separately versioned optional Transfer, Handoff, Playbooks, Router and Filter packages. Installation does not automatically activate them.

Evidence and scope

The earlier eight-call observation is historical, and this version does not turn it into a new package-specific security-effectiveness claim. The historical eight-call observation used a specific installed 1.9.0 candidate: four protected proposals were denied, four reports were written, and two reports were correct. It is not a measurement of this version or a reliability estimate.

A separate six-call local Qwen2.5 1.5B integration observation for this hook wrote four reports, of which two had correct counts, and denied two protected writes. The eight fixed seed cases and separate offline timing runs are different evidence classes. None establishes cross-model reliability or production containment.

Historical observation and methodology separate controlled evidence from broader claims. Production-wide containment, arbitrary host protection and independent human review are not established.

Author: Dmitry Krivonosov (krivonosoff161). Questions and reproducible reports: GitHub issues.