Agentic Security Harness

Distribution version: 1.13.1. Python 3.11 or newer. Apache-2.0.

A local defensive benchmark and toolkit for testing agent boundary failures: untrusted input, authority confusion, retained history and controlled file actions. It produces replayable traces, scorecards and separately verifiable evidence.

This description belongs to the distribution you are viewing. For live publication status, exact artifact hashes and verification runs, see the release records and current project state. Version metadata alone does not attest index availability or publication.

Install this version

python -m pip install agentic-security-harness==1.13.1
ash quickstart --out reports/quickstart

Use a fresh output directory. The quickstart runs fixed offline scenarios against vulnerable and protected fixtures; it does not call a model or an external target.

Check a controlled file boundary

ash controlled-file-workflow --out reports/file-boundary
ash controlled-file-verify --out reports/file-boundary

This opt-in workflow performs actual writes to newly created synthetic files. Report-only Runtime Guard authority permits the report file and denies the protected file. A separate read-only verifier checks disk bytes and captured history. Model output cannot supply a filesystem path or executable code. The default run is offline. An existing local Ollama model can be selected explicitly using the documented options; installing this package starts no model.

Versioned instructions and limits explain the guarded/ablated fixed control separately from model observations. Permission enforcement and correctness of the generated report are different measurements. This is not an arbitrary tool executor or a hostile-code sandbox.

This version also includes ControlledFileSession, a narrow fresh-fixture hook for trusted applications to submit proposal bytes through the same boundary. An optional repository example connects that hook to a Pydantic AI tool callback; Pydantic AI is not a base dependency. The application owns context, labels and files, and the model supplies only proposal text. Storage failures retain an incomplete directory for inspection; they do not undo earlier completed writes.

Save a configured text document

The guarded workspace writer lets a trusted host configure an output directory and aliases for new UTF-8 files. An agent proposes write_text with an alias and document content. The host's deterministic Runtime Guard decides whether the proposal may create that file. The create-only writer reads back the output and records content-free intent and result receipts; an existing file is preserved.

Use WorkspacePolicy and GuardedWorkspace as an application tool, or the installed ash workspace-check, ash workspace-write and ash workspace-verify commands. The optional ash workspace-run --execute makes one request to an existing local Ollama service. No model call is needed for the Guard decision. The host owns paths, policy and classification; proposed text cannot select arbitrary paths or label its own sensitivity. This in-process tool is not an OS sandbox or a semantic checker of the saved document.

Run separate document jobs

The document workflow adds host configuration, a no-effect preview, an explicit one-generation job, read-only status and an optional Pydantic AI bridge. A later job may use a verified previous output as untrusted source text, never as inherited authority. The host still chooses the output policy and any exact JSON or checklist requirements. Quality status describes only declared deterministic checks; even checked does not certify meaning or factual accuracy. An invalid-format document remains saved as a draft but is not reusable as a chain input.

This version also binds document generation to a host-selected content-only tool: model text cannot choose an output alias, path or authority. Reusing a review-required document needs an exact-byte host review digest; a failed quality check still blocks reuse. Hosts can bind input restrictions to source bytes and original expiry times, including bounded multi-source composition. Fixed plans and dynamic admissions declare expected jobs and pending choices; coverage reports whether declared history is accounted for, not whether the documents are factually right. Optional supervised execution may fence an interrupted owned child and permit exact reviewed bytes as data for a new job; the original job remains unresolved and cannot be replayed. No base model-framework dependency, arbitrary-path tool or OS sandbox is added. The versioned operator and replay guide separates scripted transport, real-model use and human review.

Other included surfaces

• Portable trace validation, scorecards and deterministic comparison reports.
• Retained ancestry admission and the closed synthetic Runtime Gateway.
• An experimental single-plan garak adapter, not an official NVIDIA integration or support for every garak detector.
• Separately versioned optional Transfer, Handoff, Playbooks, Router and Filter packages. Installation does not automatically activate them.

Evidence and scope

The earlier eight-call observation is historical, and this version does not turn it into a new package-specific security-effectiveness claim. The historical eight-call observation used a specific installed 1.9.0 candidate: four protected proposals were denied, four reports were written, and two reports were correct. It is not a measurement of this version or a reliability estimate.

A separate six-call local Qwen2.5 1.5B integration observation for this hook wrote four reports, of which two had correct counts, and denied two protected writes. The eight fixed seed cases and separate offline timing runs are different evidence classes. None establishes cross-model reliability or production containment.

Historical observation and methodology separate controlled evidence from broader claims. Production-wide containment, arbitrary host protection and independent human review are not established.

Author: Dmitry Krivonosov (krivonosoff161). Questions and reproducible reports: GitHub issues.