agentdrill

Prove — Reproducible attack packs run in CI against your deployed agent-defense stack, with honest, published results — including the attacks we did not stop.

Part of the agentic-suite — open-source, harness-agnostic security for AI agents.

Status: v0.1.0 — in progress. The foundation has landed: the ported engine (M0) and the CLI, fail-closed configuration, CI quality gate, and identity docs (M1). The proof engine is wired end-to-end at M18; the PRD is the design of record. The intent is published early, on purpose, so we can be held to it.

---

Install & Quick Start

# Install (PyPI)
pip install agentsec-agentdrill        # Python CLI + SDK (import + CLI: agentdrill)
agentdrill --help                      # list commands

# First run in ~15 minutes, against a bundled demo stack (no real stack required)
agentdrill quickstart

# Drill your own deployed stack
agentdrill run --pack replit-destructive-command --target ./my-stack.yaml --sandbox

# Gate a policy change in CI — a rule that un-blocks an attack is rejected
agentdrill run --pack incidents-core --target ./stack.yaml --ci

# Verify a published result with nothing installed
agentdrill verify ./benchmark-release.json

The proof loop: pick a pack → point at your deployed stack → run locally or in CI → read an honest result (blocked / missed / degraded) + kill-time. See PRD 04.

See it before installing

Run agentdrill quickstart to boot a static, bundled demo stack and produce a first drill report — one pack's honest outcome split, a kill-chain timeline, and an exportable evidence bundle. It runs offline, opens from file://, needs no account, and is pure synthetic data watermarked demo — never evidence.

---

Why

Every agent-security vendor oversells detection, and almost nobody publishes reproducible evidence of what their controls actually stop. The benchmark record is brutal: simple prompt injection retains ~46% ASR against some of the best-known defenses, and one hacking prompt nearly doubles ASR against GPT-4 ReAct agents. Meanwhile red-team tooling (PyRIT, garak, promptfoo, AgentDojo) is strong but offline and point-in-time — it red-teams a snapshot, not a live policy stack.

Nothing treats agent policies as continuously regression-tested code. agentdrill is that layer: the ecosystem's proof-of-work engine. See PRD 01.

What it does

Capability	Description
Attack packs	Curated, versioned incident-replay packs (Replit destructive-command, MCP tool poisoning, Plugin4Shell supply-chain, Slack-AI exfil, credential evasion)
Runs the deployed stack	Drives the real enforcement path (MCP proxy, credential broker, confinement, compiled policy, halt) — never a simulation
Honest results	Every published table includes the misses; no scoring theater
Kill-time verification	Times revocation to actual cessation (probed, not inferred) — a full kill-chain timeline, not one latency
CI gate	A policy PR that un-blocks a known attack is rejected before merge
Public benchmark	Reproducible per-release results — a third party re-runs and gets the same numbers
Efficacy control	A blocked number only counts if the attack demonstrably succeeds unguarded (the control group no one else runs)
Evasion pack	Attacks on the ecosystem itself (tamper, hook-disable, policy-edit, telemetry-injection) run every release
Interop	Import promptfoo / garak / AgentDojo / PyRIT suites — don't reinvent
Standards-tagged	Every pack mapped to MITRE ATLAS + OWASP (LLM + Agentic)
Harness-agnostic	Works across harnesses by construction, or ships named per-harness packs

Honest by design

This is the differentiator, so it is enforced, not aspirational (PRD 07, PRD 20):

• Misses are first-class — a run cannot publish without them.
• A single run is never a claim — published numbers are mean ± std over ≥3 pinned seeds; kill-time reports p50/p95/max.
• Contamination is disclosed, not hidden — cases overlapping a judge/target model's training data are flagged in the table.
• Flag, never silently fix — expected outcomes are preregistered; a change is flagged with a re-pin note.
• Disputes get answered — a public adjudication process with a 14-day SLA on every published finding.

Compatibility

A harness is "compatible" only when its pack passes in CI; release notes carry the compatibility table (PRD 05 §Compatibility).

Tier	Harness	Support
1 — Coding agents	Claude Code · Cursor · GitHub Copilot · Codex CLI · Gemini CLI	≥2 at v0.1.0 (per-harness packs)
2 — Frameworks	LangGraph · CrewAI · OpenAI Agents SDK · AutoGen/AG2 · LlamaIndex · PydanticAI · smolagents	via promptfoo/garak interop; adapter packs in scope
3 — Protocol/enterprise	Generic MCP clients · enterprise platforms	MCP-layer coverage via agentgate; enterprise via standards

Security & supply chain

• Signed packs + provenance on every release (PRD 17).
• Trust levels (built-in / local / external) — external packs are sandbox-restricted, never used for benchmark results.
• Target attestation (live / fixture / mock) — results are labeled at the weakest link; mock never publishes.
• Standalone verifier — a zero-install, offline check of signatures, hashes, manifests, and labels; a bundle is verified on a machine that never ran the drill.
• Evidence privacy — target-side secrets redacted before bundling, with verifiable redaction receipts.
• Self-audit — the evasion pack is the sanctioned way we attack ourselves; disclosure per SECURITY.md.

Verification & quality gates

make setup            # create the venv + install dev extras
make lint             # ruff, zero violations
make typecheck        # mypy --strict
make test             # functional test suite (excludes docker/omlx infra tests)
make cov              # functional suite with the coverage gate (>=95%)
make test-all         # full suite incl. docker/omlx (needs a daemon + oMLX server)

Self-attack, benchmark, and release-verification targets (drill-self / benchmark / verify-release) land with their milestones (M14 / M10 / M32).

• Published benchmark: honest outcome split per release, misses included (PRD 07).
• Every published number is efficacy-backed and multi-seed; every row links its traces (PRD 19).

Development

Python core (pack format, runner, driver/target adapters, oracle engine, reporter, registry), CLI wrapper, and a pytest-style API — one engine, every surface (PRD 02).

make setup        # editable install with dev extras
agentdrill list packs        # discover packs / oracles / targets / drivers
agentdrill run --dry-run --pack incidents-core --target ./stack.yaml
agentdrill quickstart        # end-to-end against the bundled demo stack

Documentation

Full index: docs/prd/README.md.

Why & scope

• Press Release / FAQ · Why · Landscape · Non-Goals

How it works

• Architecture · Data Model · Kill-Chain Verification · Attack-Chain Orchestration

What ships

• What (Features) · Pack Catalog · Detection Validation & BAS-Class · Frontier & DX

Trust

• Security Baseline · Success Metrics · Efficacy, Freshness & Lifecycle · Oracle Credibility

Users, parity & governance

• Users & CUJs · Feature Parity · Predecessor Gap Closure · Decisions · Traceability

Operations

• Risks · Non-Functional Requirements · Pack Registry & Governance · Roadmap

Project

• Contributing · Governance · Code of Conduct
• Security policy
• Roadmap · Org roadmap
• License: Apache-2.0

License

Apache License 2.0 — see LICENSE.