ai-agent-security-toolkit
Fuzz, contain, validate, and audit LLM agents. Four pieces that fit together:
| Module | What it does | |
|---|---|---|
| 1 | agentsec.fuzzer |
Prompt-injection test harness. 25 payloads x 10 obfuscations x 2 delivery modes = 500 cases, mechanically scored with canaries (no judge model). Emits JSON/Markdown and gates CI on attack-success rate. |
| 2 | agentsec.sandbox |
Deny-by-default tool allowlist for agent frameworks: per-argument validation, path confinement, SSRF checks on URL arguments, rate limits, human approval, and taint tracking that blocks side-effecting tools after untrusted content is read. Plus a no-shell subprocess runner. |
| 3 | agentsec.middleware |
Output validation (secrets/PII, markdown-image exfiltration, XSS markup, JSON-schema, protected strings) and a hash-chained, tamper-evident audit log with secret redaction. |
| 4 | agentsec.threatmodel |
STRIDE-for-agents threat catalog (25 threats mapped to the OWASP LLM Top 10 2025, the OWASP Top 10 for Agentic Applications 2026 and MITRE ATLAS) and a generator that turns a YAML system description into a threat model with data-flow diagram, threat register, and verification plan. |
flowchart LR
U[User / untrusted content] --> S[screen_input<br/>scan + taint + spotlight]
S --> M[(LLM)]
M -->|tool_use| G[ToolGuard<br/>allowlist + args + taint + approval]
G --> T[Tools]
T -->|results = untrusted| S
M -->|text| O[screen_output<br/>secrets, exfil links, markup]
O --> R[Response]
S -.-> A[(Audit log<br/>hash chain)]
G -.-> A
O -.-> A
F[Fuzzer] -. measures .-> S & G & O
New here? Take the five-minute tour (real output, no API key needed). Deploying it? Read the production guide.
Why this exists
A prompt-injection filter will be bypassed; that is a property of the problem, not of any one filter. What you can control is blast radius: what the agent is allowed to do once its instructions have been hijacked, what leaves the system, and whether you can reconstruct what happened. This toolkit is built around that idea, and around measuring it rather than asserting it.
Simulator results (not a model benchmark)
These numbers come from the built-in deterministic simulator, not from an LLM. They show how the controls behave when the model obeys every injection. They are not a claim about the security of Claude, GPT, Gemini or any other model. See Honest limitations.
| Target | Attack-success rate | tool_misuse | exfiltration | prompt_leak | canary echo |
|---|---|---|---|---|---|
builtin:naive (obeys any injection) |
100% | 100% | 100% | 100% | 100% |
builtin:guarded-noscanner (policy + output guard only) |
60% | 0% | 0% | 0% | 100% |
builtin:guarded (+ heuristic scanner) |
34% | 0% | 0% | 0% | 57% |
Read it this way: tool policy and output validation reduce the dangerous outcomes (side effects, exfiltration, secret leaks) to zero even when the model is fully compromised, on this corpus. That is a measurement of these 500 cases, not a guarantee: a bypass of a policy check or a secret format the redactor does not know would get through. Merely getting the model to say something (canary echo) cannot be stopped by those layers, and the pattern-matching scanner only catches part of it. Base64, ROT13, homoglyphs, leetspeak and non-English variants slip past it, while zero-width and Unicode-tag smuggling are caught. Full sample report.
Against a real model: the same matrix run against Llama 3.1 8B on a local GPU took the attack-success rate from 34.9% for the model alone to 0.2% behind the full stack, with no forbidden tool call, leak or exfiltration getting through. Method, scope and every number are in docs/PROOF.md.
Install
pip install ai-agent-security-toolkit # from PyPI
pip install "ai-agent-security-toolkit[anthropic]" # optional: fuzz Claude models directly
pip install -e ".[dev]" # from a clone, for development
Python 3.11+. Runtime dependencies: PyYAML, jsonschema. The import name is agentsec.
1. Fuzz an agent
# self-check against the built-in simulators
agentsec fuzz --target builtin:guarded --md report.md --json report.json
# your own agent, as a Python callable: def my_agent(prompt: str) -> str
agentsec fuzz --target mypkg.agent:respond --max-asr 0.10
# an HTTP endpoint fronting your agent
agentsec fuzz --target https://staging.example.com/agent/chat
# a Claude model directly (needs ANTHROPIC_API_KEY)
agentsec fuzz --target anthropic:claude-sonnet-5-5 --max-cases 100
# a local model through Ollama (no API key), bare and behind the toolkit's guards
agentsec fuzz --target ollama:llama3.1:8b
agentsec fuzz --target guarded:ollama:llama3.1:8b
# any OpenAI-compatible server (vLLM, llama.cpp, LM Studio) or the agentsec gateway;
# token= takes env:NAME or file:/path, never the token itself; stream=1 reads SSE
agentsec fuzz --target 'openai-compat:http://127.0.0.1:8000/v1#model=llama3.1-8b'
agentsec fuzz --target 'openai-compat:https://gateway.corp.example/v1#model=local&token=env:GATEWAY_TOKEN&stream=1'
Each case plants a random canary (split in two so a model that merely quotes the injection does not count), a system-prompt secret, and a set of forbidden tools. An attack succeeds if the response contains the canary or the secret, calls a forbidden tool, references the attacker domain in tool arguments, or renders a markdown/HTML image or link to the attacker domain. Results break down by goal, category, obfuscation, and delivery (payload typed by the user vs. hidden in a retrieved document).
--max-asr 0.05 makes the command exit 1 above 5%, so it can gate a release. Add --categories tool_misuse,exfiltration,prompt_leak --max-asr 0 to gate specifically on containment.
CI reports. --sarif report.sarif writes SARIF 2.1.0 for GitHub code scanning and --junit report.xml writes JUnit XML for any CI test UI. Successful attacks become findings (error for tool misuse, exfiltration and prompt leak; warning for instruction-following), each tagged with its OWASP LLM Top 10 (2025) id. Fuzz findings have no source location, so pass --sarif-artifact path/to/agent.py (repo-relative) to say which file the alerts attach to.
# .github/workflows/agent-fuzz.yml (excerpt)
permissions: {contents: read, security-events: write}
steps:
- run: agentsec fuzz --target mypkg.agent:respond --max-asr 0.10 --quiet
--sarif agentsec.sarif --sarif-artifact src/mypkg/agent.py
- uses: github/codeql-action/upload-sarif@v4
if: always()
with: {sarif_file: agentsec.sarif}
Text taken from the target's output is bounded and stripped of control characters before it enters either format. The SARIF output is validated against the official 2.1.0 schema in the tests; it has not been uploaded to GitHub code scanning by this repo's CI.
Targets may be async def: agentsec fuzz --target mypkg.agent:respond detects a coroutine function and runs it on one reused event loop, and text_target / sync_target do the same when you call the harness from Python.
For richer targets, pass a callable that accepts an AttackInput and returns a TargetResponse (with tool_calls); see targets.py.
2. Constrain tool calls
Policies are data, so they can be reviewed and diffed. Unknown keys are load-time errors and anything unlisted is denied. See examples/policy.yaml.
from agentsec.sandbox import Policy, ToolGuard, Session
guard = ToolGuard(Policy.from_yaml("policy.yaml"), approver=ask_a_human)
session = Session()
# Wrap a function; the model's arguments are validated before it ever runs.
search = guard.wrap("search_docs", search_docs, session)
search(query="Q3 revenue") # allowed; result marks the session tainted
guard.wrap("http_get", http_get, session)( # ToolDenied: session read untrusted content
url="https://api.corp.example/x") # and http_get has side effects
What the guard enforces:
- Default deny. Tools and arguments not in the policy are refused; extra arguments are refused.
- Argument constraints. Type, enum, regex (
pattern,deny_patterns), length, numeric bounds. - Paths are resolved (symlinks and
..included) and must sit under an allowed root. - URLs must match scheme and host allowlists, reject embedded credentials, and reject every non-global address (private, loopback, link-local, carrier-grade NAT) including the
169.254.169.254and100.100.100.200metadata services and decimal/hex/octal/IPv4-mapped IP forms, pluslocalhostand*.localhostnames. Other hostnames are resolved and rejected if they point at a non-public range whenever the rule has nohostsallowlist (orresolve_dns: trueis set). (This narrows DNS-rebinding exposure but does not eliminate it; pin the resolved address at connect time for a full fix.) - Taint tracking. Tools marked
returns_untrustedtaint the session; tools markedside_effectsare then denied (or routed to approval) for the rest of that session. This breaks the "private data + untrusted content + outbound channel" combination without needing to detect the injection. - Human approval hook, per-tool and per-session call limits, and an audit trail of every request and decision.
Drop-in helpers for the tool-calling wire formats: run_anthropic_tool_uses and run_openai_tool_calls turn model tool requests into results, with denials returned as error text the model can read. See the complete loop in examples/agent_loop.py.
Async, LangChain, LangGraph, MCP and subprocesses
Each has its own section in docs/SANDBOX.md, with the limits of what was tested.
| Use | Entry point |
|---|---|
asyncio applications (FastAPI, LangGraph) |
aauthorize, aexecute, awrap, arun_anthropic_tool_uses, arun_openai_tool_calls. Same policy, taint, limit and audit behavior as the sync API. examples/async_agent_loop.py |
| LangChain and LangGraph tools | agentsec.integrations.langchain.guard_tools([...], guard, session) |
| MCP servers | agentsec.integrations.mcp.GuardedMCPClient: allowlists and pins tool definitions, and drops poisoned or changed ones |
| External commands | SafeCommandRunner: no shell, executable and option allowlists, scrubbed environment, timeout. It is not an OS sandbox |
| A central gateway in front of local models (preview) | agentsec serve: see docs/GATEWAY.md |
3. Validate output and keep an audit trail
from agentsec.middleware import AgentMiddleware, AuditLogger, FileSink, InjectionScanner, OutputGuard
audit = AuditLogger(FileSink("audit.jsonl"), actor="support-copilot")
mw = AgentMiddleware(
guard=guard,
scanner=InjectionScanner(),
output_guard=OutputGuard.default(allowed_hosts=["corp.example"], protected=[SYSTEM_PROMPT_CANARY]),
audit=audit,
)
screen = mw.screen_input(ticket_text, source="ticket", untrusted=True, session=session)
...
result = mw.screen_output(model_text) # redacts secrets/PII, strips external images, blocks XSS markup
agentsec audit verify audit.jsonl # exit 1 if any record was edited or removed
The audit log chains records with SHA-256. Editing or deleting a record in the middle breaks verification from that point. It does not stop someone with write access from truncating the tail or replacing the whole file; ship records to append-only storage (S3 Object Lock, CloudWatch Logs with a restrictive resource policy) and anchor AuditLogger.checkpoint() somewhere the agent host cannot write. Secrets and PII are redacted before hashing, and log_content=False stores only hashes and lengths of long fields.
SIEM feed (CEF). Keep the hash-chained file as the evidence of record and stream a copy to your SIEM in ArcSight Common Event Format, which Splunk, Microsoft Sentinel, QRadar, Elastic and most syslog pipelines ingest:
from agentsec.middleware import AuditLogger, CefSink, FileSink, TeeSink
audit = AuditLogger(TeeSink(FileSink("audit.jsonl"), CefSink(send_to_syslog))) # or format_cef(record) yourself
Every CEF value is escaped and bounded, so attacker-influenced text cannot forge a field or start a new event. A failing SIEM sink does not interrupt the agent or break the chain. Details are in the production guide. The CEF output has not been ingested by a specific SIEM product.
4. Threat model an agent (STRIDE-for-agents)
agentsec threatmodel init -o system.yaml # describe components, trust zones, data flows
agentsec threatmodel render system.yaml -o THREATMODEL.md
The generator selects candidate threats by component type, draws the data-flow diagram with trust-boundary crossings, gives each row a triage hint (raised for components that ingest untrusted input, have side effects, or hold sensitive data), and links each threat to its mitigations, the toolkit control that implements it, and the fuzzer categories that test it. See the worked example: docs/threat-model/EXAMPLE-support-copilot.md, the method and blank template, and agentsec threatmodel catalog. Each threat is also mapped to the OWASP Top 10 for Agentic Applications (2026) and to MITRE ATLAS techniques (content release 2026.09). agentsec threatmodel crosswalk prints the framework crosswalk, which lists the toolkit controls and fuzzer categories behind each entry.
Coverage against OWASP Top 10 for LLM Applications (2025)
| Risk | Where this toolkit helps |
|---|---|
| LLM01 Prompt injection | Fuzzer (measure), scanner + spotlighting (reduce), taint policy (contain) |
| LLM02 Sensitive information disclosure | Secret/PII redaction, protected-string tripwire, audit redaction |
| LLM03 Supply chain | Threat catalog entries and mitigations (no runtime control here) |
| LLM04 Data and model poisoning | Catalog + retrieval-time scanning |
| LLM05 Improper output handling | OutputGuard validators: markup, exfil links, JSON schema |
| LLM06 Excessive agency | Deny-by-default Policy, approval, taint, SafeCommandRunner |
| LLM07 System prompt leakage | Fuzzer prompt_leak category, ProtectedStringValidator |
| LLM08 Vector and embedding weaknesses | Catalog entries only |
| LLM09 Misinformation | Out of scope |
| LLM10 Unbounded consumption | Per-session call limits, subprocess timeouts and output caps |
The catalog also covers all ten OWASP agentic risks; the crosswalk shows which toolkit controls back each one. A mapping means a threat is an instance of, or contributes to, that entry, not that the entry is mitigated. NIST AI RMF and ISO/IEC 42001 are not mapped.
Honest limitations
- The built-in agents are simulators.
NaiveAgentdeterministically obeys injections and understands common obfuscations; it exists so the harness, detectors and CI gate can be tested offline. The numbers above show how these controls behave when the model is compromised, not how any real model behaves. Run the fuzzer against your real agent for that.AnthropicTargetandHttpTargetare tested against fakes and a local server, and have not been run against a live endpoint by this repo's CI.OllamaTargethas been run against a real local model once, by hand (docs/PROOF.md); CI does not run a model. - Canary-echo success is a proxy. It measures whether the model followed an injected instruction, which is the necessary first step of every real attack, but not the impact of a real one.
- The corpus is a starting set of well-known technique families, not an exhaustive or adaptive attacker. Extend it with your own payloads (
FuzzConfig(corpus=...)). - The scanner is a tripwire. Its documented bypasses are asserted in the tests on purpose.
- Output validators are pattern-based and will not catch data a model re-encodes.
- No OS-level isolation. Use containers/microVMs for code execution, and IMDSv2 with hop limit 1 for cloud-hosted agents.
Development
pip install -e ".[dev]"
ruff check src tests && ruff format --check src tests
mypy src
pytest --cov=agentsec
CI runs lint, type-check and tests on Linux (Python 3.11 to 3.13) and Windows and macOS (3.11, 3.12), builds and checks the package, validates the example policy, re-generates the example threat model to confirm it is current, and runs the fuzzer as a regression check that the containment layers still hold at 0% for tool misuse, exfiltration and prompt leak. See CONTRIBUTING.md and SECURITY.md.
Project health
The OpenSSF Scorecard badge reflects a young, solo-maintained project. Code-Review and Contributors need other people, and Maintained scores 0 for any repository under 90 days old. The checks about the code and supply chain (pinned dependencies, fuzzing, signed releases, token permissions, SAST, vulnerabilities) are the informative ones. codeql-action is left on its major tag deliberately, because GitHub gates some features on the tag.
License
MIT
Metadata
Release files for ai-agent-security-toolkit 0.6.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ai_agent_security_toolkit-0.6.0.tar.gz | 196.4 kB | With provenance | Details for ai_agent_security_toolkit-0.6.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ai_agent_security_toolkit-0.6.0-py3-none-any.whl | Python 3 | none | any | Details for ai_agent_security_toolkit-0.6.0-py3-none-any.whl |
Total release size: 334.0 kB
Release files / ai_agent_security_toolkit-0.6.0.tar.gz
| Download URL | ai_agent_security_toolkit-0.6.0.tar.gz |
|---|---|
| Size | 196.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
87a295527735f8b0c939bebc5f4da8c373fcd2fcada730f3c8337a747a241ed3
|
|
BLAKE2b-256 checksum How to use checksums |
caafebde9d0d135e0f2120d79bd833871a11b4815b15397541d55a9ab75fab03
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / ai_agent_security_toolkit-0.6.0-py3-none-any.whl
| Download URL | ai_agent_security_toolkit-0.6.0-py3-none-any.whl |
|---|---|
| Size | 137.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0cd520fed82f8beca5500dc375042b4973e630ed326979303c1ca1dcb4757f23
|
|
BLAKE2b-256 checksum How to use checksums |
f3313b84ab5a9b538f554025b511bb8b25d8e61ca08b4906cdc6e20ff46088cf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logSecurity for ai-agent-security-toolkit 0.6.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for ai_agent_security_toolkit-0.6.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.