aisec-suite
A CI security gate for the inputs an AI agent trusts but code scanners ignore: MCP tool metadata, agent memory files and RAG documents.
Headline result
I scanned 135 public MCP server repositories (pinned commits, findings hand-labelled) and built a tool-poisoning detector that I evaluated on attacks it was not developed against:
| detector | recall, seen attacks | recall, held-out attacks | false-positive rate (3,974 real tool descriptions) |
|---|---|---|---|
| naive regex | 42.6% | 0.0% | 0.0% |
| aisec structural detector | 74.1% | 53.7% | 0.05% (2 / 3,974) |
On attacks written by other people it catches 14.0% of MCPTox (485 poisoned tools; naive regex 6.8%; no false positives on the servers' 362 real tools) and 14.0% of MCP-Poison-Bench (86 metadata payloads; naive regex 10.5%). It catches none of MCP-Poison-Bench's 74 held-out payloads, which are phrased as ordinary documentation, and neither does any baseline. Both evaluations were pre-registered and run once; see docs/STUDY.md section 3c. A learned text classifier trained to close that gap did not: 0.8% on MCPTox at a 2.9% false-positive rate (section 3d), so it is not shipped. The memory-file and RAG scanners do worse on external attacks (BIPIA, InjecAgent): they catch the stock override phrase that opens many published injections and 0-3% of attacks written without it, no better than a naive regex (section 3e). A provenance-based runtime guard (aisec.runtime_guard), which holds calls that send data to destinations the user never named, stops 88.7% of AgentDojo attacks that have ground-truth calls (56.9% of all pairs as pre-registered) and asks for confirmation on 22.7% of benign tasks (section 3f). Three later versions are reported in sections 3g-3i, with their failures: v3 stops 93.2% of InjecAgent's 1,054 attacks but only 14.0% of the 2,040 attack pairs in ASB, because it reads effects from tool names and ASB's attacker tools are named without verbs; v4 also reads tool descriptions and lifts ASB to 48.1%, but holds 39.2% of ASB's benign pairs and failed its own pre-registered ship rule, so it is a library, not a CLI option. None of this has been tested against a live agent, and the later versions were evaluated on data I had already seen.
Pattern matching memorises its own examples and finds almost nothing new. Scoring decoded text structurally (hidden Unicode, look-alike letters, base64/base32/hex/escape decoding, co-occurrence of model-directed, data-flow and concealment cues) generalises much better, but it still misses almost half of unseen attacks, so it is a review aid, not a security boundary.
It also holds up against cheap obfuscation: of the seen attacks it catches, at least 87.5% are still caught after look-alike letters, hard-wrapping, letter-spacing, sentence fragmentation, splitting across fields or base32/escape encoding (the previous version kept 14.7% under look-alike letters). The same study raised mcpaudit's precision from 38.8% to 69.4% at 80.6% recall.
Reachability (--reach) now follows commands across IPC bridges into engine-side scripts: on labelled findings it never wrongly cleared a real finding, and removed 4 of 30 unseen false positives (single-language reach removed none).
Every number on this page except the guard v2-v4 results is re-derived from public data weekly by the benchmarks workflow, which fails if any of them moves. The guard v2-v4 numbers (sections 3g-3i of the study) were each run once under a pre-registered protocol and can be reproduced with benchmarks/guard_v2_eval.py, guard_v3_eval.py and guard_v4_eval.py against the public datasets; they are not in the weekly gate.
Across 135 real MCP servers I found no tool poisoning in the wild. Unconstrained-scope findings were by-design capabilities, not vulnerabilities, which is why the gate is a review aid rather than a vulnerability scanner.
Full method, confidence intervals and the caveats (including that the held-out set has now been evaluated three times) are in docs/STUDY.md.
Try it in two minutes
pip install "aisec-suite[core]"
bash examples/demo-mcp-server/run_demo.sh
The demo scans a small MCP server whose add tool hides an instruction to read ~/.ssh/id_rsa and leak it through a parameter. The gate fails on it, then passes on the fixed version. See examples/demo-mcp-server.
What it scans
| Surface | Scanner | Status |
|---|---|---|
| MCP tool descriptions, schemas, permission scope | mcpaudit + built-in structural detector and reachability (--reach) |
Published (pip install "aisec-suite[core]") |
Agent memory / context files (CLAUDE.md, .cursorrules, ...) |
memsentry |
Published |
| RAG source documents | ragsentry |
Published |
| Fine-tuning datasets, agent traces | trainsentry, loopcheck |
Public source, not on PyPI; adapters included |
| Package installs, package worms | agent-install-guardrail, wormsentry |
Private, adapters included |
| Model weights, inference receipts, activation traces, streaming side channels, formal checks of tool state machines | weightspecter, attestinfer, circuitshield, tokenspecter, specprove-sec |
Research preview, private, evaluated only on synthetic fixtures |
Only the Published rows are installable from PyPI today. Adapters for the other scanners activate when those packages are installed from source; otherwise the CLI reports them as unavailable instead of failing.
Usage
aisec scan . # MCP + memory
aisec scan . --rag ./docs # add RAG documents
aisec scan . --reach annotate # add handler reachability evidence
aisec scan . --sarif aisec.sarif --json findings.json --fail-on high
aisec modules # list scanners and whether each is installed
Exit codes with --fail-on: 0 nothing at or above the threshold, 1 findings at or above it, 2 a scanner crashed (reported as a warning, never mistaken for a finding).
Red-teaming an LLM target
aisec redteam attacks a chat model running under a system prompt with adaptive multi-turn strategies (crescendo
escalation, steganographic carriers, roleplay), scores every reply 1-10 with a deterministic or LLM judge, and writes a
JSON report with the attack path, payloads, turn history and whether the prompt was extracted or bypassed.
aisec redteam --json redteam.json # offline run against the built-in mock target
aisec redteam --target http --url http://localhost:11434/v1/chat/completions --model llama3.2 \
--system-prompt-file prompt.txt --canary "ZX-CANARY-4411" --fail-on-breach # a local or hosted model, or your own app
aisec redteam --target claude --system-prompt-file prompt.txt --canary "ZX-CANARY-4411" --judge claude
The mock target is scripted, so the first command checks the engine, not any model. It has not been evaluated against
real models yet. Details and limits: docs/REDTEAM.md.
Runtime guard for MCP servers
Scanners catch careless attacks; the runtime guard holds the dangerous step whatever the wording. Put it in front of any MCP server: once the agent has read a tool result, calls that send data to a destination the user never named, or that delete or change things the user never asked to change, are not forwarded and the agent is told why.
aisec guard-mcp --request "Summarise my unread email" -- npx my-mcp-server
Runnable demo with the official MCP client: examples/guarded-mcp. Measured on AgentDojo in
docs/STUDY.md section 3f.
--guard v3 (also on aisec guard-check) switches to the newer engine: it sorts tools into effect classes (delete, pay,
share, run, ...) by the verbs in their names and treats any email, URL, phone number or IBAN in any argument as a
destination, with no policy to write. v1 stays the default. Measured, with the misses: 93.2% of InjecAgent's 1,054
attacks stopped (v1 54.8%), 14.0% of ASB's 2,040 (v1 1.5%), and on AgentDojo 24 of 97 benign tasks held for confirmation
(v1 22). It reads names, so it misses attacks whose tool names contain no verb it knows (docs/STUDY.md sections 3g and
3h). A policy file for v3 may only list trusted destinations.
Reusable GitHub Action
Use the bundled composite action to standardize the scanner in another workflow. The example pins the action itself to an immutable commit; update that SHA only when intentionally upgrading the action:
permissions:
contents: read
security-events: write
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: Pyhroff/aisec-suite/.github/actions/aisec-scan@7b355445867cde3724365c6e9a132e4875a78569 # pinned action revision
with:
fail-on: high
sarif: aisec.sarif
json: aisec.json
policy-profile: balanced
- uses: github/codeql-action/upload-sarif@87ef0dc97def48aa960fbf026a2563ee9dbdb470 # v3
with:
sarif_file: aisec.sarif
For maximum reproducibility in production, pin both the aisec-suite action reference and its GitHub Action dependencies to immutable commit SHAs. The bundled composite action validates its dependency-extra input before using it in a shell command.
Research-preview demos
python -m research_preview wargame | guard-proxy | dashboard | remediate are scripted walkthroughs of the private research modules. They run on fixed, synthetic inputs and their outcomes are pre-determined. They do not intercept traffic and are not detection results. The code lives in research_preview/, which is not part of the published wheel, and needs the private packages plus numpy.
Package and release status
Latest release: 0.7.0 (PyPI: 0.2.0, 0.3.0, 0.4.0, 0.5.0, 0.6.0, 0.7.0; earlier 2.x labels in the repository history were never published). Pin versions in CI rather than tracking the default branch. Policy as code, baselines, finding lifecycle, reporting formats and the release pipeline are documented in docs/REFERENCE.md.
Architecture
files in a repository / PR
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
mcpaudit memsentry ragsentry (+ optional private adapters)
+ structural
detector
+ reachability
│ │ │
└─────────────────┼─────────────────┘
▼
normalized Finding (fingerprint, severity, evidence)
│
policy / baseline lifecycle (new · existing · resolved)
│
terminal · JSON · SARIF 2.1.0 · Markdown job summary · exit code
flowchart LR
A["AI agent input<br/>MCP metadata · RAG docs · memory files"] --> B["AST & heuristic rules<br/>mcpaudit · memsentry · ragsentry"]
B --> C["Provenance guard"]
C --> D["Finding + policy/baseline"]
D --> E["SARIF 2.1.0 CI output<br/>exit code"]
Design principles: static-first, never execute the scanned code, never follow symlinks out of the repo, fail closed on scanner crashes, and state measured limits next to every number.
Development
pip install -c constraints.txt -e ".[core,dev]"
ruff check . && mypy
pytest -q -rs --cov # -rs lists tests skipped because private scanners are absent
CI runs the unit tests on Python 3.10-3.13 with in-memory fake scanners, plus an integration job that installs the published scanners (versions in constraints.txt). Detector quality is guarded by tests/test_poison_quality.py. See CONTRIBUTING.md.
Limitations
- Static analysis of metadata and source: runtime behaviour, rug-pulls after review and attacks outside the implemented feature families are out of scope.
- The poisoning detector misses almost half of held-out attacks; positives are synthetic and single-author (see
docs/STUDY.mdsection 3). - Reachability is conservative: most findings stay
unknown, and bridge resolution covers GDScript (Godot) and C# (Unity editor plugins) dispatchers only. A Unity server whose C# plugin is not in the same repository staysunknown. - A clean scan is not a security guarantee. Treat findings as review input, not verdicts.
Upstream security work
- BerriAI/litellm:
- PrefectHQ/fastmcp:
- PR #6063 (open): wall-clock timeout and external callback limit in MontySandboxProvider and CodeMode
Related projects
mcpaudit, memsentry, ragsentry (published) · adversagen (attack-side research used to test the detectors) · agent-security-ci (CI wiring example). See docs/PORTFOLIO.md.
License
MIT
Metadata
Release files for aisec-suite 0.7.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| aisec_suite-0.7.0.tar.gz | 125.4 kB | With provenance | Details for aisec_suite-0.7.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| aisec_suite-0.7.0-py3-none-any.whl | Python 3 | none | any | Details for aisec_suite-0.7.0-py3-none-any.whl |
Total release size: 221.3 kB
Release files / aisec_suite-0.7.0.tar.gz
| Download URL | aisec_suite-0.7.0.tar.gz |
|---|---|
| Size | 125.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
afc4162663dbe350a7aeafff7c9e5f4f708609c5e10f2311ccae1e9d00de780f
|
|
BLAKE2b-256 checksum How to use checksums |
126f950c4b724961545fb27a5718e7fa7b1772870a0536f7e49df436c51e4c31
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logRelease files / aisec_suite-0.7.0-py3-none-any.whl
| Download URL | aisec_suite-0.7.0-py3-none-any.whl |
|---|---|
| Size | 95.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f0382c36ff3186127c81fba7a057474430d03e0cbcf6a16d018db7564ec49ea9
|
|
BLAKE2b-256 checksum How to use checksums |
52c7bd95bd27242a8182ae1a5224de8a851ad99028214b5df19760d1082ce775
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logSecurity for aisec-suite 0.7.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for aisec_suite-0.7.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.