aisec-suite
A CI security gate for the inputs an AI agent trusts but code scanners ignore: MCP tool metadata, agent memory files and RAG documents.
Headline result
I scanned 135 public MCP server repositories (pinned commits, findings hand-labelled) and built a tool-poisoning detector that I evaluated on attacks it was not developed against:
| detector | recall, seen attacks | recall, held-out attacks | false-positive rate (3,974 real tool descriptions) |
|---|---|---|---|
| naive regex | 42.6% | 0.0% | 0.0% |
| aisec structural detector | 61.1% | 48.1% | 0.03% (1 / 3,974) |
Pattern matching memorises its own examples and finds almost nothing new. Scoring decoded text structurally (hidden Unicode, base64/hex/entity decoding, co-occurrence of model-directed, data-flow and concealment cues) generalises much better, but it still misses about half of unseen attacks, so it is a review aid, not a security boundary. The same study raised mcpaudit's precision from 38.8% to 69.4% at 80.6% recall.
Full method, confidence intervals and the caveats (including that the held-out set was evaluated twice) are in docs/STUDY.md.
Try it in two minutes
pip install "aisec-suite[core]"
bash examples/demo-mcp-server/run_demo.sh
The demo scans a small MCP server whose add tool hides an instruction to read ~/.ssh/id_rsa and leak it through a parameter. The gate fails on it, then passes on the fixed version. See examples/demo-mcp-server.
What it scans
| Surface | Scanner | Status |
|---|---|---|
| MCP tool descriptions, schemas, permission scope | mcpaudit + built-in structural detector and reachability (--reach) |
Published (pip install "aisec-suite[core]") |
Agent memory / context files (CLAUDE.md, .cursorrules, ...) |
memsentry |
Published |
| RAG source documents | ragsentry |
Published |
| Fine-tuning datasets, agent traces, package installs, package worms | trainsentry, loopcheck, agent-install-guardrail, wormsentry |
Private, adapters included, not on PyPI |
| Model weights, inference receipts, activation traces, streaming side channels, formal checks of tool state machines | weightspecter, attestinfer, circuitshield, tokenspecter, specprove-sec |
Research preview, private, evaluated only on synthetic fixtures |
Only the published row is installable from PyPI today. Adapters for the other scanners activate when those packages are installed from source; otherwise the CLI reports them as unavailable instead of failing.
Usage
aisec scan . # MCP + memory
aisec scan . --rag ./docs # add RAG documents
aisec scan . --reach annotate # add handler reachability evidence
aisec scan . --sarif aisec.sarif --json findings.json --fail-on high
aisec modules # list scanners and whether each is installed
Exit codes with --fail-on: 0 nothing at or above the threshold, 1 findings at or above it, 2 a scanner crashed (reported as a warning, never mistaken for a finding).
Reusable GitHub Action
Use the bundled composite action to standardize the scanner in another workflow. The example pins the action itself to an immutable commit; update that SHA only when intentionally upgrading the action:
permissions:
contents: read
security-events: write
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: Pyhroff/aisec-suite/.github/actions/aisec-scan@7b355445867cde3724365c6e9a132e4875a78569 # pinned action revision
with:
fail-on: high
sarif: aisec.sarif
json: aisec.json
policy-profile: balanced
- uses: github/codeql-action/upload-sarif@87ef0dc97def48aa960fbf026a2563ee9dbdb470 # v3
with:
sarif_file: aisec.sarif
For maximum reproducibility in production, pin both the aisec-suite action reference and its GitHub Action dependencies to immutable commit SHAs. The bundled composite action validates its dependency-extra input before using it in a shell command.
Release pipeline
Version tags (vX.Y.Z) build and validate both wheel and source distributions. The release workflow verifies that the tag matches the package version and publishes through PyPI trusted publishing (OIDC); no PyPI API token is stored in the repository.
CI security gate
Every push and pull request scans this repository with the published scanners (aisec scan . --all, private adapters skipped) under the checked-in policy. The security workflow gates on the checked-in policy and uploads SARIF results to GitHub code scanning. GitHub Actions dependencies are pinned to immutable commit SHAs and checkout disables persisted credentials.
Package and release status
Latest release: 0.4.0. Earlier development builds were labelled 2.x in the repository but never published; PyPI history runs 0.1.0 -> 0.4.0. Pin versions in CI rather than tracking the default branch.
The release workflow enforces an important invariant: a pushed vX.Y.Z tag must exactly match the version declared in pyproject.toml before publication. PyPI publishing uses trusted publishing (OIDC), so no long-lived PyPI API token is required.
Common reporting model
Integrated scanners are normalized into a shared Finding model so downstream automation can consume results consistently.
Supported outputs include terminal reports, normalized JSON, SARIF 2.1.0, Markdown summaries, severity gates, and reusable baselines. Every normalized finding carries a stable fingerprint plus confidence, target, evidence, and remediation metadata where available.
aisec scan . --write-baseline .aisec-baseline.json
aisec scan . --baseline .aisec-baseline.json --fail-on high
Finding lifecycle
Baselines are fingerprint-based and remain stable across line-number shifts. When a baseline is supplied, the scanner classifies findings as new, existing, or resolved:
aisec scan . --baseline .aisec-baseline.json --json findings.json
# lifecycle: new=2 existing=5 resolved=1
Only new findings remain in the active finding output, so --fail-on evaluates newly introduced risk rather than repeatedly failing on already-accepted findings. Resolved fingerprints are reported for cleanup and audit visibility.
Controlled benchmark
Run the end-to-end regression corpus through the umbrella CLI (the corpus, agent-test-range, is currently private):
aisec benchmark ../agent-test-range --json benchmark/metrics.json
The command delegates execution to the real scanner regression harness and validates its versioned metrics contract. It reports overall and per-scanner precision, recall, F1, false-positive rate, and the underlying confusion-matrix counts.
This benchmark is intentionally a controlled regression corpus, not a universal claim about production detection accuracy.
Detection quality
The suite exposes reusable precision, recall, F1, and false-positive-rate calculations for controlled regression corpora. These metrics are intended for reproducible benchmark cases—not claims of universal real-world detection rates.
Policy as code
Use a versioned JSON policy to define enforcement and narrowly scoped finding exclusions:
aisec scan . --all --policy examples/security-policy.json
Policy version 1 supports:
fail_on:low,medium,high, orcriticalexclude: selectors forscanner,rule,file, andseverity- shell-style glob matching for selectors
Exclusions are explicit and visible in scan warnings; default behavior is unchanged without a policy file. Keep policy files in version control and review exclusions like code.
Policy profiles
For teams that do not need a checked-in policy file, the CLI provides deterministic enforcement profiles:
| Profile | Fails at | Intended use |
|---|---|---|
strict |
low | release/security validation |
balanced |
high | normal CI |
dev |
critical | local development |
aisec scan . --all --policy-profile balanced
aisec policy init examples/security-policy.json --profile balanced
aisec policy show balanced
aisec policy validate examples/security-policy.json
aisec policy init creates a reviewable policy from a deterministic profile and refuses to overwrite an existing file unless --force is supplied.
A checked-in --policy file and --policy-profile are mutually exclusive. A policy file's fail_on value overrides the CLI threshold, while exclusions are always surfaced in warnings.
aisec policy validate rejects unknown top-level fields, unsupported versions, malformed exclusions, and invalid severities so policy drift fails closed.
Security reporting and triage
When GITHUB_STEP_SUMMARY is available, every scan writes a concise security report to the GitHub Actions job summary. The report includes severity counts, baseline lifecycle state, warnings, and a prioritized triage queue.
Priority mapping is deterministic: P0 = critical, P1 = high, P2 = medium, P3 = low. Findings with scanner-provided remediation guidance display it directly in the queue.
aisec scan . --all --summary scan-summary.md
JSON and SARIF remain the machine-readable integrations; the Markdown report is designed for human review and pull-request workflows.
Experimental demos
aisec experimental wargame | guard-proxy | dashboard | remediate are scripted walkthroughs of the research-preview modules. They run on fixed, synthetic inputs and their outcomes are pre-determined. They do not intercept traffic and are not detection results. They live in aisec/experimental/ and need the private packages.
Architecture
files in a repository / PR
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
mcpaudit memsentry ragsentry (+ optional private adapters)
+ structural
detector
+ reachability
│ │ │
└─────────────────┼─────────────────┘
▼
normalized Finding (fingerprint, severity, evidence)
│
policy / baseline lifecycle (new · existing · resolved)
│
terminal · JSON · SARIF 2.1.0 · Markdown job summary · exit code
Design principles: static-first, never execute the scanned code, never follow symlinks out of the repo, fail closed on scanner crashes, and state measured limits next to every number.
Development
pip install -e ".[core,dev]"
ruff check .
pytest -q -rs # -rs lists tests skipped because private scanners are absent
CI runs the unit tests on Python 3.10-3.12 with in-memory fake scanners, plus an integration job that installs the published scanners from PyPI. Tests for the private research modules skip in CI because those packages are not public.
Related projects
mcpaudit, memsentry, ragsentry (published) · adversagen (attack-side research used to test the detectors) · agent-security-ci (CI wiring example). See docs/PORTFOLIO.md.
License
MIT
Metadata
Release files for aisec-suite 0.4.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| aisec_suite-0.4.0.tar.gz | 69.8 kB | With provenance | Details for aisec_suite-0.4.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| aisec_suite-0.4.0-py3-none-any.whl | Python 3 | none | any | Details for aisec_suite-0.4.0-py3-none-any.whl |
Total release size: 130.6 kB
Release files / aisec_suite-0.4.0.tar.gz
| Download URL | aisec_suite-0.4.0.tar.gz |
|---|---|
| Size | 69.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
46940c35e680cae40f52fc904e0e887ee6ab96cbc89031779be65eebbbcce97d
|
|
BLAKE2b-256 checksum How to use checksums |
9159353cc1d7f17162ef9d457fb21439eda2f0a648e98a6017600940217597ef
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.
Transparency logRelease files / aisec_suite-0.4.0-py3-none-any.whl
| Download URL | aisec_suite-0.4.0-py3-none-any.whl |
|---|---|
| Size | 60.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d249e9d7ef6b541780d6c9ea501a41e07e3b24f5d3c9c5788f56a8df70f6ac40
|
|
BLAKE2b-256 checksum How to use checksums |
3517b3e6f76bfaa21483f07f1d58ae8bc06c1d0b14d9e2f232bde18737acb2f1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.
Transparency logSecurity for aisec-suite 0.4.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for aisec_suite-0.4.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.