Skip to main content

aisec-suite

CI Python License

A CI security gate for the inputs an AI agent trusts but code scanners ignore: MCP tool metadata, agent memory files and RAG documents.

Headline result

I scanned 135 public MCP server repositories (pinned commits, findings hand-labelled) and built a tool-poisoning detector that I evaluated on attacks it was not developed against:

detector recall, seen attacks recall, held-out attacks false-positive rate (3,974 real tool descriptions)
naive regex 42.6% 0.0% 0.0%
aisec structural detector 61.1% 48.1% 0.03% (1 / 3,974)

Pattern matching memorises its own examples and finds almost nothing new. Scoring decoded text structurally (hidden Unicode, base64/hex/entity decoding, co-occurrence of model-directed, data-flow and concealment cues) generalises much better, but it still misses about half of unseen attacks, so it is a review aid, not a security boundary. The same study raised mcpaudit's precision from 38.8% to 69.4% at 80.6% recall.

Full method, confidence intervals and the caveats (including that the held-out set was evaluated twice) are in docs/STUDY.md.

Try it in two minutes

pip install "aisec-suite[core]"
bash examples/demo-mcp-server/run_demo.sh

The demo scans a small MCP server whose add tool hides an instruction to read ~/.ssh/id_rsa and leak it through a parameter. The gate fails on it, then passes on the fixed version. See examples/demo-mcp-server.

What it scans

Surface Scanner Status
MCP tool descriptions, schemas, permission scope mcpaudit + built-in structural detector and reachability (--reach) Published (pip install "aisec-suite[core]")
Agent memory / context files (CLAUDE.md, .cursorrules, ...) memsentry Published
RAG source documents ragsentry Published
Fine-tuning datasets, agent traces, package installs, package worms trainsentry, loopcheck, agent-install-guardrail, wormsentry Private, adapters included, not on PyPI
Model weights, inference receipts, activation traces, streaming side channels, formal checks of tool state machines weightspecter, attestinfer, circuitshield, tokenspecter, specprove-sec Research preview, private, evaluated only on synthetic fixtures

Only the published row is installable from PyPI today. Adapters for the other scanners activate when those packages are installed from source; otherwise the CLI reports them as unavailable instead of failing.

Usage

aisec scan .                                    # MCP + memory
aisec scan . --rag ./docs                       # add RAG documents
aisec scan . --reach annotate                   # add handler reachability evidence
aisec scan . --sarif aisec.sarif --json findings.json --fail-on high
aisec modules                                   # list scanners and whether each is installed

Exit codes with --fail-on: 0 nothing at or above the threshold, 1 findings at or above it, 2 a scanner crashed (reported as a warning, never mistaken for a finding).

Reusable GitHub Action

Use the bundled composite action to standardize the scanner in another workflow. The example pins the action itself to an immutable commit; update that SHA only when intentionally upgrading the action:

permissions:
  contents: read
  security-events: write

steps:
  - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
    with:
      persist-credentials: false
  - uses: Pyhroff/aisec-suite/.github/actions/aisec-scan@7b355445867cde3724365c6e9a132e4875a78569 # pinned action revision
    with:
      fail-on: high
      sarif: aisec.sarif
      json: aisec.json
      policy-profile: balanced
  - uses: github/codeql-action/upload-sarif@87ef0dc97def48aa960fbf026a2563ee9dbdb470 # v3
    with:
      sarif_file: aisec.sarif

For maximum reproducibility in production, pin both the aisec-suite action reference and its GitHub Action dependencies to immutable commit SHAs. The bundled composite action validates its dependency-extra input before using it in a shell command.

Release pipeline

Version tags (vX.Y.Z) build and validate both wheel and source distributions. The release workflow verifies that the tag matches the package version and publishes through PyPI trusted publishing (OIDC); no PyPI API token is stored in the repository.

CI security gate

Every push and pull request scans this repository with the published scanners (aisec scan . --all, private adapters skipped) under the checked-in policy. The security workflow gates on the checked-in policy and uploads SARIF results to GitHub code scanning. GitHub Actions dependencies are pinned to immutable commit SHAs and checkout disables persisted credentials.

Package and release status

Latest release: 0.4.0. Earlier development builds were labelled 2.x in the repository but never published; PyPI history runs 0.1.0 -> 0.4.0. Pin versions in CI rather than tracking the default branch.

The release workflow enforces an important invariant: a pushed vX.Y.Z tag must exactly match the version declared in pyproject.toml before publication. PyPI publishing uses trusted publishing (OIDC), so no long-lived PyPI API token is required.

Common reporting model

Integrated scanners are normalized into a shared Finding model so downstream automation can consume results consistently.

Supported outputs include terminal reports, normalized JSON, SARIF 2.1.0, Markdown summaries, severity gates, and reusable baselines. Every normalized finding carries a stable fingerprint plus confidence, target, evidence, and remediation metadata where available.

aisec scan . --write-baseline .aisec-baseline.json
aisec scan . --baseline .aisec-baseline.json --fail-on high

Finding lifecycle

Baselines are fingerprint-based and remain stable across line-number shifts. When a baseline is supplied, the scanner classifies findings as new, existing, or resolved:

aisec scan . --baseline .aisec-baseline.json --json findings.json
# lifecycle: new=2 existing=5 resolved=1

Only new findings remain in the active finding output, so --fail-on evaluates newly introduced risk rather than repeatedly failing on already-accepted findings. Resolved fingerprints are reported for cleanup and audit visibility.

Controlled benchmark

Run the end-to-end regression corpus through the umbrella CLI (the corpus, agent-test-range, is currently private):

aisec benchmark ../agent-test-range --json benchmark/metrics.json

The command delegates execution to the real scanner regression harness and validates its versioned metrics contract. It reports overall and per-scanner precision, recall, F1, false-positive rate, and the underlying confusion-matrix counts.

This benchmark is intentionally a controlled regression corpus, not a universal claim about production detection accuracy.

Detection quality

The suite exposes reusable precision, recall, F1, and false-positive-rate calculations for controlled regression corpora. These metrics are intended for reproducible benchmark cases—not claims of universal real-world detection rates.

Policy as code

Use a versioned JSON policy to define enforcement and narrowly scoped finding exclusions:

aisec scan . --all --policy examples/security-policy.json

Policy version 1 supports:

  • fail_on: low, medium, high, or critical
  • exclude: selectors for scanner, rule, file, and severity
  • shell-style glob matching for selectors

Exclusions are explicit and visible in scan warnings; default behavior is unchanged without a policy file. Keep policy files in version control and review exclusions like code.

Policy profiles

For teams that do not need a checked-in policy file, the CLI provides deterministic enforcement profiles:

Profile Fails at Intended use
strict low release/security validation
balanced high normal CI
dev critical local development
aisec scan . --all --policy-profile balanced
aisec policy init examples/security-policy.json --profile balanced
aisec policy show balanced
aisec policy validate examples/security-policy.json

aisec policy init creates a reviewable policy from a deterministic profile and refuses to overwrite an existing file unless --force is supplied.

A checked-in --policy file and --policy-profile are mutually exclusive. A policy file's fail_on value overrides the CLI threshold, while exclusions are always surfaced in warnings.

aisec policy validate rejects unknown top-level fields, unsupported versions, malformed exclusions, and invalid severities so policy drift fails closed.

Security reporting and triage

When GITHUB_STEP_SUMMARY is available, every scan writes a concise security report to the GitHub Actions job summary. The report includes severity counts, baseline lifecycle state, warnings, and a prioritized triage queue.

Priority mapping is deterministic: P0 = critical, P1 = high, P2 = medium, P3 = low. Findings with scanner-provided remediation guidance display it directly in the queue.

aisec scan . --all --summary scan-summary.md

JSON and SARIF remain the machine-readable integrations; the Markdown report is designed for human review and pull-request workflows.

Experimental demos

aisec experimental wargame | guard-proxy | dashboard | remediate are scripted walkthroughs of the research-preview modules. They run on fixed, synthetic inputs and their outcomes are pre-determined. They do not intercept traffic and are not detection results. They live in aisec/experimental/ and need the private packages.

Architecture

            files in a repository / PR
                        │
      ┌─────────────────┼─────────────────┐
      ▼                 ▼                 ▼
   mcpaudit         memsentry         ragsentry        (+ optional private adapters)
  + structural
    detector
  + reachability
      │                 │                 │
      └─────────────────┼─────────────────┘
                        ▼
          normalized Finding (fingerprint, severity, evidence)
                        │
        policy / baseline lifecycle (new · existing · resolved)
                        │
        terminal · JSON · SARIF 2.1.0 · Markdown job summary · exit code

Design principles: static-first, never execute the scanned code, never follow symlinks out of the repo, fail closed on scanner crashes, and state measured limits next to every number.

Development

pip install -e ".[core,dev]"
ruff check .
pytest -q -rs     # -rs lists tests skipped because private scanners are absent

CI runs the unit tests on Python 3.10-3.12 with in-memory fake scanners, plus an integration job that installs the published scanners from PyPI. Tests for the private research modules skip in CI because those packages are not public.

mcpaudit, memsentry, ragsentry (published) · adversagen (attack-side research used to test the detectors) · agent-security-ci (CI wiring example). See docs/PORTFOLIO.md.

License

MIT

Metadata

Release files for aisec-suite 0.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aisec-suite 0.4.0
File Size Uploaded Provenance Actions
aisec_suite-0.4.0.tar.gz 69.8 kB With provenance Details for aisec_suite-0.4.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for aisec-suite 0.4.0
File Interpreter ABI Platform Actions
aisec_suite-0.4.0-py3-none-any.whl Python 3 none any Details for aisec_suite-0.4.0-py3-none-any.whl

Total release size: 130.6 kB

Release files / aisec_suite-0.4.0.tar.gz

Download URL aisec_suite-0.4.0.tar.gz
Size 69.8 kB
Tags Source
SHA-256 checksum
How to use checksums
46940c35e680cae40f52fc904e0e887ee6ab96cbc89031779be65eebbbcce97d
BLAKE2b-256 checksum
How to use checksums
9159353cc1d7f17162ef9d457fb21439eda2f0a648e98a6017600940217597ef
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

Release files / aisec_suite-0.4.0-py3-none-any.whl

Download URL aisec_suite-0.4.0-py3-none-any.whl
Size 60.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
d249e9d7ef6b541780d6c9ea501a41e07e3b24f5d3c9c5788f56a8df70f6ac40
BLAKE2b-256 checksum
How to use checksums
3517b3e6f76bfaa21483f07f1d58ae8bc06c1d0b14d9e2f232bde18737acb2f1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

Security for aisec-suite 0.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for aisec_suite-0.4.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page