Skill Scanner

A best-effort security scanner for AI Agent Skills that detects prompt injection, data exfiltration, and malicious code patterns. It combines pattern-based detection (YAML + YARA-X), AST and dataflow analysis, an optional LLM-as-a-judge, and a bounded CEL decision layer over typed detector facts.

Important: This scanner provides best-effort detection, not comprehensive or complete coverage. A scan that returns no findings does not guarantee that a skill is free of all threats. See Scope and Limitations below.

Supports OpenAI Codex Skills and Cursor Agent Skills formats following the Agent Skills specification. With --lenient, also scans non-standard formats such as Claude Code .claude/commands/*.md and flat markdown skill repos.

Highlights

• Multi-Engine Detection - Static analysis, behavioral dataflow, LLM semantic analysis, and cloud-based scanning for layered, best-effort coverage
• Typed CEL Decisions - The core scanner uses the official cel-go v0.32.0 runtime to correlate bounded facts after deterministic detection and before optional LLM analysis
• Measured Presets - low-noise and quiet presets and LLM caps, with recall, false-positive rate and F1 published for each (Recommended Settings)
• CI/CD Ready - SARIF output for GitHub Code Scanning, reusable GitHub Actions workflow, exit codes for build failures
• Pre-commit Hook - Standard pre-commit framework integration to scan skills before every commit
• Extensible - Plugin architecture for custom analyzers

Join the Cisco AI Discord to discuss, share feedback, or connect with the team.

Scope and Limitations

Skill Scanner is a detection tool. It identifies known and probable risk patterns, but it does not certify security.

Key limitations:

• No findings ≠ no risk. A scan that returns "No findings" indicates that no known threat patterns were detected. It does not guarantee that a skill is secure, benign, or free of vulnerabilities.
• Coverage is inherently incomplete. The scanner combines signature-based detection, LLM-based semantic analysis, behavioral dataflow analysis, optional cloud services, and configurable rule packs. While this approach improves coverage, no automated tool can detect every technique, especially novel or zero-day attacks.
• False positives and false negatives can occur. Presets, the LLM judge and scoped suppressions reduce noise, but no configuration eliminates all incorrect classifications. Pick a measured setup from Recommended Settings and tune the scan policy to your risk tolerance.
• Human review remains essential. Automated scanning is one component of a defense-in-depth strategy. High-risk or production deployments should pair scanner results with manual code review and/or threat modeling.

Measured results

On held-out skills (MaliciousSkillBench's frozen test split, 839 malicious and 545 harmless):

• Rules alone catch 7.7% of malicious skills at HIGH, at a 4.0% false-positive rate.
• With the LLM judge (Gemma 4 26B, balanced), 66.7% reach review at MEDIUM+ (15.4% FPR), and 33.7% are blocked at HIGH (6.4% FPR).

Every figure, with its corpus and method, is in Measured Results and on the evaluation Space.

Documentation

The documentation website is cisco-ai-defense.github.io/docs/skill-scanner. Deep-dive pages live in docs/.

Installation

Prerequisites: CPython 3.11–3.14 and uv (recommended) or pip.

Wheels include the CEL helper for glibc Linux x86-64/ARM64, macOS 14+ x86-64/ARM64 and Windows x86-64. Other platforms build from the source distribution, which needs Go 1.27.1+. See Installation and Configuration for details.

# Using uv (recommended)
uv pip install cisco-ai-skill-scanner

# As a standalone tool
uv tool install cisco-ai-skill-scanner   # or: pipx install cisco-ai-skill-scanner

# Using pip
pip install cisco-ai-skill-scanner

# Using Homebrew (macOS 14+)
brew tap cisco-ai-defense/skill-scanner https://github.com/cisco-ai-defense/skill-scanner
brew install cisco-ai-defense/skill-scanner/skill-scanner

The presets and settings in Recommended Settings need 2.2.0 or newer (skill-scanner --version).

# AWS Bedrock support (IAM credentials, no API key)
pip install cisco-ai-skill-scanner[bedrock]

# Google AI Studio / Gemini support
pip install cisco-ai-skill-scanner[google]

# Google Vertex AI support
pip install cisco-ai-skill-scanner[vertex]

# Azure OpenAI support
pip install cisco-ai-skill-scanner[azure]

# On-device Apple Foundation Model (experimental: macOS 26+, Apple Intelligence, full Xcode)
pip install "apple-fm-sdk>=0.2.1,<0.3"

# All cloud providers
pip install cisco-ai-skill-scanner[all]

Quick Start

Recommended settings

Every recommended setup runs the LLM judge (--use-llm): rules alone catch only about 8% of held-out malicious skills.

Rates are for the MEDIUM+ review queue on MaliciousSkillBench's held-out split, with Gemma 4 26B as the judge. Measured precision for each, and the same setups for pre-commit, GitHub Actions, Python and the REST API, are in Recommended Settings.

Environment Setup

# The LLM judge, used by every recommended setup (local models: see LLM Providers)
export SKILL_SCANNER_LLM_API_KEY="your_api_key"
export SKILL_SCANNER_LLM_MODEL="claude-sonnet-5-5"   # the default

# On-device Apple Foundation Model (experimental, no API key). Behavioral
# alignment verification is skipped with a warning on this model.
# export SKILL_SCANNER_LLM_MODEL="apple-fm/system"
# Optional: disabled, minimal, low, medium, high, xhigh, or max
export SKILL_SCANNER_LLM_REASONING_EFFORT="low"

# For VirusTotal binary scanning
export VIRUSTOTAL_API_KEY="your_virustotal_api_key"

# For Cisco AI Defense
export AI_DEFENSE_API_KEY="your_aidefense_api_key"

Interactive Wizard

Not sure which flags to use? Run skill-scanner with no arguments to launch the interactive wizard:

skill-scanner

The wizard walks you through selecting a scan target, analyzers, policy, and output format, then shows the assembled command before running it. It recommends the LLM judge and leaves the meta-analyzer off. Great for learning the CLI.

CLI Usage

# First test: core analyzers only (static + bytecode + pipeline + correlation)
skill-scanner scan /path/to/skill

# Real use: add the LLM judge
skill-scanner scan /path/to/skill --use-llm --policy balanced --fail-on-severity high

# Scan with behavioral analyzer (dataflow analysis)
skill-scanner scan /path/to/skill --use-behavioral

# Scan with all engines
skill-scanner scan /path/to/skill --use-behavioral --use-llm --use-aidefense

# Rules + LLM judge, with the preset that has the fewest false positives
skill-scanner scan /path/to/skill --use-llm --policy quiet

# Decomposed judge: three focused passes, about three times the tokens
skill-scanner scan /path/to/skill --use-llm --llm-decompose

# Scan with trigger analyzer for vague description checks
skill-scanner scan /path/to/skill --use-trigger

# Run LLM analyzer multiple times and keep majority-agreed findings
skill-scanner scan /path/to/skill --use-llm --llm-consensus-runs 3

# Scan multiple skills recursively
skill-scanner scan-all /path/to/skills --recursive --use-behavioral

# Scan multiple skills with cross-skill overlap detection
skill-scanner scan-all /path/to/skills --recursive --check-overlap

# Scan a GitHub repository (owner/repo shorthand or full URL)
skill-scanner scan-repo owner/repo
skill-scanner scan-repo https://github.com/owner/repo --use-llm

# Lenient mode: tolerate malformed skills instead of failing
skill-scanner scan /path/to/skill --lenient
skill-scanner scan-all /path/to/skills --recursive --lenient

# Lenient mode with non-standard skill formats (no SKILL.md required)
skill-scanner scan .claude/commands/deploy --lenient
skill-scanner scan-all .claude/commands --recursive --lenient

# Use a custom metadata filename instead of SKILL.md
skill-scanner scan /path/to/skill --skill-file README.md

# CI/CD: rules + judge, fail the build on HIGH
skill-scanner scan-all ./skills --recursive --use-llm --policy low-noise --fail-on-severity high --format sarif --output results.sarif

# Generate interactive HTML report with attack correlation groups
skill-scanner scan /path/to/skill --use-llm --format html --output report.html

# Use custom YARA rules
skill-scanner scan /path/to/skill --custom-rules /path/to/my-rules/

# Use custom taxonomy + threat mapping profiles (JSON/YAML)
skill-scanner scan /path/to/skill --taxonomy /path/to/taxonomy.json --threat-mapping /path/to/threat_mapping.json

# VirusTotal hash scan with optional unknown-file uploads
skill-scanner scan /path/to/skill --use-virustotal --vt-upload-files

# Use a scan policy preset (balanced, low-noise, quiet, strict, permissive) with the judge
skill-scanner scan /path/to/skill --use-llm --policy low-noise

# Inspect CEL decisions without suppressing findings
skill-scanner scan /path/to/skill --cel-mode shadow --format json

# Use a custom org policy file
skill-scanner scan /path/to/skill --policy my_org_policy.yaml

# Generate a policy file to customise, starting from a preset
skill-scanner generate-policy --preset low-noise -o my_org_policy.yaml

# Interactive policy configurator (TUI)
skill-scanner configure-policy

Consensus mode keeps a finding only when it appears in more than half of the configured runs. When those votes disagree on severity, the highest observed severity wins, independent of response order. Failed runs and successful runs that omit the finding cast no vote but remain in the denominator. This makes severity selection stable for majority-agreed findings. It does not make an individual LLM sample deterministic, and descriptive fields from equal-severity votes, single-run output, and non-majority findings can still vary between scans.

LLM provider note: --llm-provider accepts anthropic, openai or openai-compatible. For Bedrock, Vertex AI, Azure, Gemini, Ollama, gateways and local servers, set provider-specific model strings and environment variables (see LLM Providers). If --use-llm is set and the judge cannot start, the scan stops with an error rather than passing with rules only.

Python SDK

from skill_scanner import SkillScanner
from skill_scanner.core.analyzers import BehavioralAnalyzer

# Create scanner with analyzers
scanner = SkillScanner(analyzers=[
    BehavioralAnalyzer(),
])

# Scan a skill
result = scanner.scan_skill("/path/to/skill")

print(f"Findings: {len(result.findings)}")
print(f"Max severity: {result.max_severity}")

# Note: is_safe indicates no HIGH/CRITICAL findings were detected.
# It does not guarantee the skill is free of all risk.
if not result.is_safe:
    print("Issues detected -- review findings before deployment")

Security Analyzers

CLI Options

The balanced (default), low-noise, quiet and strict presets use CEL shadow; permissive uses CEL off. Every bundled CEL rule currently has rollout: shadow, so even a global --cel-mode enforce retains findings until an individual rule is qualified and promoted. The ATR pack remains opt-in through --rule-packs atr and is not part of the current core + CEL release gate.

Example Output

$ skill-scanner scan ./my-skill --use-behavioral

============================================================
Skill: my-skill
============================================================
Status: [OK] No findings
Max Severity: NONE
Total Findings: 0
Scan Duration: 0.15s

Note: "No findings" means the scanner did not detect any known threat patterns -- it is not a guarantee that the skill is free of all risk. See Scope and Limitations.

GitHub Actions

Scan skills automatically on every push or PR using the reusable workflow:

# .github/workflows/scan-skills.yml
name: Scan Skills
on:
  pull_request:
    paths: [".cursor/skills/**"]
jobs:
  scan:
    uses: cisco-ai-defense/skill-scanner/.github/workflows/scan-skills.yml@2.3.0
    with:
      scanner_version: "2.3.0"
      skill_path: .cursor/skills
      policy: low-noise
      use_llm: true
      llm_model: anthropic/claude-sonnet-5-5
    secrets:
      llm_api_key: ${{ secrets.SKILL_SCANNER_LLM_API_KEY }}
    permissions:
      security-events: write
      contents: read
      actions: read

Results appear as inline annotations in PRs via GitHub Code Scanning. See the full guide for LLM integration, secret configuration, and branch protection setup.

Pre-commit Hook

Scan skills, with the judge, before every commit using the pre-commit framework:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/cisco-ai-defense/skill-scanner
    rev: 2.3.0  # the latest release tag (no "v" prefix)
    hooks:
      - id: skill-scanner

Turn the judge on in .skill_scannerrc at the repository root (use_llm is off by default):

{
  "skills_path": ".claude/skills",
  "policy": "low-noise",
  "use_llm": true,
  "llm_model": "anthropic/claude-sonnet-5-5",
  "severity_threshold": "high",
  "fail_fast": true
}

The hook scans only the skills a commit touches. The key comes from SKILL_SCANNER_LLM_API_KEY, or from cloud credentials for Bedrock and Vertex AI. llm_model and llm_provider fall back to SKILL_SCANNER_LLM_MODEL and SKILL_SCANNER_LLM_PROVIDER, so the hook can point at a local model. If the judge cannot be built, the commit is blocked with exit code 2 instead of passing on the rules alone. For a bedrock/ model, add additional_dependencies: [boto3] to the hook. Run pre-commit install once, or skill-scanner-pre-commit --install without the pre-commit framework.

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

License

Apache 2.0 - See LICENSE for details.

Copyright 2026 Cisco Systems, Inc. and its affiliates

GitHub • Discord • PyPI