skillscan 0.4.0

CLI tool that scans SKILL.md agent skill files for security threats

SkillScan

Security scanner for AI agent skills. Audit. Predict. Test.

SkillScan detects malicious patterns in SKILL.md agent skill files before they reach your machine. Three-stage pipeline: static analysis catches known patterns, LLM behavioral prediction catches temporal attacks, and Docker Sandbox execution catches everything else.

Try It Now

pip install skillscan

# Scan an example malicious skill (included in the repo)
git clone https://github.com/NMitchem/SkillScan.git && cd SkillScan
skillscan audit examples/malicious/keychain-stealer.md

Why SkillScan?

The AI agent skills ecosystem is growing fast -- and so are supply-chain attacks targeting it. Static scanners catch obvious patterns, but sophisticated attacks use temporal triggers, obfuscation, and delayed activation that only emerge at runtime.

SkillScan is the only tool that combines all three detection approaches:

Capability	SkillScan	Static Scanners
Pattern matching (shell, URLs, env vars)	Yes	Yes
YARA rules (59 rules, 7 categories)	Yes	Partial
macOS-specific detection (Keychain, AppleScript)	Yes	No
LLM behavioral prediction	Yes	No
Temporal/delayed-activation detection	Yes	No
Docker Sandbox execution with honeypots	Yes	No
SARIF output for GitHub Security tab	Yes	Some

Three-Stage Pipeline

Audit -- Static analysis with 8 built-in analyzers and 59 YAML rules across 7 categories. Catches shell injection, obfuscated payloads, suspicious URLs, credential theft patterns, and macOS-specific threats in milliseconds.

Predict -- LLM behavioral dry-run that simulates what an agent would actually do with the skill at runtime. Reveals temporal triggers, delayed-activation payloads, and multi-step attack chains that static analysis cannot see.

Test -- Sandbox execution inside a Docker microVM with honeypot canary files (fake credentials, SSH keys, API tokens). If the skill reads, copies, or exfiltrates any canary, the test flags it as malicious with critical severity.

Installation

pip install skillscan

Requires Python 3.10+.

Usage

Quick Start

Scan a single skill file:

skillscan audit path/to/SKILL.md

Scan a directory of skills:

skillscan audit ./skills/ --dir

Output as JSON or SARIF:

skillscan audit SKILL.md --format json
skillscan audit SKILL.md --format sarif

Predict Mode

Predict mode performs an LLM behavioral dry-run that simulates what an agent would do with a skill at runtime. Instead of only scanning for static patterns, it asks an LLM to role-play as the agent, revealing risks that emerge from actual tool use -- including temporal triggers, delayed-activation payloads, and multi-step attack chains that static analysis misses.

Run a prediction against a skill file:

skillscan predict path/to/SKILL.md

Choose an LLM provider:

skillscan predict path/to/SKILL.md --provider claude
skillscan predict path/to/SKILL.md --provider openai
skillscan predict path/to/SKILL.md --provider gemini
skillscan predict path/to/SKILL.md --provider ollama

Control how many scenarios are simulated:

skillscan predict path/to/SKILL.md --scenarios 5

Combined Scan

The scan meta-command chains audit, predict, and test in a single pass, giving you static analysis, behavioral prediction, and sandbox execution together:

skillscan scan path/to/SKILL.md

This is the recommended way to get full coverage on a skill file. If Docker Desktop is not available, the test stage is skipped automatically.

Test Mode (Sandbox Execution)

Test mode runs the skill with a real AI agent inside a Docker Sandbox microVM. Unlike predict mode (which simulates behavior via LLM dry-run), test mode actually executes the skill in an isolated sandbox environment and observes what the agent does at runtime -- file writes, network calls, process spawns, and credential access are all recorded and analyzed.

Prerequisites

• Docker Desktop 4.58+ with the Sandbox plugin enabled. Sandbox testing uses Docker's microVM isolation to safely execute untrusted skills. Docker Desktop is required (not available in CI).

Agent Selection

Choose which agent(s) to test with:

skillscan test path/to/SKILL.md --agent claude
skillscan test path/to/SKILL.md --agent openclaw
skillscan test path/to/SKILL.md --agent claude,openclaw

Test Depth

Control how many invocations the agent performs:

# Quick mode (default): 1 invocation
skillscan test path/to/SKILL.md

# Standard mode: 3 invocations
skillscan test path/to/SKILL.md --standard

# Deep mode: 7 invocations
skillscan test path/to/SKILL.md --deep

Extended Observation

Keep the sandbox running longer to catch delayed-activation behavior:

skillscan test path/to/SKILL.md --run-for 10m

Honeypot Canary Detection

During sandbox execution, SkillScan plants honeypot canary files -- fake credentials, SSH keys, and API tokens -- in well-known locations inside the microVM. If the skill (or the agent acting on the skill's instructions) reads, copies, or exfiltrates any of these canary files, the test immediately flags the skill as malicious with a critical-severity finding. This catches data-theft behavior that neither static analysis nor LLM prediction can reliably detect.

Note: Sandbox testing requires Docker Desktop and is not available in CI environments. When Docker is not present, scan gracefully skips the test stage and runs only audit + predict.

CLI Reference

skillscan audit

Audit one or more SKILL.md files for security threats.

skillscan audit <path> [options]

Flag	Description	Default
--dir	Treat path as a directory and scan all skills inside it	false
--format {text,json,sarif}	Output format	text
--threshold <float>	Risk score threshold for pass/fail (0.0-10.0)	6.0
--no-color	Disable ANSI color codes in text output	false
--llm	Enable optional LLM-powered analysis	false
--provider {auto,claude,openai,gemini,ollama}	LLM provider to use	auto

Exit code is 0 if all skills pass, 1 if any skill fails or on error.

skillscan predict

Run an LLM behavioral dry-run against a skill file.

skillscan predict <path> [options]

Flag	Description	Default
--provider {claude,openai,gemini,ollama}	LLM provider to use	claude
--scenarios <int>	Number of behavioral scenarios to simulate	3
--format {text,json}	Output format	text
--no-color	Disable ANSI color codes in text output	false

Predict mode detects temporal and delayed-activation risks that static analysis cannot catch, such as skills that behave differently on the second invocation or after a time delay.

skillscan scan

Run audit, predict, and test in a single pass.

skillscan scan <path> [options]

Accepts the same flags as audit, predict, and test. Exit code is 0 if all stages pass, 1 if any stage fails. The test stage is skipped when Docker Desktop is not available.

skillscan install

Fetch, scan, and install a skill in one step.

skillscan install <source> [options]

Flag	Description	Default
--target <dir>	Override install directory	auto-detect
--threshold <float>	Risk score threshold for pass/fail	6.0
--force	Install even if the scan fails	false
--no-color	Disable ANSI color codes	false

Sources can be a local path, a GitHub URL, or a direct HTTP URL to a SKILL.md file.

Detection Coverage

SkillScan combines 8 built-in analyzers with a YAML rule engine containing 59 rules across 7 categories.

Built-in Analyzers

Analyzer	What it detects
shell	Dangerous shell commands, sudo usage, privilege escalation
macos	macOS-specific threats (LaunchAgents, TCC bypass, Gatekeeper)
obfuscation	Base64-encoded payloads, hex strings, Unicode tricks
url	Suspicious URLs, IP-based endpoints, known malicious domains
envvar	Reads of sensitive environment variables and credentials
entitlement	Overly broad macOS entitlements and permissions
typosquat	Typosquatted package names in npm, pip, and other registries
pattern	General suspicious patterns (eval, exec, dynamic code loading)

YAML Rule Categories

Category	Rules	Description
Prompt Injection	12	Instruction override, role hijacking, delimiter attacks
MCP Security	8	Tool poisoning, schema manipulation, permission escalation
SSRF & Cloud	6	Internal network probing, cloud metadata access
Exfiltration	8	Data theft via DNS, HTTP, webhooks, and side channels
Credentials	10	Hardcoded keys, token extraction, keychain access
Supply Chain	10	Dependency confusion, post-install scripts, registry attacks
Unicode Attacks	5	Bidi overrides, homoglyph substitution, invisible characters

Rules are defined in src/skillscan/data/rules/ as YAML files. Each rule includes a regex pattern, severity level, and descriptive message.

GitHub Action

Add SkillScan to your CI pipeline to block malicious skills before merge.

name: SkillScan
on: [push, pull_request]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: NMitchem/SkillScan@v0.4.0
        with:
          path: '.'
          threshold: '6.0'
          format: 'sarif'
          upload-sarif: 'true'

When upload-sarif is enabled, findings appear directly in the GitHub Security tab.

Contributing

Adding Rules

1. Create or edit a YAML file in src/skillscan/data/rules/.
2. Follow the existing format:

category: your_category
rules:
  - id: YOUR_RULE_ID
    pattern: "regex pattern here"
    severity: high
    message: "Description of what this rule detects"

3. Add tests in tests/ to verify your rules fire on expected input.
4. Run the full test suite: pytest tests/ -v

Running Tests

pip install -e ".[dev]"
pytest tests/ -v

Roadmap

• v0.2 -- Static analysis with YAML rules, SARIF output, GitHub Action
• v0.3 -- Predict: LLM behavioral dry-run and risk prediction
• v0.4 -- Test: Sandbox execution and runtime threat detection

What's next: registry integration, community rule packs, and IDE plugins.

License

MIT -- see LICENSE for details.