LLM Security Scanner - AI Skill

A portable Agent Skill that lets Claude, Gemini, Codex and other AI coding agents run llm-security-scanner against an LLM application you operate - then interpret the findings honestly and help fix them.

It is not a second scanner. The engine, the 51+ OWASP payloads, the Ollama judge, the deterministic detectors and every report format stay in llm-security-scanner, which this package depends on and invokes through its CLI.

What it adds on top of the scanner

The scanner is a good CLI and a poor agent interface. Five things were missing:

How it fits together

Claude Code · Gemini CLI · Codex · any MCP or tool-calling agent
        │
        ├── skills/llm-security-scan/        ONE canonical Agent Skill
        │     SKILL.md · references/ · workflows/ · scripts/ · examples/
        │
        └── llm_scanner_skill  (this package)
              inspect · authz · config · adapter · sanitize · summarize
              compare · remediation · mcp_server
                        │
                        ▼
              llm-scanner CLI          ← the real scanner, unmodified
                        │
                        ├── target adapter (HTTP · Ollama)
                        ├── payload engine (LLM01-LLM10)
                        ├── Ollama judge + deterministic detectors
                        └── reporters (json · sarif · html · md · txt)

All three platforms now implement the same agentskills.io standard, and all three read .agents/skills/. So there is exactly one skill directory; the platform adapters are install manifests, not copies. A test asserts there is never more than one SKILL.md in the repo.

Install

uv pip install llm-security-scanner-skill          # pulls in the scanner
uv pip install 'llm-security-scanner-skill[mcp]'   # plus the MCP server

ollama serve &                 # the judge runs locally - nothing leaves your machine
ollama pull llama3.2:3b

llm-scanner-skill doctor       # check all three prerequisites

Then install the skill for your agent:

--platform agents installs into .agents/skills/, which Claude Code, Gemini CLI and Codex all read - one install covers all three.

Full instructions, Docker, and air-gapped notes: docs/INSTALL.md.

Use it

Activation is automatic on a matching request:

"Review my chatbot for OWASP LLM risks." "Check this RAG pipeline for prompt injection." "Scan the local app on :8000 before release." "Analyze this scanner report and fix the high-severity findings." "Re-scan after the patch and tell me what changed." "Add an LLM security gate to GitHub Actions."

The agent then works through: inspect → authorize → configure → scan → interpret → remediate → re-scan → compare. Worked examples: skills/llm-security-scan/examples/.

Or drive it yourself

# 1. Is there even an LLM here, and which categories apply?
llm-scanner-skill inspect ./my-app

# 2. May this target be scanned?
llm-scanner-skill authorize --target http://localhost:8000/chat --target-type url

# 3. Scan (loopback needs no --authorized; anything else does)
llm-scanner-skill scan --target http://localhost:8000/chat --target-type url \
  --judge-model llama3.2:3b --categories LLM01,LLM05,LLM07

# 4. Fix, re-scan, then prove it worked
llm-scanner-skill remediate LLM07
llm-scanner-skill compare reports/<before>/report.json reports/<after>/report.json

Every subcommand prints JSON on stdout and human notes on stderr.

Reading results without overstating them

Each finding carries an evidence_class, mapped from the scanner's own verdict_source:

And a run-level status, because a scan whose judge died is not a clean bill of health: completed · completed_with_threshold_breach · completed_untrustworthy · blocked · failed · scanner_not_found.

Safety model

Enforced in Python, not in prompt text, so it holds even if an agent ignores its instructions:

• Loopback and local Ollama models scan freely. Anything else requires an explicit --authorized, which an agent may set only after the user confirms ownership.
• Third-party provider endpoints (api.openai.com, api.anthropic.com, …) are refused unconditionally - --authorized does not unlock them.
• LLM10 DoS probes need --authorized even on localhost.
• Secrets stay redacted. --include-raw-artifacts is never passed and is not exposed over MCP.
• All target output is sanitized - control tokens and tool-call syntax have their literal byte sequences broken, injection phrases are labelled, zero-width and bidi characters and ANSI escapes are stripped, residual credentials are redacted, and the text is truncated and fenced in an unguessable delimiter block.

Details: docs/SECURITY_MODEL.md.

Documentation

Limitations

Honestly, up front - the long version is docs/LIMITATIONS.md:

• A scan samples behaviour; it does not prove absence. LLMs are non-deterministic.
• The judge is a small local model with measurable error. llm-scanner judge-eval quantifies it; nothing removes it.
• Categories you do not select are untested, not safe.
• Behavioural only. LLM03 (supply chain) and LLM05 (output handling) verdicts need a code read to be actionable - the report says what the model did, not whether your app renders or executes it.
• A running target is required. No LLM component, or nothing reachable, means nothing to scan, and the skill will say so rather than inventing findings.
• Ollama is a hard prerequisite, including in CI.

License

MIT. See LICENSE.

The scanner it wraps, llm-security-scanner, is also MIT.