llm-pentest 0.1.1

Security testing toolkit for LLM-based systems

LLM Pentest

Security testing toolkit for LLM-based systems — chatbots, RAG pipelines, and AI agents.

Tests for prompt injection, data exfiltration, tool abuse, and jailbreaking.

Install

# One-liner (macOS/Linux — auto-detects OS and arch)
curl -fsSL https://raw.githubusercontent.com/jcazako/llm-pentest/main/install.sh | bash

# Homebrew (macOS/Linux)
brew tap jcazako/tap && brew install llm-pentest

# pip / pipx
pip install llm-pentest
pipx install llm-pentest

# From source
git clone https://github.com/jcazako/llm-pentest && cd llm-pentest
pip install -e .

Quick Start

# Interactive setup
llm-pentest init

# Or use a built-in profile
llm-pentest scan --profile openai

# Or provide a config file
cp examples/target.yaml my-target.yaml
llm-pentest scan -c my-target.yaml

Usage

# List payloads and chains
llm-pentest payloads

# Scan specific category
llm-pentest scan -c target.yaml --category prompt_injection

# Scan single payload
llm-pentest scan -c target.yaml --payload pi-001

# Parallel scanning (5 concurrent requests)
llm-pentest scan -c target.yaml -n 5

# Only fail CI on high/critical findings
llm-pentest scan -c target.yaml --fail-on high

# Use a profile instead of config file
llm-pentest scan --profile openai --model gpt-4o

# HTML report
llm-pentest scan -c target.yaml --format html -o report.html

# SARIF report (GitHub Code Scanning compatible)
llm-pentest scan -c target.yaml --format sarif -o report.sarif

# Baseline comparison
llm-pentest scan -c target.yaml -o baseline.json
# ... fix vulnerabilities ...
llm-pentest scan -c target.yaml -o current.json --baseline baseline.json

# Standalone diff
llm-pentest diff -b baseline.json --current current.json

# Custom detection hooks
llm-pentest scan -c target.yaml --hooks my_hooks.py

# LLM-as-judge for better detection accuracy
llm-pentest scan -c target.yaml --judge-endpoint https://api.openai.com/v1/chat/completions --judge-key $OPENAI_API_KEY

# Dry run (no requests sent)
llm-pentest scan -c target.yaml --dry-run

Target Profiles

Built-in profiles for common providers (llm-pentest profiles):

Profile	Endpoint	Default Model
openai	api.openai.com	gpt-4o-mini
anthropic	api.anthropic.com	claude-sonnet-4-20250514
ollama	localhost:11434	llama3

Attack Categories

Category	Payloads	OWASP	Description
prompt_injection	pi-001 to pi-009	LLM01	System prompt extraction, instruction override, encoding bypass, indirect injection
data_exfiltration	de-001 to de-005	LLM06	RAG context leak, credential extraction, PII harvesting
tool_abuse	ta-001 to ta-004	LLM07, LLM08	Unauthorized tool use, parameter injection, CoT hijacking, DoS
jailbreak	jb-001 to jb-004	LLM01	Role-play, hypothetical framing, token smuggling, persona splitting

Attack Chains

Chains compose multiple payloads into multi-step attack sequences:

- id: chain-001
  name: "Jailbreak then Extract Credentials"
  steps:
    - payload_id: jb-001
      prompt_index: 0
    - payload_id: de-002
      prompt_index: 0

Custom Payloads

Drop YAML files into payloads/ or pass --payloads-dir:

- id: custom-001
  name: "My Custom Attack"
  category: prompt_injection
  severity: high
  owasp: ["LLM01"]
  prompts:
    - "Your attack prompt here"
  detection:
    - pattern: "expected response pattern"
      confidence: 0.8

Custom Detection Hooks

Write Python functions for domain-specific detection:

# my_hooks.py — any function named detect_* is auto-registered
def detect_internal_data(payload, prompt, response):
    if "internal-project-name" in response.lower():
        return "Leaked internal project name"
    return None

llm-pentest scan -c target.yaml --hooks my_hooks.py

CI/CD Integration

# Fail only on high/critical findings
llm-pentest scan -c target.yaml --fail-on high || echo "Vulnerabilities found!"

# SARIF for GitHub Code Scanning
llm-pentest scan -c target.yaml --format sarif -o results.sarif

# Baseline regression check
llm-pentest scan -c target.yaml --baseline previous.json --fail-on high

License

MIT