securitybench 0.1.7

Security Bench - Security testing for LLM pipelines

Security Bench

Website: securitybench.ai

Security testing framework for AI/LLM pipelines - Test your AI systems for prompt injection, jailbreaks, data leakage, and other security vulnerabilities.

Overview

Security Bench is a comprehensive security testing tool designed for real-world AI deployments. Unlike benchmarks that only test base models, Security Bench tests your entire pipeline - including RAG systems, tool-calling agents, and multi-component architectures.

Key Features

• Two Testing Modes - LLM endpoint scanning AND local code/config auditing
• Pipeline-First Testing - Test deployed systems, not just models
• Lynis-Style Auditing - Rich terminal output with grades, findings, and remediation
• Privacy-Preserving - 100% local execution, no data leaves your environment
• Comprehensive Coverage - 500+ LLM tests, 266 local checks across 32 categories
• Simple to Advanced - Zero-config for quick scans, detailed config for complex systems

Quick Start

# Install
pip install securitybench

# Scan an LLM endpoint for vulnerabilities
sb scan https://api.example.com/chat

# Balanced scan (recommended for benchmarking)
sb scan https://api.example.com/chat --balanced

# Audit local project for security issues
sb audit

# Code-only analysis
sb code ./src

LLM Endpoint Testing

# Quick security check
sb scan https://api.example.com/chat --header "Authorization: Bearer sk-..." --limit 20

# Balanced scan across all 31 attack categories
sb scan https://api.example.com/chat --balanced

# Full scan with configuration file
sb scan --config securitybench.yaml

Local Security Auditing (Lynis-style)

# Full audit (code + config + infrastructure)
sb audit

# Code analysis only (prompt injection patterns, secrets, etc.)
sb code

# Configuration checks only (exposed keys, insecure settings)
sb config

# Infrastructure checks only (Docker, K8s, permissions)
sb infra

# Get remediation guidance for a specific finding
sb fix CODE-001

Test Modes

Security Bench provides flexible test modes to balance speed vs. coverage:

Mode	Tests	Use Case
--limit 20	20 random	Quick smoke test
--limit 50	50 random	Default, general testing
--balanced	155 (5 × 31 categories)	Recommended for benchmarking
--balanced --per-category 10	310	Thorough coverage
--categories SPE,PIN	Filtered	Focus on specific attacks

Balanced Mode (Recommended)

The --balanced flag ensures even sampling across all 31 attack categories:

sb scan https://api.example.com/chat --balanced --delay 2

• Default: 5 tests per category = 155 tests total
• Adjustable: --per-category 10 for 310 tests
• Stable: Always includes all categories, even as the test database grows
• Comparable: Results are comparable across models on the leaderboard

Rate Limiting

Use --delay to avoid rate limits when testing external APIs:

sb scan https://openrouter.ai/api/v1 -m gpt-4 --balanced --delay 2

This adds a 2-second pause between API calls (155 tests × 2s ≈ 5 minutes).

Test Categories (32 Total)

Security Bench provides comprehensive coverage across 32 attack categories, expandable as new techniques are discovered.

Core Categories (10 with tests, 22 planned)

Injection & Manipulation:

• SPE - System Prompt Extraction (14,138 tests)
• PIN - Prompt Injection Direct (64 tests)
• IND - Indirect Injection (114 tests)
• JBR - Jailbreak (123 tests)
• OBF - Obfuscation (84 tests)
• MTM - Multi-Turn Manipulation (19 tests)
• GHJ - Goal Hijacking
• CTX - Context Manipulation

Information Leakage:

• ILK - Information Leakage (3,298 tests)
• SEC - Secret Extraction (115 tests)
• EXF - Data Exfiltration
• MEX - Model Extraction

Code & Execution:

• CEX - Code Execution (331 tests)
• OPS - Output Manipulation

Advanced Threats:

• AGY - Excessive Agency
• RAG - RAG/Vector Poisoning
• VEC - Vector/Embedding Attacks
• MEM - Memory Poisoning
• IAT - Inter-Agent Trust
• MCP - Model Context Protocol
• COT - Chain-of-Thought Manipulation
• IMG - Multi-modal Injection

Plus 10 more: Social Engineering, Compliance, Hallucination Exploitation, Resource Exhaustion, and others.

See SB_TAXONOMY_V2.1_FLAT.md for complete taxonomy.

Editions

Community (Open Source)

• Core testing engine
• ~330 baseline security tests
• CLI and MCP server
• Apache 2.0 License

PRO (Commercial)

• +750 advanced tests
• Industry-specific attack scenarios
• Continuous updates with latest techniques
• Priority support

Documentation

• Installation Guide
• Configuration
• MCP Server Setup
• Test Categories

Evaluation

Security Bench uses an LLM-as-Judge approach for evaluating test results. The judge model analyzes responses to determine if attacks succeeded (e.g., system prompt leaked, jailbreak bypassed safety filters). For specific attack types like TensorTrust, pattern matching is used instead.

Project Status

Status: Beta Version: 0.1.0

Working:

• ✅ LLM endpoint scanning (sb scan)
• ✅ Local security auditing (sb audit, sb code, sb config, sb infra)
• ✅ Rich terminal output with grades and findings
• ✅ JSON output for CI/CD integration

Coming Soon:

• MCP server for AI assistant integration
• HTML/PDF report generation

Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

Acknowledgments

See ACKNOWLEDGMENTS.md for research inspiration and credits.

License

Elastic License 2.0 (ELv2) - Free to use, but you may not offer it as a hosted service. See LICENSE for details.

---

Made with ❤️ for the AI security community