ProbeAgent

Offensive security testing for AI agents. They scan configs. We attack your agent.

What is ProbeAgent?

ProbeAgent is a CLI tool that performs automated red-teaming of AI agents. It launches realistic multi-turn attacks — prompt injection, credential exfiltration, indirect injection, social manipulation, and more — against any HTTP-accessible agent.

Most AI security tools scan static configurations or check for known patterns. ProbeAgent actually attacks your running agent and tells you whether it's Safe, At Risk, or Compromised.

How It Works

flowchart LR
    CLI[probeagent attack] --> Engine
    Engine --> |for each category| Attack[Attack Module]
    Attack --> |reset conversation| Target
    Attack --> |multi-turn prompts| Target
    Target --> |response| Analyzer
    Analyzer --> |grade| Report[Safe / At Risk / Compromised]

Why ProbeAgent?

Installation

pip install probeagent-ai

Or install from source for development:

git clone https://github.com/sumamovva/probeagent.git
cd probeagent
pip install -e ".[dev]"

For PyRIT integration (evasion converters + dynamic red teaming):

pip install 'probeagent-ai[pyrit]'

Quickstart

Instant demo (no setup required)

pip install probeagent-ai
probeagent demo

This attacks a built-in mock target — a vulnerable agent and a hardened one — and shows a side-by-side comparison. No API keys, no server, no config.

Scan your own agent

# Validate your target is reachable
probeagent validate https://your-agent.example.com/api

# Run a quick security scan
probeagent attack https://your-agent.example.com/api --profile quick

# Full scan with parallel execution
probeagent attack https://your-agent.example.com/api --profile standard --parallel

Demo

Instant demo

Run a complete security assessment in seconds with zero setup:

probeagent demo

Add the War Room tactical display for a visual experience:

probeagent demo --game

Live demo (real API)

For demos against a real Claude-powered email agent with built-in vulnerabilities:

export ANTHROPIC_API_KEY=sk-ant-...
pip install 'probeagent-ai[demo]'
probeagent demo --live

The live demo starts a local email agent server with three endpoints at increasing security hardness, then attacks them. See tools/demo_email_agent.py for details.

Commands

probeagent demo

Run a full demo — attack a vulnerable + hardened target and compare results.

probeagent demo                    # Instant, uses mock target
probeagent demo --game             # With War Room tactical display
probeagent demo --live             # Real API (requires ANTHROPIC_API_KEY)
probeagent demo --profile standard # Use a different attack profile

Options:

• --live — Use real API (starts demo email agent server)
• --game — Launch War Room UI after attacks
• --profile, -p — Attack profile: quick, standard, or thorough (default: quick)

probeagent attack <url>

Run security attacks against a target AI agent.

probeagent attack https://agent.example.com/api --profile quick
probeagent attack https://agent.example.com/api --profile standard --output json -f report.json
probeagent attack https://agent.example.com/api -p standard --converters stealth --parallel

Options:

• --profile, -p — Attack profile: quick, standard, or thorough (default: quick)
• --target-type — Target type: http or openclaw (default: http)
• --output, -o — Output format: terminal, markdown, json (default: terminal)
• --output-file, -f — Write report to file
• --timeout, -t — Request timeout in seconds (default: 30)
• --parallel — Run attack categories in parallel for faster scans
• --converters — Apply evasion converters: basic, advanced, stealth, or comma-separated names (requires PyRIT)
• --redteam — Enable dynamic LLM-driven attacks via PyRIT RedTeamOrchestrator (requires PyRIT)

probeagent validate <url>

Check if a target is reachable and detect its API format.

probeagent list-attacks

Show all available attack modules with severity and status.

probeagent init

Create a default .probeagent.yaml config file in the current directory.

probeagent game [url]

Launch the War Room tactical display UI in your browser for interactive testing.

Attack Categories

11 attack categories with 69 strategies total:

Attack Profiles

PyRIT Integration

ProbeAgent optionally integrates with Microsoft PyRIT for advanced capabilities:

• Evasion Converters (--converters): Transform attack payloads with Base64, ROT13, Unicode substitution, leetspeak, and more to test resilience against obfuscated attacks
• Dynamic Red Teaming (--redteam): Use an LLM-driven orchestrator to generate novel attack strategies in real time

# Apply stealth evasion converters
probeagent attack https://agent.example.com/api -p standard --converters stealth

# Dynamic red teaming
probeagent attack https://agent.example.com/api -p standard --redteam

# Combine both
probeagent attack https://agent.example.com/api -p standard --converters advanced --redteam

Install with: pip install 'probeagent-ai[pyrit]'

Responsible Use

ProbeAgent is designed for authorized security testing only. Before using ProbeAgent:

• Ensure you have explicit permission to test the target system
• Only test systems you own or have written authorization to test
• Follow your organization's security testing policies
• Report vulnerabilities through proper disclosure channels

Unauthorized use of this tool against systems you don't own or have permission to test may violate laws and regulations.

Attribution

ProbeAgent's indirect injection and config manipulation attacks are inspired by research from Zenity Labs. PyRIT integration uses components from Microsoft PyRIT (MIT License). See ATTRIBUTION.md for full credits.

Development

# Install with dev dependencies
pip install -e ".[dev]"

# Run tests
python -m pytest tests/ -v

# Lint
ruff check src/ tests/

# Format
ruff format src/ tests/

See CONTRIBUTING.md for full development guidelines.

Roadmap

• Phase 1: CLI, HTTP target, scoring, reporting
• Phase 2: 9 attack categories with 56 multi-turn strategies
• Phase 3: OpenClaw target adapter, parallel execution, War Room UI
• Phase 4: Zenity-inspired attacks (indirect injection, config manipulation), PyRIT integration
• Phase 5: MCP target adapter, CI/CD integration, SaaS dashboard

License

Apache 2.0 — see LICENSE for details.