probegpt

Automated red-team discovery for AI models.

probegpt iteratively generates, sends, and scores adversarial prompts ("probes") against a target language model. It uses a separate generator model to create probes and a judge model to evaluate whether each probe succeeded — no manual prompt writing required.

Features

• 8 built-in attack strategies — objective, mutation, encoding, roleplay, persuasion, linguistic, structural, distraction
• Multi-provider support — Azure OpenAI, OpenRouter, GCP Vertex AI, AWS Bedrock, Cerebras, Mistral
• Textual TUI — interactive terminal UI for configuration and live run monitoring
• Extensible — drop custom strategy files into ~/.config/probegpt/strategies/
• JSON export — save all results for offline analysis

Installation

pip install probegpt

Or with uv:

uv tool install probegpt

Quick start

probegpt

The TUI will open. Navigate to Config first to set up your target, generator, and judge providers, then return to the main menu and select Run Discovery.

Configuration

Config is stored at ~/.config/probegpt/config.json (XDG-compliant). The TUI writes this file automatically when you save from the Config screen.

Provider credentials

A .env file in the working directory is loaded automatically.

Strategies

Each strategy can be enabled/disabled and weighted independently from the Strategies tab in the Config screen.

Custom strategies

Place a Python file in ~/.config/probegpt/strategies/. It must define a class that subclasses probegpt.strategies.base.BaseStrategy:

# ~/.config/probegpt/strategies/my_strategy.py
from probegpt.strategies.base import BaseStrategy

class MyStrategy(BaseStrategy):
    name = "my_strategy"

    async def generate(self, objective: str, count: int, seed_pool: list) -> list[str]:
        # Return a list of probe strings
        ...

The file is loaded automatically on startup.

Architecture

probegpt/
├── cli/          # Textual TUI (app.py) and entry point (main.py)
├── core/         # Config, models (Probe, DiscoveryResult), discovery loop
├── strategies/   # 8 built-in strategies + custom loader
├── judge/        # ObjectiveJudge — evaluates target responses
├── grading/      # Grader — scores probes, manages seed pool
├── providers/    # Factory for all supported LLM providers
├── output/       # AbstractOutput, JsonOutput
└── data/         # YAML prompt templates and seed probes

Development

git clone https://github.com/your-org/probegpt
cd probegpt
uv sync
uv run probegpt

Run linting:

uv run ruff check .
uv run ruff format .

License

MIT