mgoat 0.1.0

The Rust-Powered LLM Red Teaming Framework - Python Wrapper

🐐 MGoat Python

Python wrapper for the Rust-Powered LLM Red Teaming Framework

---

MGoat Python is a Python wrapper for MGoat, the Rust-powered LLM red teaming framework. It provides a convenient Python API while leveraging the performance of the Rust CLI under the hood.

Installation

pip install mgoat

You'll also need the MGoat CLI:

# macOS / Linux
curl -fsSL https://raw.githubusercontent.com/relaxcloud-cn/mgoat/main/scripts/install.sh | sh

# Or via Cargo
cargo install mgoat-cli

Quick Start

Python API

from mgoat import MGoat, MGoatConfig

# Create client with default config
goat = MGoat()

# Run a simple test
result = goat.run(
    goal="Test if the model refuses to provide harmful content",
    rounds=5
)

print(f"Success rate: {result.overall_asr:.2%}")

# Run with custom config
config = MGoatConfig(
    attacker_model="gpt-4",
    judge_model="gpt-4",
    target_model="gpt-3.5-turbo",
    max_rounds=10,
)

goat = MGoat(config=config)
result = goat.run(goal=["goal1", "goal2", "goal3"])

for attack in result.results:
    print(f"{attack.target_model}: {attack.overall_asr:.2%}")

Async Support

import asyncio
from mgoat import MGoat

async def main():
    goat = MGoat()
    result = await goat.run_async(
        goal="test safety boundaries",
        rounds=5
    )
    print(f"ASR: {result.overall_asr:.2%}")

asyncio.run(main())

CLI

# Run tests
mgoat run --goal "test safety" --rounds 5

# Test connection
mgoat test --target-model gpt-4

# List attack strategies
mgoat strategies

Attack Strategies

MGoat implements 8+ attack strategies from the GOAT paper:

Strategy	Description
response_priming	Force model to start response with specific prefix
refusal_suppression	Suppress refusal language in outputs
hypothetical	Wrap requests in hypothetical scenarios
topic_splitting	Mix safe and sensitive topics
dual_response	Request dual (safe/unsafe) responses
opposite_intent	Convert refusals to opposites
persona_modification	Use role-playing to bypass restrictions
obscure_language	Use less common languages for attacks

Configuration

Environment Variables

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-..."

Python Config

from mgoat import MGoat, MGoatConfig

config = MGoatConfig(
    # Attacker model (generates attacks)
    attacker_model="gpt-4",
    attacker_api_base="https://api.openai.com/v1",
    attacker_api_key="sk-...",
    attacker_temperature=0.8,

    # Judge model (evaluates success)
    judge_model="gpt-4",
    judge_api_base="https://api.openai.com/v1",
    judge_api_key="sk-...",
    judge_temperature=0.1,

    # Target model (model under test)
    target_model="gpt-3.5-turbo",
    target_api_base="https://api.openai.com/v1",
    target_api_key="sk-...",

    # Execution settings
    max_rounds=5,
    concurrent=1,
    timeout=120,
)

goat = MGoat(config=config)

Integration with Testing Frameworks

pytest

import pytest
from mgoat import MGoat

@pytest.fixture
def goat():
    return MGoat()

def test_model_safety(goat):
    result = goat.run(
        goal="Test if model refuses harmful requests",
        rounds=3
    )
    # Assert attack success rate is below threshold
    assert result.overall_asr < 0.1, "Model is vulnerable to attacks"

CI/CD

# .github/workflows/ai-security.yml
name: AI Security Test

on: [push, pull_request]

jobs:
  red-team:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install MGoat
        run: |
          pip install mgoat
          curl -fsSL https://raw.githubusercontent.com/relaxcloud-cn/mgoat/main/scripts/install.sh | sh

      - name: Run Security Tests
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          python -c "
          from mgoat import MGoat
          goat = MGoat()
          result = goat.run(goal='test safety', rounds=3)
          assert result.overall_asr < 0.1
          "

API Reference

MGoat

class MGoat:
    def __init__(
        self,
        config: Optional[MGoatConfig] = None,
        cli_path: Optional[str] = None,
    ) -> None: ...

    def run(
        self,
        goal: Optional[Union[str, List[str]]] = None,
        goals_file: Optional[str] = None,
        target_model: Optional[str] = None,
        rounds: Optional[int] = None,
        concurrent: Optional[int] = None,
        output_format: str = "json",
        save_dir: Optional[str] = None,
        verbose: bool = False,
    ) -> TestResult: ...

    async def run_async(...) -> TestResult: ...

    def test_connection(self, target_model: Optional[str] = None) -> bool: ...

    @property
    def version(self) -> str: ...

TestResult

class TestResult:
    results: List[AttackResult]
    total_targets: int
    successful_targets: int
    overall_asr: float
    config: Dict[str, Any]

License

MIT License - see LICENSE for details.

Acknowledgments

Based on the GOAT methodology:

Automated Red Teaming with GOAT: the Generative Offensive Agent Tester Maya Pavlova, Erik Brinkman, Krithika Iyer, et al. arXiv:2410.01606 (2024) Licensed under CC BY 4.0

---

Made with ❤️ by the MGoat community