aibreaker

Official Python SDK for AI Breaker Lab.

Install

pip install aibreaker

Quick start

from aibreaker import BreakerClient

client = BreakerClient(
    api_key="client_key",
    groq_api_key="gsk_...",   # forwarded to backend for test generation + judging
)

report = client.break_model(
    target={
        "type":       "openai",
        "base_url":   "https://api.openai.com",
        "api_key":    "sk-...",
        "model_name": "gpt-4o-mini",
    },
    description="Customer-support chatbot for an e-commerce platform",
    num_tests=20,
    fail_threshold=5.0,
)

print(report)
# Report a3f8b2c1  ✓ PASSED
#   Score         : 7.40 / 10
#   Failures      : 3 / 20
#   Hallucinations: 1
#   Agreement     : 100%
#   Red flags     : 0

if not report.passed:
    for f in report.failures:
        print(f"  ✗ [{f.test_type}] score={f.score}  {f.question}")
    raise SystemExit(1)

Supported target types

Works with any OpenAI-compatible endpoint: Groq, Gemini, vLLM, Ollama, etc.

LangChain example

target = {
    "type": "langchain",
    "chain_import_path": "my_module.my_chain",  # must be importable by the backend
    "invoke_key": "question",
}

Report object

Error handling

All SDK calls raise BreakerError when the backend returns a non-2xx response (e.g. 401/429/5xx) or when polling fails (timeouts, network errors).

from aibreaker import BreakerClient, BreakerError

client = BreakerClient(
    api_key="client_key",
    endpoint="https://llm-eval-engine-production.up.railway.app",
    poll_interval=5,  # seconds between GET /report/{id}
    timeout=600,      # total seconds to wait before giving up
)

try:
    report = client.break_model(
        target={
            "type": "openai",
            "base_url": "https://api.openai.com",
            "api_key": "sk-...",
            "model_name": "gpt-4o-mini",
        },
        description="Customer-support chatbot for an e-commerce platform",
        num_tests=20,
    )
except BreakerError as e:
    print(f"BreakerError: {e}")
    raise

print(report.score, report.passed)

timeout and poll_interval are set on BreakerClient(...). Increase timeout for long-running jobs (large num_tests, slow target, heavy load). Decrease poll_interval if you want more frequent status checks, but keep it reasonable to avoid extra load and 429s.

CI / CD

See the aibreaker GitHub Action for one-step CI integration.

- uses: your-org/aibreaker-action@v1
  with:
    api_key: ${{ secrets.BREAKER_API_KEY }}
    groq_api_key: ${{ secrets.GROQ_API_KEY }}
    endpoint: https://llm-eval-engine-production.up.railway.app
    description: "Customer support chatbot"
    fail_threshold: "5.0"