agenttest

The pytest of AI agents. Catch regressions before they reach prod.

You ship an agent. You change a prompt. Did it get better or worse?

You have no idea. No test suite. No CI. No diff. Just deploy and hope.

Every team shipping AI agents hits the same wall: your "eval" is manually pasting examples into a playground. One prompt tweak could break everything—or fix everything—and you won't know until a user complains.

30-Second Quickstart

pip install agenttest-py
export ANTHROPIC_API_KEY=your_key
agenttest init
agenttest run

Or from scratch:

# agent_test_example.py
from agenttest import eval, judge

def my_agent(query: str) -> str:
    return "Your agent's response"  # Replace with real agent

@eval
def test_customer_support():
    response = my_agent("I want a refund")
    assert judge.tone(response) == "empathetic"
    assert judge.no_hallucination(response)

agenttest run

Features

• Code-first — Tests are just Python. No YAML. No config hell.
• LLM-as-judge — 9 built-in scorers: tone, hallucination, relevance, toxicity, faithfulness, conciseness, custom criteria, A/B compare.
• Local & CI — Runs anywhere. Add 4 lines to GitHub Actions. No account. No dashboard.
• agenttest diff — Side-by-side view of how your agent's responses changed between two runs. The git diff for agent behavior.
• Caching — Judge results cached in .agenttest_cache/ to avoid redundant API calls.
• Parallel — --workers 4 for faster runs.

agenttest diff — The Git Diff for Agent Behavior

See exactly how your agent's responses changed between two runs:

agenttest run --tag v1    # Before your prompt change
agenttest run --tag v2    # After your prompt change
agenttest diff v1 v2

test_customer_support_refund:
  BEFORE: "I cannot help with refunds"           pass
  AFTER:  "I'd be happy to process that for you" pass
  DELTA:  ✓ improved

test_helpful_tone:
  BEFORE: "Our policy states no returns"         fail
  AFTER:  "I'm sorry to hear that. Let me help"   pass
  DELTA:  +1 ↑

Judge Functions

CI in 4 Lines

# .github/workflows/agenttest.yml
- run: pip install agenttest-py-py
- run: agenttest run
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Every PR shows whether your agent got better or worse.

agenttest vs Braintrust / DeepEval / Promptfoo

agenttest = pytest for agents. No dashboards. No SaaS. Your tests, your repo, your CI.

Config

# agenttest.toml
[agenttest]
model = "claude-3-5-haiku-latest"
timeout_seconds = 30
workers = 4
fail_threshold = 0.8
cache = true

[agenttest.env]
ANTHROPIC_API_KEY = "$ANTHROPIC_API_KEY"

Contributing

See CONTRIBUTING.md.

License

MIT