evalloop

Sentry for AI behavior. Closed-loop eval monitoring for LLM-powered products.

You changed your prompt on Friday. Your bot broke. Your users noticed before you did.

evalloop wraps your LLM client with one line. Every call is scored against your known-good baselines in the background — zero added latency. Regressions surface in your terminal before they reach your users.

Who this is for

You're building an AI product — a chatbot, support bot, summarizer, coding assistant. You're making prompt changes regularly. You have no eval system. You've shipped a regression at least once and found out from a user.

Before evalloop:

Prompt change pushed Friday 5pm
→ Users complain Monday morning
→ 4 hours debugging which change broke what
→ Average detection time: 3 days

After evalloop:

Prompt change pushed Friday 5pm
→ evalloop watch alerts Friday 5:02pm
→ Developer reverts before closing laptop
→ Average detection time: 2 minutes

Install

pip install evalloop

Requires Python 3.9+. Then run the setup wizard:

evalloop init

evalloop auto-detects your scoring backend from environment variables — no extra config needed:

If you're already wrapping an Anthropic or OpenAI client, evalloop uses your existing key automatically — no extra signup required.

Quickstart

from evalloop import wrap
import anthropic

# One line change — evalloop uses your existing ANTHROPIC_API_KEY automatically
client = wrap(anthropic.Anthropic(), task_tag="qa")

resp = client.messages.create(
    model="claude-haiku-4-5-20251001",
    max_tokens=256,
    messages=[{"role": "user", "content": "What is the capital of France?"}],
)
# ^ scored against your baselines in the background. Zero added latency.

Then check your dashboard:

$ evalloop status

🟢  [qa]
   Calls captured : 142
   Scored         : 142
   Avg (7d)       : 0.81
   Avg (24h)      : 0.79
   Trend (recent) : ▆▆▇▇▆▅▆▇▇▆▇▆▇▇▆▆▇▆▇▆

Or watch for regressions automatically:

$ evalloop watch --interval 60

evalloop watch — polling every 60s. Ctrl-C to stop.

🔴  [qa]
   Calls captured : 201
   Avg (7d)       : 0.81
   Avg (24h)      : 0.71
   ⚠  Regression   : score dropped 10.0pp in last 24h

^C
Watch stopped.

How it works

your call
    │
    ▼
wrap(client).messages.create()
    │  returns immediately (zero latency added)
    │
    └─▶ background thread
            │
            ├─▶ infer task tag (from system prompt, if not set)
            ├─▶ score(output, baselines)
            │       ├─▶ heuristics (empty, too short, too long)
            │       └─▶ cosine similarity via Voyage AI
            └─▶ sqlite insert → ~/.evalloop/calls.db

• Zero latency — scoring runs in a background thread
• Silent on errors — disk full, API down, DB locked → log to stderr, never crash your app
• Graceful exit — on normal Python exit, flushes the queue so no captures are lost
• Degraded mode — if Voyage AI is unavailable, heuristic-only scoring continues (flags degraded_mode)
• Self-hosted — all data stays local in ~/.evalloop/

Auto tag inference

If you don't set task_tag, evalloop infers it from your system prompt:

client = wrap(anthropic.Anthropic())  # no task_tag needed

resp = client.messages.create(
    model="claude-haiku-4-5-20251001",
    max_tokens=256,
    system="You are a helpful assistant that summarizes documents.",
    # ↑ evalloop detects → task_tag="summarization"
    messages=[{"role": "user", "content": "Summarize this article..."}],
)

Supported task types for auto-inference: qa, summarization, code, customer-service, classification

CLI commands

# First-time setup — detect keys, choose scoring backend, install baselines
evalloop init

# Score trends for all tags
evalloop status

# Filter to one tag
evalloop status --tag qa

# Watch for regressions (polls every 60s by default)
evalloop watch
evalloop watch --tag qa --interval 30

# Export calls + scores for sharing or analysis
evalloop export                         # JSON to stdout
evalloop export --format csv -o out.csv # CSV to file
evalloop export --tag qa --limit 500    # filtered export

# Manage baselines
evalloop baseline add "your good output" --tag my-task
evalloop baseline list
evalloop baseline install               # install all built-in defaults
evalloop baseline install --tag qa     # install for one tag
evalloop baseline install --overwrite  # replace existing defaults

# List available built-in task types
evalloop defaults

Scoring

evalloop uses consistency-first scoring: deterministic heuristics run first, then the best available semantic backend.

Backend is auto-detected from your environment. Run evalloop init to configure.

Scores range 0.0–1.0. A score below your 7-day average by >5pp triggers a regression flag.

Baselines

Baselines are your known-good outputs. evalloop ships with curated defaults for common task types — you get meaningful scores immediately on install.

# Install defaults for all task types
evalloop baseline install

# Add your own known-good output
evalloop baseline add "The capital of France is Paris." --tag qa

# List all tags with baselines
evalloop baseline list

Built-in task types: qa, summarization, code, customer-service, classification

OpenAI support

from evalloop import wrap
import openai

# pip install "evalloop[openai]"
client = wrap(openai.OpenAI(), task_tag="summarization")

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You summarize documents."},
        {"role": "user", "content": "Summarize: ..."},
    ],
)

Data & Privacy

evalloop stores the following locally in ~/.evalloop/calls.db:

For PII-sensitive environments (HIPAA, GDPR), opt out of input storage:

client = wrap(anthropic.Anthropic(), task_tag="support", store_inputs=False)
# Input messages are never written to disk. Output + scores are still captured.

All data stays on your machine. evalloop has no cloud component.

Export & share

Share your eval data with your team:

# Export last 1000 calls as JSON
evalloop export > eval-report.json

# Export as CSV for Excel/Sheets
evalloop export --format csv -o eval-report.csv

# Export a specific tag
evalloop export --tag qa --limit 500

Architecture

• Storage: SQLite at ~/.evalloop/calls.db (WAL mode — concurrent reads while writing)
• Baselines: JSONL files at ~/.evalloop/baselines/<tag>.jsonl
• Scoring backends: Voyage AI voyage-3-lite (semantic embeddings) or Claude Haiku (LLM-as-judge) — auto-detected from env vars
• Score model provenance: stored per-row so history remains interpretable if backend changes
• No cloud dependency — everything runs locally

License

MIT