aevals 0.1.1

Agent evals framework that gives Claude Code the tools to instrument your codebase, capture traces, write evals, and catch LLM regressions.

aevals

Agent evals framework that gives Claude Code the tools to instrument your codebase, capture traces, write evals, and catch LLM regressions.

pip install aevals
aevals init        # detects your agent, generates config
aevals run         # runs scenarios, reports pass/fail

Why

Most teams building agents know they should eval. They don't. The problem isn't motivation — it's that nobody knows where to start.

aevals closes that gap. Point it at your codebase and it figures out the rest — which SDKs you use, where your entrypoint is, what tools your agent has. You go from nothing to a working eval suite without writing boilerplate.

Install

pip install aevals

# Add instrumentation for your provider
pip install aevals[openai]       # OpenAI
pip install aevals[anthropic]    # Anthropic
pip install aevals[google]       # Google GenAI
pip install aevals[bedrock]      # AWS Bedrock
pip install aevals[mistral]      # Mistral
pip install aevals[cohere]       # Cohere

Quick start

1. Initialize — scans your project, detects SDKs and entrypoints, generates aevals.yaml:

aevals init

2. Define scenarios:

# aevals.yaml
config_version: 1
entry: src.agent:main            # module:callable

judge:
  model: openai/gpt-5.4           # any litellm model

scenarios:
  - name: simple-booking
    input: "Book a flight from SFO to JFK for next Tuesday"
    rubric:
      - "Agent calls search_flights before book_flight"
      - "Agent confirms with user before booking"
      - "Final output includes a confirmation number"
    constraints:
      max_steps: 5
      max_duration_ms: 10000

3. Run:

aevals run

── simple-booking ──────────────────────────────────────
  3 spans | 4.2s | 1,840 tokens

  Constraints:
    ✓ steps: 3 <= 5
    ✗ duration: 4200ms > 10000ms

  Rubric: (judge: openai/gpt-5.4)
    ✓ Agent calls search_flights before book_flight
    ✓ Agent confirms with user before booking
    ✓ Final output includes a confirmation number

── Summary ─────────────────────────────────────────────
  1 scenario, 0 passed, 1 failed

How it works

Each scenario spawns your agent in an isolated subprocess. OpenLLMetry auto-instruments your SDK and captures every LLM call as OpenTelemetry spans. The spans are parsed into a trajectory, then scored on two tracks:

Constraints — deterministic, zero LLM cost:

Constraint	Checks
max_duration_ms	Wall-clock time under limit
max_steps	Number of LLM calls under limit
tool_sequence	Required tools called in order (subsequence match)
no_repeat_calls	No tool called N+ times with identical arguments
output_contains	Final output includes a substring

Rubric — natural-language assertions scored pass/fail by a judge model against the full trajectory (every LLM call, tool invocation, intermediate step). Uses litellm, so any model it supports works as a judge. No judge configured? Rubrics stay pending and don't fail the run.

A scenario passes when all constraints pass AND all rubric items pass.

CI

# .github/workflows/eval.yml
- name: Run evals
  run: aevals run --json
  # Exit codes: 0 = all pass, 1 = any fail, 2 = no traces

Constraints need no API keys. Add judge keys as secrets for rubric evaluation; if omitted, rubrics stay pending and don't block the pipeline.

Claude Code

aevals ships as an MCP server. aevals init writes the config to .claude/mcp.json automatically.

aevals mcp-serve

OTel compatibility

Traces are standard OpenTelemetry. Pipe them to Langfuse, Phoenix, Jaeger, or any OTel backend.

Development

pip install -e ".[dev]"
pytest
ruff check src/ tests/

License

MIT