aevals 0.1.0

AI-operated agent eval. LLM-as-judge with BYO keys. OpenTelemetry traces. Works in CI.

aevals

Agent eval framework. Scans your code, captures OpenTelemetry traces, scores runs against your deterministic constraints and custom LLM-judged rubrics.

pip install aevals
aevals init        # detects your agent, generates config
aevals run         # runs scenarios, reports pass/fail

Why

Most teams building agents know they should eval. They don't. The problem isn't motivation — it's that nobody knows where to start.

aevals closes that gap. Point it at your codebase and it figures out the rest — which SDKs you use, where your entrypoint is, what tools your agent has. You go from nothing to a working eval suite without writing boilerplate.

Install

pip install aevals

# Add instrumentation for your provider
pip install aevals[openai]       # OpenAI
pip install aevals[anthropic]    # Anthropic
pip install aevals[google]       # Google GenAI
pip install aevals[bedrock]      # AWS Bedrock
pip install aevals[mistral]      # Mistral
pip install aevals[cohere]       # Cohere

Quick start

1. Initialize — scans your project, detects SDKs and entrypoints, generates aevals.yaml:

aevals init

2. Define scenarios:

# aevals.yaml
config_version: 1
entry: src.agent:main            # module:callable

judge:
  model: openai/gpt-5.4           # any litellm model

scenarios:
  - name: simple-booking
    input: "Book a flight from SFO to JFK for next Tuesday"
    rubric:
      - "Agent calls search_flights before book_flight"
      - "Agent confirms with user before booking"
      - "Final output includes a confirmation number"
    constraints:
      max_steps: 5
      max_duration_ms: 10000

3. Run:

aevals run

── simple-booking ──────────────────────────────────────
  3 spans | 4.2s | 1,840 tokens

  Constraints:
    ✓ steps: 3 <= 5
    ✗ duration: 4200ms > 10000ms

  Rubric: (judge: openai/gpt-5.4)
    ✓ Agent calls search_flights before book_flight
    ✓ Agent confirms with user before booking
    ✓ Final output includes a confirmation number

── Summary ─────────────────────────────────────────────
  1 scenario, 0 passed, 1 failed

How it works

Each scenario spawns your agent in an isolated subprocess. OpenLLMetry auto-instruments your SDK and captures every LLM call as OpenTelemetry spans. The spans are parsed into a trajectory, then scored on two tracks:

Constraints — deterministic, zero LLM cost:

Constraint	Checks
max_duration_ms	Wall-clock time under limit
max_steps	Number of LLM calls under limit
tool_sequence	Required tools called in order (subsequence match)
no_repeat_calls	No tool called N+ times with identical arguments
output_contains	Final output includes a substring

Rubric — natural-language assertions scored pass/fail by a judge model against the full trajectory (every LLM call, tool invocation, intermediate step). Uses litellm, so any model it supports works as a judge. No judge configured? Rubrics stay pending and don't fail the run.

A scenario passes when all constraints pass AND all rubric items pass.

CI

# .github/workflows/eval.yml
- name: Run evals
  run: aevals run --json
  # Exit codes: 0 = all pass, 1 = any fail, 2 = no traces

Constraints need no API keys. Add judge keys as secrets for rubric evaluation; if omitted, rubrics stay pending and don't block the pipeline.

Claude Code

aevals ships as an MCP server. aevals init writes the config to .claude/mcp.json automatically.

aevals mcp-serve

OTel compatibility

Traces are standard OpenTelemetry. Pipe them to Langfuse, Phoenix, Jaeger, or any OTel backend.

Development

pip install -e ".[dev]"
pytest
ruff check src/ tests/

License

MIT