pytest-agent-eval 0.2.0

A pytest plugin for LLM evaluation tests with threshold-based pass/fail

pytest-agent-eval

LLM evaluation tests that actually mean something. A pytest plugin for testing LLM agents with threshold-based pass/fail scoring, multi-turn transcripts, and LLM-as-judge rubrics — without breaking your CI bill.

Highlights

• 🎯 Threshold-based pass/fail — run each test N times, pass when ≥ threshold% succeed
• 📝 YAML or Python transcripts — pick the authoring style your team prefers
• 🔍 YAML auto-discovery — drop *.yaml files in any configured directory and they become pytest tests automatically
• 🛡 CI-safe by default — eval tests skip unless --agent-eval-live or EVAL_LIVE=1
• ⚡ Parallel-ready — pytest -n auto (via pytest-xdist) just works
• 📄 Markdown reports — full per-run trace with --agent-eval-report=eval.md

Installation

# pip
pip install pytest-agent-eval

# uv
uv add pytest-agent-eval

Supported frameworks

pytest-agent-eval ships first-class adapters for the major Python agent frameworks. Each is an optional extra so you only install what you use.

Framework	Extra	Adapter
pydantic-ai	(default)	pytest_agent_eval.adapters.pydantic_ai.PydanticAIAdapter
LangChain / LangGraph	langchain	pytest_agent_eval.adapters.langchain.LangChainAdapter
OpenAI SDK	openai	pytest_agent_eval.adapters.openai.OpenAIAdapter
smolagents	smolagents	pytest_agent_eval.adapters.smolagents.SmolagentsAdapter
LiveKit (voice)	livekit	pytest_agent_eval.adapters.livekit.LiveKitAdapter

pip install "pytest-agent-eval[langchain]"
pip install "pytest-agent-eval[openai]"
pip install "pytest-agent-eval[smolagents]"
pip install "pytest-agent-eval[livekit]"
# or with uv:
uv add "pytest-agent-eval[langchain]"
uv add "pytest-agent-eval[openai]"
uv add "pytest-agent-eval[smolagents]"
uv add "pytest-agent-eval[livekit]"

Bringing your own framework? Any async def agent(messages) -> (reply, tool_calls) callable works directly — no base class needed.

What you can test

pytest-agent-eval separates the kinds of checks you might want into composable evaluators:

• Deterministic checks — ContainsEvaluator(any_of=["confirmed", "booked"]) for substring/regex assertions over the agent reply.
• Tool-call assertions — ToolCallEvaluator(must_include=["create_booking"], ordered=True) to verify that the agent called the right tools, in the right order.
• LLM-as-judge — JudgeEvaluator(rubric="Reply must be friendly, include a date, and confirm the booking.") for open-ended quality checks the agent under test should meet.

Mix and match per turn — every evaluator participates in the threshold score.

Quick start

You can author evals two ways. YAML is the recommended starting point — it lets non-Python contributors (PMs, QA, domain experts) write tests, keeps eval data readable in code review, and turns each .yaml file into a pytest test automatically.

1 — Configure where YAMLs live

# pyproject.toml
[tool.agent_eval]
model     = "openai:gpt-4o"   # used by the LLM-as-judge
threshold = 0.8               # default pass fraction
runs      = 3                 # default reps per transcript
yaml_dirs = ["tests/evals"]

2 — Wire up your agent once

# tests/conftest.py
import pytest
from pytest_agent_eval.adapters.pydantic_ai import PydanticAIAdapter
from my_app import build_agent

@pytest.fixture
def llm_eval_agent():
    return PydanticAIAdapter(build_agent())

3 — Write transcripts as YAML

# tests/evals/booking_single_turn.yaml
id: booking_single_turn
threshold: 0.8
runs: 3

turns:
  - user: "Book me a slot tomorrow at 10am."
    expect:
      reply_contains_any: ["confirmed", "booked"]
      tool_calls_include: ["create_booking"]
      judge:
        rubric: "Reply must confirm the booking and include a reference number."

4 — Run it

pytest --agent-eval-live

============================ test session starts =============================
plugins: agent-eval-0.1.0, asyncio-1.0.0
collected 2 items

tests/evals/booking_single_turn.yaml::booking_single_turn PASSED       [ 50%]
tests/evals/booking_multi_turn.yaml::booking_multi_turn PASSED         [100%]

============================== 2 passed in 14.03s ============================

By default eval tests are skipped outside of explicit live runs (so a missed pytest . doesn't burn API credits). Flip --agent-eval-live on, set EVAL_LIVE=1, or live = true in [tool.agent_eval] for local-only auto-on. See Configuration for the full precedence rules.

Multi-turn conversations

Real agents fail on context, not single-shot replies. A model that nails turn 1 might forget the user's name by turn 3, or call the wrong tool once the user changes their mind. Multi-turn YAML transcripts test the whole conversation arc, with each turn asserting against the agent's state at that point:

# tests/evals/booking_multi_turn.yaml
id: booking_multi_turn
threshold: 0.66          # tolerate 1/3 flaky runs
runs: 3
tags: [gate:booking, smoke]

turns:
  # Turn 1 — initial booking
  - user: "Book me a slot tomorrow at 10am."
    expect:
      reply_contains_any: ["confirmed", "booked"]
      tool_calls_include: ["create_booking"]

  # Turn 2 — agent must remember the booking from turn 1
  - user: "Actually, can you move it to 11am instead?"
    expect:
      tool_calls_include: ["update_booking"]
      tool_calls_exclude: ["create_booking"]   # must update, not double-book
      judge:
        rubric: "Confirms the new time AND references the original 10am booking."

  # Turn 3 — context propagates further
  - user: "Email me the confirmation."
    expect:
      tool_calls_include: ["send_email"]
      reply_contains_any: ["sent", "email"]

Each turn's full conversation history is built up as the test runs — your agent receives all prior (user, assistant) pairs as context, the same way it would in production. Failures point at the exact turn that broke, not just "the test failed."

Voice agents are supported via the [livekit] extra — see Voice testing. Each turn declares an audio: turn.wav path; the adapter streams it through a real LiveKit AgentSession and asserts on the same surface (tool calls, reply, judge rubric).

Sample report

Add --agent-eval-report=eval.md to get a human-readable trail of every run, every turn, and every evaluator's reasoning. Useful for CI artifacts and PR diffs:

pytest --agent-eval-live --agent-eval-report=eval.md

# LLM Eval Report — 2026-04-30

## Summary

| Transcript            | Runs | Passed | Score | Threshold | Status  |
|-----------------------|------|--------|-------|-----------|---------|
| booking_single_turn   | 3    | 3      | 1.00  | 0.80      | ✅ PASS |
| booking_multi_turn    | 3    | 2      | 0.67  | 0.66      | ✅ PASS |

## Details

### booking_multi_turn
**Run 1** ✅
- Turn 1: PASS
- Turn 2: PASS
  - Judge: Reply confirmed move to 11am and acknowledged original 10am slot.
- Turn 3: PASS
**Run 2** ❌
- Turn 1: PASS
- Turn 2: FAIL
  - Tool calls expected to include 'update_booking', got ['create_booking']
- Turn 3: PASS
**Run 3** ✅
- ...

The two-out-of-three pass still clears the 0.66 threshold, so the suite passes — that's the point of running each transcript multiple times instead of treating LLM tests as binary.

Python API

YAML covers most cases; drop into Python when you need parametrization, programmatic test generation, or per-test fixtures:

import pytest
from pytest_agent_eval import Turn, Expect, ContainsEvaluator, ToolCallEvaluator, JudgeEvaluator

@pytest.mark.agent_eval(threshold=0.8, runs=3)
async def test_booking(agent_eval):
    result = await agent_eval.run(
        agent=my_agent,
        turns=[
            Turn(
                user="Book me a slot tomorrow at 10am",
                expect=Expect(evaluators=[
                    ContainsEvaluator(any_of=["confirmed", "booked"]),
                    ToolCallEvaluator(must_include=["create_booking"]),
                    JudgeEvaluator(rubric="Reply must include a reference number."),
                ]),
            )
        ],
    )
    result.assert_threshold()

See the full documentation for the complete YAML reference, configuration options, parallel execution, and CI patterns. Contributions are welcome — see CONTRIBUTING.md.

License

MIT — see LICENSE.