pytest-pyeval

A pytest plugin integrating pydantic-evals

Run evals via pytest with the power of fixtures and using a familiar Arrange, Act, Evaluate pattern.

Example

from pyeval import dataset, execute, Case, EqualsExpected, Contains


def uppercase_text(text: str) -> str:
    return text.upper()


@dataset(
    Case(
        name="uppercase_basic",
        inputs="hello world",
        expected_output="HELLO WORLD",
    ),
    Case(
        name="uppercase_with_numbers",
        inputs="hello 123",
        expected_output="HELLO 123",
    ),
)
def eval_uppercase(case: Case):
    result = execute(uppercase_text, case)

    result.evaluate(EqualsExpected())
    result.evaluate(Contains(value="HELLO", case_sensitive=True))

$ uv run pyeval

============================== test session starts ==============================
platform darwin -- Python 3.13.1, pytest-9.0.2, pluggy-1.6.0
plugins: anyio-4.12.1, pyeval-0.1.0
collected 2 items

tests/evals/eval_example.py ●●                                                                         [100%]

============================= 2 evaluated in 0.02s ==============================

Installation

uv add --dev pytest-pyeval

Running evals

pytest-pyeval keeps evals separate from your regular test suite. Evals are excluded from pytest by default, since they are typically slower, hit live APIs, and run on a different cadence to unit tests.

pyeval                     # discover and run all evals in the project
pyeval evals/              # run evals under a specific path
pyeval evals/eval_foo.py   # run a single eval file