pytest-pyeval 0.2.0

pytest plugin integrating pydantic-evals

pytest-pyeval

A pytest plugin integrating pydantic-evals

Run evals via pytest with the power of fixtures and using a familiar Arrange, Act, Evaluate pattern.

Example

from pyeval import dataset, Case, EqualsExpected, Contains


def uppercase_text(text: str) -> str:
    return text.upper()


@dataset(
    Case(
        name="uppercase_basic",
        inputs="hello world",
        expected_output="HELLO WORLD",
    ),
    Case(
        name="uppercase_with_numbers",
        inputs="hello 123",
        expected_output="HELLO 123",
    ),
)
def eval_uppercase(case: Case):
    result = execute(uppercase_text, case)

    result.evaluate(EqualsExpected())
    result.evaluate(Contains(value="HELLO", case_sensitive=True))

$ uv run pytest

============================== test session starts ==============================
platform darwin -- Python 3.13.1, pytest-9.0.2, pluggy-1.6.0
plugins: anyio-4.12.1, pyeval-0.1.0
collected 2 items

tests/evals/eval_example.py ●●                                                                         [100%]

============================= 2 evaluated in 0.02s ==============================

Installation

uv add --dev pytest-pyeval