Pydantic Evals
This is a library for evaluating non-deterministic (or "stochastic") functions in Python. It provides a simple, Pythonic interface for defining and running stochastic functions, and analyzing the results of running those functions.
While this library is developed as part of Pydantic AI, it only uses Pydantic AI for a small subset of generative functionality internally, and it is designed to be used with arbitrary "stochastic function" implementations. In particular, it can be used with other (non-Pydantic AI) AI libraries, agent frameworks, etc.
As with Pydantic AI, this library prioritizes type safety and use of common Python syntax over esoteric, domain-specific use of Python syntax.
Full documentation is available at ai.pydantic.dev/evals.
Example
While you'd typically use Pydantic Evals with more complex functions (such as Pydantic AI agents or graphs), here's a quick example that evaluates a simple function against a test case using both custom and built-in evaluators:
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext, IsInstance
# Define a test case with inputs and expected output
case = Case(
name='capital_question',
inputs='What is the capital of France?',
expected_output='Paris',
)
# Define a custom evaluator
class MatchAnswer(Evaluator[str, str]):
def evaluate(self, ctx: EvaluatorContext[str, str]) -> float:
if ctx.output == ctx.expected_output:
return 1.0
elif isinstance(ctx.output, str) and ctx.expected_output.lower() in ctx.output.lower():
return 0.8
return 0.0
# Create a dataset with the test case and evaluators
dataset = Dataset(
name='capital_eval',
cases=[case],
evaluators=[IsInstance(type_name='str'), MatchAnswer()],
)
# Define the function to evaluate
async def answer_question(question: str) -> str:
return 'Paris'
# Run the evaluation
report = dataset.evaluate_sync(answer_question)
report.print(include_input=True, include_output=True)
"""
Evaluation Summary: answer_question
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Case ID ┃ Inputs ┃ Outputs ┃ Scores ┃ Assertions ┃ Duration ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ capital_question │ What is the capital of France? │ Paris │ MatchAnswer: 1.00 │ ✔ │ 10ms │
├──────────────────┼────────────────────────────────┼─────────┼───────────────────┼────────────┼──────────┤
│ Averages │ │ │ MatchAnswer: 1.00 │ 100.0% ✔ │ 10ms │
└──────────────────┴────────────────────────────────┴─────────┴───────────────────┴────────────┴──────────┘
"""
Using the library with more complex functions, such as Pydantic AI agents, is similar — all you need to do is define a task function wrapping the function you want to evaluate, with a signature that matches the inputs and outputs of your test cases.
Logfire Integration
Pydantic Evals uses OpenTelemetry to record traces for each case in your evaluations.
You can send these traces to any OpenTelemetry-compatible backend. For the best experience, we recommend Pydantic Logfire, which includes custom views for evals:
You'll see full details about the inputs, outputs, token usage, execution durations, etc. And you'll have access to the full trace for each case — ideal for debugging, writing path-aware evaluators, or running the similar evaluations against production traces.
Basic setup:
import logfire
logfire.configure(
send_to_logfire='if-token-present',
environment='development',
service_name='evals',
)
...
my_dataset.evaluate_sync(my_task)
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file pydantic_evals-2.24.0.tar.gz.
File metadata
- Download URL: pydantic_evals-2.24.0.tar.gz
- Upload date:
- Size: 85.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
0ff5fd9ed6502645236bcce6eec2a7ee39b1e174347335f472905a4e85f7c9f3
|
|
| MD5 |
2f195eab148c3fc315b092da58335b67
|
|
| BLAKE2b-256 |
3aa89db0909b4e3bcccbb8a6a560bf18e1fe4b576fa34286fd8630e985b5b1c5
|
Provenance
The following attestation bundles were made for pydantic_evals-2.24.0.tar.gz:
Publisher:
ci.yml on pydantic/pydantic-ai
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
pydantic_evals-2.24.0.tar.gz -
Subject digest:
0ff5fd9ed6502645236bcce6eec2a7ee39b1e174347335f472905a4e85f7c9f3 - Sigstore transparency entry: 2341953602
- Sigstore integration time:
-
Permalink:
pydantic/pydantic-ai@7a64d049c3f5271a975cd1d64b2fa876d83ede1d -
Branch / Tag:
refs/tags/v2.24.0 - Owner: https://github.com/pydantic
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
ci.yml@7a64d049c3f5271a975cd1d64b2fa876d83ede1d -
Trigger Event:
push
-
Statement type:
File details
Details for the file pydantic_evals-2.24.0-py3-none-any.whl.
File metadata
- Download URL: pydantic_evals-2.24.0-py3-none-any.whl
- Upload date:
- Size: 100.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
5a751012040c235ee1b8b8e25fc5e31a50f8be397ccfe3193ed323ee85d622ee
|
|
| MD5 |
970e9093ad8e33b6e32cf2e0f0cb8544
|
|
| BLAKE2b-256 |
4120bf5ea1048958c8d692a3d02a8bdac1b8d20bbd7b41945dcba4e5e33f85bf
|
Provenance
The following attestation bundles were made for pydantic_evals-2.24.0-py3-none-any.whl:
Publisher:
ci.yml on pydantic/pydantic-ai
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
pydantic_evals-2.24.0-py3-none-any.whl -
Subject digest:
5a751012040c235ee1b8b8e25fc5e31a50f8be397ccfe3193ed323ee85d622ee - Sigstore transparency entry: 2341954390
- Sigstore integration time:
-
Permalink:
pydantic/pydantic-ai@7a64d049c3f5271a975cd1d64b2fa876d83ede1d -
Branch / Tag:
refs/tags/v2.24.0 - Owner: https://github.com/pydantic
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
ci.yml@7a64d049c3f5271a975cd1d64b2fa876d83ede1d -
Trigger Event:
push
-
Statement type:
