EntroPy (entropy-ai): AI agent evaluation for Python
PyPI | Documentation | Quickstart | Issues
Test AI agent reliability with repeated Monte Carlo trials, behavioral entropy, drift detection, and regression checks. EntroPy is a Python evaluation library for non-deterministic agents, including LangChain, LangGraph and OpenAI Agents.
Measure uncertainty, drift, stochastic behavior, and emergent failures — via Monte Carlo evaluation over repeated runs and entropy-based metrics. Framework-agnostic, local-first, zero vendor lock-in.
Install
pip install entropy-ai # from PyPI — core (pandas, matplotlib, jinja2, rich, pyyaml)
pip install "entropy-ai[langchain]" # optional: a framework adapter
pip install "entropy-ai[pdf,jupyter]" # optional: PDF / Jupyter reports
Developing EntroPy itself? Clone the repo and use an editable install instead:
pip install -e . # editable install from the repo
pip install -e ".[pdf,jupyter]" # + report extras (what CI installs)
Quick start
from entropy import evaluate, assert_stable
def my_agent(text):
return text.strip().lower()
results = evaluate(my_agent, {" Hello ": "hello", "WORLD": "world"}, trials=20)
print(results["success_rate"]) # 1.0
print(results["behavioral_entropy"]) # 0.0: identical outputs across trials
assert_stable(results, success_rate__gte=0.95)
This example runs locally without an API key. Replace my_agent with your
synchronous agent callable. trials is a positive integer and runs per input;
2 inputs with 20 trials call the agent 40 times. Provider calls may incur costs.
Success means matching the expected answer above; without expected or check,
success only means the agent did not report an error, not that its answer is correct.
seed does not seed your agent or make a remote model deterministic.
Use EntroPy to compare repeated outputs, test tool trajectories, inject faults,
and gate releases with assert_stable. Trace events are needed for tool and
trajectory metrics; choose metrics that match the evidence your agent records.
Need more control? Use Suite:
from entropy import Suite, Dataset, Case
dataset = Dataset([Case(input="q1", expected="correct answer")])
results = Suite(seed=42).run(my_agent, dataset, trials=100)
evaluate / Suite.run accept a list of raw inputs, a {input: expected} dict,
or a Dataset — and a shared check= callable as the success criterion.
The canonical output (spec §9 / §21) includes: success_rate, reliability,
variance, robustness, entropy, confidence_interval, behavioral_entropy,
behavioral_variance, trajectory_entropy, trajectory_diversity, tool_reliability,
drift_score, goal_stability, recovery_score, emergent_behavior, and more.
56 metrics total; entropy.default_metrics() lists the canonical set.
Async agent evaluation
Use evaluate_async for agents that return awaitables, including async def
functions. It returns the same metrics and accepts the same expected, check,
metrics, and trials options as evaluate.
import asyncio
from entropy import evaluate_async
async def my_async_agent(text):
await asyncio.sleep(0)
return text.strip().lower()
async def main():
results = await evaluate_async(my_async_agent, {" Hello ": "hello"}, trials=20)
print(results["success_rate"]) # 1.0
asyncio.run(main())
In a notebook or async application, use await evaluate_async(...) directly.
For recorded batches or a custom extractor, use await Suite().run_async(...).
Trials run sequentially, preserving input and trial order. Checks and extractors
are synchronous; blocking synchronous agents still block the event loop.
Cancellation propagates; ordinary agent exceptions count as failed trials.
Existing framework adapters remain synchronous: pass an async callable around
an SDK's async API when using this evaluator.
Detailed evaluation, Pass@k and agent comparison
Inspect failures per input, estimate success within k attempts, and compare a prompt or model change against a baseline using paired bootstrap intervals.
from entropy import evaluate_detailed, compare_agents
cases = {" Hello ": "hello", "WORLD": "world", " Test ": "test"}
def baseline(text):
return text.strip()
def candidate(text):
return text.strip().lower()
report = evaluate_detailed(candidate, cases, trials=10, pass_k=[1, 5, 10])
print(report["pass_at_k"])
print(report["cases"][0]["confidence_interval"])
comparison = compare_agents(
{"baseline": baseline, "candidate": candidate}, cases,
trials=10, resamples=1000, seed=42,
)
print(comparison["comparisons"]["candidate"]["status"])
evaluate_detailed(...): aggregate metrics, per-input success rates, Wilson intervals, behavioral entropy, supplied cost/latency, failed outputs and errors.evaluate_detailed_async(...): the same report for async callables.compare_agents(...): named synchronous agents evaluated on identical inputs; metric deltas, improved/regressed case counts and paired bootstrap intervals.pass_at_k(total, correct, k): unbiased repeated-attempt success estimator.wilson_interval(correct, total, confidence=0.95): success-rate uncertainty that remains nonzero even when all observed trials succeed or fail.bootstrap_interval(values, confidence=0.95, resamples=2000, seed=0): reproducible percentile interval for an arithmetic mean (options are keyword-only).
Use expected answers or check= to measure correctness. All k values must be
between 1 and trials. Comparisons resample inputs, pairing each candidate's
per-input success rate with the baseline; at least two inputs are required for
an interval. Results are exploratory, without multiple-comparison correction.
Small datasets and correlated agent runs limit inference; constant observations
can yield point bootstrap intervals. seed controls resampling, not provider randomness.
Reports retain raw inputs and failed outputs in memory; review them before sharing.
CLI equivalents:
entropy run --agent mymodule:agent --dataset cases.json --trials 20 --detailed --pass-k 1 5 10 --out details.json
entropy benchmark --agents mymodule:old mymodule:new --dataset cases.json --trials 20 --paired --resamples 2000 --out comparison.json
Read the evaluation and comparison guide for report fields, statistical assumptions, and examples.
Tracing
from entropy import trace, event, instrument
with trace() as t:
event("action", "tool:search")
# or auto-wrap any agent:
wrapped = instrument(my_agent) # returns an AgentRun per call
Custom metrics (plugin system)
from entropy import metric
@metric("my_metric")
class MyMetric:
def compute(self, batch): ... # batch has outputs/events/successes/costs/...
Simulation, Chaos, Observability
from entropy import AdversarialSimulator, ChaosRunner, Watcher
adv = AdversarialSimulator(seed=0)
for v in adv.iterate("book a flight"):
print(v) # typo / injection / distraction / leak variants
ChaosRunner(seed=0).run(agent, dataset, ["api_fail", "timeout"]) # fault injection
Watcher(agent, dataset, trials=20).watch(rounds=10) # live drift + anomalies
Framework adapters (lazy)
from entropy import from_langchain, from_langgraph, from_openai, from_custom
runnable = from_langchain(my_langchain_agent) # maps framework runs -> AgentRun
All 9 adapters (LangChain, LangGraph, OpenAI Agents, CrewAI, PydanticAI,
AutoGen, Google ADK, MCP, custom) import their framework lazily, so
import entropy works with none of them installed.
CLI
entropy init # scaffold a project
entropy run --agent m:a --dataset d.json --trials 100
entropy test --agent m:a --dataset d.json --baseline base.json [--update]
entropy benchmark --agents m:a m:b --dataset d.json
entropy chaos --agent m:a --dataset d.json --faults api_fail,timeout
entropy simulate --kind adversarial --text "book a flight"
entropy ui --agent m:a --dataset d.json --serve
entropy watch --agent m:a --dataset d.json --rounds 10
entropy report --results r.json --format html
entropy dataset --path d.json
entropy doctor # check optional deps
Reports
from entropy import export_report
export_report(results, "html", "report.html") # also: json, markdown, pdf, jupyter
Layout
core/ (models, tracing, exporters, suite) · metrics/ (registry + all
categories) · datasets/ · simulation/ · chaos/ · regression/ ·
benchmarks/ · observability/ · reports/ · dashboard/ · integrations/
(adapters) · cli/ · plugins/.
Tests: python -m pytest tests/.
Documentation
-
Source:
docs/— plain HTML/CSS/JS with no build step, deployed to GitHub Pages. -
Source:
docs/index.html,docs/guide/*.html,docs/api.html,docs/examples.html, plusdocs/assets/(CSS, JS, logo, search index). -
Local preview: serve the folder, e.g.
python -m http.server -d docs 8000then openhttp://localhost:8000. -
Deploy: push to
main; the.github/workflows/pages.ymlworkflow uploadsdocs/and publishes it via GitHub Pages. Alternatively, set Settings → Pages → Build and deployment → Source to Deploy from a branch and choose themainbranch/docsfolder.
Metadata
Release files for entropy-ai 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| entropy_ai-0.2.0.tar.gz | 58.4 kB | Without provenance | Details for entropy_ai-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| entropy_ai-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for entropy_ai-0.2.0-py3-none-any.whl |
Total release size: 120.0 kB
Release files / entropy_ai-0.2.0.tar.gz
| Download URL | entropy_ai-0.2.0.tar.gz |
|---|---|
| Size | 58.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ba9943ab28800843b9177504a1cdf25d35e5dd4c476fe74f59d017a8a6e40e9c
|
|
BLAKE2b-256 checksum How to use checksums |
74898d52f301ba07b12431ab2a00f6db34aa22df64c4901f126d043c40e9d34a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.8
|
Release files / entropy_ai-0.2.0-py3-none-any.whl
| Download URL | entropy_ai-0.2.0-py3-none-any.whl |
|---|---|
| Size | 61.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
62dba520208b5524b7eecc2333a1ed3bfb8cb7578cccaf0c9722c97c4568b8be
|
|
BLAKE2b-256 checksum How to use checksums |
067ae3576f0fee6e72de2e6a6ae75de1ecfd9e0f3e26e29e938504c5113b7646
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.8
|
Security for entropy-ai 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.