AgentEval (Python)

Reliability and audit-ready testing for LLM agents. Run each scenario N times and get a determinism (flakiness) score, grounding checks, and a self-contained audit-ready HTML report.

This package is the Python bridge to the AgentEval engine (agenteval-core on npm) — one engine, no duplicated logic. It requires Node.js ≥ 20 on your PATH (the CLI tells you if it's missing).

60-second demo (no API keys)

pip install agenteval-python   # installs the `agenteval` CLI and `import agenteval`
mkdir demo && cd demo
agenteval init --demo-python   # scaffolds a mock Python agent — one scenario deliberately flaky
agenteval run --html report.html
open report.html               # the audit report, with the flaky scenario caught

Wrap your real agent (~10 lines)

my_agent.py:

import agenteval

@agenteval.adapter
def my_agent(input):
    result = run_my_agent(input["user_message"])  # LangGraph, CrewAI, raw loop, ...
    return agenteval.Trace(
        final_text=result.text,
        tool_calls=[agenteval.ToolCall(name=t.name, input=t.args) for t in result.tools],
        citations=[agenteval.Citation(source=c.source, quote=c.quote) for c in result.citations],
    )

if __name__ == "__main__":
    my_agent.serve()

agenteval.config.yaml:

adapter:
  command: python3
  args: [my_agent.py]
agentName: My agent
scenarios: ./scenarios
runs: 3

Then agenteval run --html report.html. Scenarios are plain YAML (see the main README for the assertion vocabulary: tool calls, text content, citation grounding, refusals, LLM-judge rubrics).

Already have traces?

Score traces you already collect (OpenTelemetry GenAI spans, LangSmith runs, or AgentEval's own trace shape) without re-running your agent:

import agenteval
agenteval.write_traces(traces, "traces.json")

agenteval eval --traces traces.json ./scenarios --html report.html
# or: --format otel | langsmith

License

MIT

Project links

Data verified by PyPI on Aug 31, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for lkank from gravatar.com lkank

Credits

Author: Lokesh Kank

GitHub Statistics

Data verified by PyPI on Aug 31, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT
View SPDX License List

Requires

Python >=3.9

Tags

agent-evaluation agents ai-testing audit compliance llm llm-eval llm-reliability rag-evaluation

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

agenteval_python-0.1.0.tar.gz (5.7 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

agenteval_python-0.1.0-py3-none-any.whl (6.2 kB view details)

Uploaded Python 3

File details

Details for the file agenteval_python-0.1.0.tar.gz.

File metadata

  • Download URL: agenteval_python-0.1.0.tar.gz
  • Upload date:
  • Size: 5.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for agenteval_python-0.1.0.tar.gz
Algorithm Hash digest
SHA256 b315bbe6607a03ce3b92a5846bfae2e3c40c901b005e593718d88bdadf9f8f7e
MD5 a9f5f5fdfff44fbceed89e9b231ba205
BLAKE2b-256 6cde7ac2d85f2a9709dbd740634ab26bb8bc68e9bcfdc51a731010020f80cc49

See more details on using hashes here.

Provenance

The following attestation bundles were made for agenteval_python-0.1.0.tar.gz:

Publisher: release.yml on lokesh75-kank/agenteval

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file agenteval_python-0.1.0-py3-none-any.whl.

File metadata

File hashes

Hashes for agenteval_python-0.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 d62501d89c567eea795d03e0ae4db71f81f4f95a89f47df9fc3aed2ef9c0cdcf
MD5 53c9d93b1144c3100a1a5e78ee067142
BLAKE2b-256 65c06f64cbcb300e98a6778d1bda93d2da15047a98d4c6002ccb64af6ea95896

See more details on using hashes here.

Provenance

The following attestation bundles were made for agenteval_python-0.1.0-py3-none-any.whl:

Publisher: release.yml on lokesh75-kank/agenteval

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page