Eval AI Library
Based on firstlinesoftware/eval-ai-library. This is an independently maintained version with additional features and PyPI distribution.
Comprehensive AI model evaluation framework for RAG systems and AI agents. Supports 35+ evaluation metrics, 30+ LLM providers (OpenAI, Anthropic, Google, Azure, AWS Bedrock, Vertex AI, Mistral, Groq, xAI, DeepSeek, Cohere, OpenRouter, Together, Fireworks, Ollama, MLX, and more via LiteLLM), built-in test data generation from documents, an interactive web dashboard for visualization and analysis, and a first-class tracing subsystem with framework integrations for LangChain, LlamaIndex, CrewAI, AutoGen, Haystack, Semantic Kernel, Claude Agent SDK, smolagents, phidata, and OpenAI Assistants (plus an OpenTelemetry exporter). Implements advanced techniques including G-Eval probability-weighted scoring and Temperature-Controlled Verdict Aggregation via Generalized Power Mean.
Installation
pip install eval-ai-library
Full version with document parsing and OCR support:
pip install eval-ai-library[full]
Lite version (core evaluation only):
pip install eval-ai-library[lite]
Quick Start
from eval_lib import EvalAI
evaluator = EvalAI(model="gpt-4o")
result = evaluator.evaluate(
input="What is Python?",
actual_output="Python is a programming language.",
expected_output="Python is a high-level programming language.",
metrics=["answer_relevancy", "faithfulness"]
)
print(result.score)
Documentation
Full documentation is available at library.eval-ai.com.
License
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
Citation
If you use this library in your research, please cite:
@software{eval_ai_library,
author = {Meshkov, Aleksandr},
title = {Eval AI Library: Comprehensive AI Model Evaluation Framework},
year = {2025},
url = {https://github.com/meshkovQA/Eval-ai-library.git}
}
References
This library implements techniques from:
@misc{meshkov2026adaptiverigoraievaluation,
title={Adaptive Rigor in AI System Evaluation using Temperature-Controlled Verdict Aggregation via Generalized Power Mean},
author={Aleksandr Meshkov},
year={2026},
eprint={2604.08595},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2604.08595},
}
@inproceedings{liu2023geval,
title={G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment},
author={Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang},
booktitle={Proceedings of EMNLP},
year={2023}
}
Support
- Issues: GitHub Issues
- Documentation: library.eval-ai.com
1 maintainer
Metadata
1 maintainer
Release files for eval-ai-library 0.8.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| eval_ai_library-0.8.0.tar.gz | 533.3 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| eval_ai_library-0.8.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 967.4 kB
Release files / eval_ai_library-0.8.0.tar.gz
| Download URL | eval_ai_library-0.8.0.tar.gz |
|---|---|
| Size | 533.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3d8bc38db60877557fef10d46fc4f7a1b16f6e542afbd52ecef799eca3f228f5
|
|
BLAKE2b-256 checksum How to use checksums |
2e093016c0ec8517adb5b2f1150e0947ef72cc2d874bcc6e188f665cb3580d95
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.7
|
Release files / eval_ai_library-0.8.0-py3-none-any.whl
| Download URL | eval_ai_library-0.8.0-py3-none-any.whl |
|---|---|
| Size | 434.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b32e25cd6d4f037e9632f1f7718a4e6dd8961e4641c5fab38c86bf2716d0b034
|
|
BLAKE2b-256 checksum How to use checksums |
38826ee0990f128bde7cd34fa45c79e5b407a100927b193c9a88f341561836a2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.7
|
Security for eval-ai-library 0.8.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
