Eval AI Library
Based on firstlinesoftware/eval-ai-library. This is an independently maintained version with additional features and PyPI distribution.
Comprehensive AI model evaluation framework for RAG systems and AI agents. Supports 35+ evaluation metrics, 30+ LLM providers (OpenAI, Anthropic, Google, Azure, AWS Bedrock, Vertex AI, Mistral, Groq, xAI, DeepSeek, Cohere, OpenRouter, Together, Fireworks, Ollama, MLX, and more via LiteLLM), built-in test data generation from documents, an interactive web dashboard for visualization and analysis, and a first-class tracing subsystem with framework integrations for LangChain, LlamaIndex, CrewAI, AutoGen, Haystack, Semantic Kernel, Claude Agent SDK, smolagents, phidata, and OpenAI Assistants (plus an OpenTelemetry exporter). Implements advanced techniques including G-Eval probability-weighted scoring and Temperature-Controlled Verdict Aggregation via Generalized Power Mean.
Installation
pip install eval-ai-library
Full version with document parsing and OCR support:
pip install eval-ai-library[full]
Lite version (core evaluation only):
pip install eval-ai-library[lite]
Quick Start
from eval_lib import EvalAI
evaluator = EvalAI(model="gpt-4o")
result = evaluator.evaluate(
input="What is Python?",
actual_output="Python is a programming language.",
expected_output="Python is a high-level programming language.",
metrics=["answer_relevancy", "faithfulness"]
)
print(result.score)
Documentation
Full documentation is available at library.eval-ai.com.
License
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
Citation
If you use this library in your research, please cite:
@software{eval_ai_library,
author = {Meshkov, Aleksandr},
title = {Eval AI Library: Comprehensive AI Model Evaluation Framework},
year = {2025},
url = {https://github.com/meshkovQA/Eval-ai-library.git}
}
References
This library implements techniques from:
@inproceedings{liu2023geval,
title={G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment},
author={Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang},
booktitle={Proceedings of EMNLP},
year={2023}
}
Support
- Issues: GitHub Issues
- Documentation: library.eval-ai.com
1 maintainer
Metadata
1 maintainer
Release files for eval-ai-library 0.7.21
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| eval_ai_library-0.7.21.tar.gz | 288.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| eval_ai_library-0.7.21-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 582.9 kB
Release files / eval_ai_library-0.7.21.tar.gz
| Download URL | eval_ai_library-0.7.21.tar.gz |
|---|---|
| Size | 288.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0353f68d54a3c22f305a1ee7892e51d6e4b6ee3f80bfeccfa588ce9e4ab23125
|
|
BLAKE2b-256 checksum How to use checksums |
fa17941c5b390a8604fe1d87c6c0feadffc423145009618dcd408ec3814bc8c4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.7
|
Release files / eval_ai_library-0.7.21-py3-none-any.whl
| Download URL | eval_ai_library-0.7.21-py3-none-any.whl |
|---|---|
| Size | 294.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2d59c6c8e987ff2e27eef39dda4bd49800f2b9a9c5a32de1277f010c6a799323
|
|
BLAKE2b-256 checksum How to use checksums |
d694e666f8c4b7c4c89b71e771c550c7198c605aeedc6951949524986d1f5cae
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.7
|
