Eval AI Library
Based on firstlinesoftware/eval-ai-library. This is an independently maintained version with additional features and PyPI distribution.
Comprehensive AI model evaluation framework for RAG systems and AI agents. Supports 35+ evaluation metrics, 30+ LLM providers (OpenAI, Anthropic, Google, Azure, AWS Bedrock, Vertex AI, Mistral, Groq, xAI, DeepSeek, Cohere, OpenRouter, Together, Fireworks, Ollama, MLX, and more via LiteLLM), built-in test data generation from documents, an interactive web dashboard for visualization and analysis, and a first-class tracing subsystem with framework integrations for LangChain, LlamaIndex, CrewAI, AutoGen, Haystack, Semantic Kernel, Claude Agent SDK, smolagents, phidata, and OpenAI Assistants (plus an OpenTelemetry exporter). Implements advanced techniques including G-Eval probability-weighted scoring and Temperature-Controlled Verdict Aggregation via Generalized Power Mean.
Installation
pip install eval-ai-library
Full version with document parsing and OCR support:
pip install eval-ai-library[full]
Lite version (core evaluation only):
pip install eval-ai-library[lite]
Quick Start
from eval_lib import EvalAI
evaluator = EvalAI(model="gpt-4o")
result = evaluator.evaluate(
input="What is Python?",
actual_output="Python is a programming language.",
expected_output="Python is a high-level programming language.",
metrics=["answer_relevancy", "faithfulness"]
)
print(result.score)
Documentation
Full documentation is available at library.eval-ai.com.
License
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
Citation
If you use this library in your research, please cite:
@software{eval_ai_library,
author = {Meshkov, Aleksandr},
title = {Eval AI Library: Comprehensive AI Model Evaluation Framework},
year = {2025},
url = {https://github.com/meshkovQA/Eval-ai-library.git}
}
References
This library implements techniques from:
@inproceedings{liu2023geval,
title={G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment},
author={Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang},
booktitle={Proceedings of EMNLP},
year={2023}
}
Support
- Issues: GitHub Issues
- Documentation: library.eval-ai.com
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file eval_ai_library-0.7.19.tar.gz.
File metadata
- Download URL: eval_ai_library-0.7.19.tar.gz
- Upload date:
- Size: 287.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.12.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
8fdb498c37c13bd6643b22d3855895df0b198b63cd3449d1f8461fe65aa459ff
|
|
| MD5 |
2f201e59ea1d8ff50f31ea93ba1c30ac
|
|
| BLAKE2b-256 |
1aeada28f8b9144bd0692abe67c09342562bca1c3c660ed8c23533b64303ae05
|
File details
Details for the file eval_ai_library-0.7.19-py3-none-any.whl.
File metadata
- Download URL: eval_ai_library-0.7.19-py3-none-any.whl
- Upload date:
- Size: 293.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.12.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
41f2b1d3fca168b66c261fccd310f975c6e18817e384eaa087d6590954145f1e
|
|
| MD5 |
3acab3daea6926a06f5b909b20e527c2
|
|
| BLAKE2b-256 |
f9ef2ed0c301ee4d957aab8abf50e9ec03eefe415e2e109987c8b446ecbb6ccc
|
