FlexEval is a tool for designing custom metrics, completion functions, and LLM-graded rubrics for evaluating the behavior of LLM-powered systems.
Verified details
These details have been verified by PyPIProject links
GitHub Statistics
Maintainers
Project description
FlexEval LLM Evals
FlexEval is a tool for designing custom metrics, completion functions, and LLM-graded rubrics for evaluating the behavior of LLM-powered systems.
Documentation: https://digitalharborfoundation.github.io/FlexEval
Additional details about FlexEval can be found in our paper at the Educational Data Mining 2024 conference.
Usage
Basic usage:
import flexeval
from flexeval.schema import Eval, EvalRun, FileDataSource, Metrics, FunctionItem, Config
data_sources = [FileDataSource(path="vignettes/conversations.jsonl")]
eval = Eval(metrics=Metrics(function=[FunctionItem(name="flesch_reading_ease")]))
config = Config(clear_tables=True)
eval_run = EvalRun(
data_sources=data_sources,
database_path="eval_results.db",
eval=eval,
config=config,
)
flexeval.run(eval_run)
This example computes Flesch reading ease for every turn in a list of conversations provided in JSONL format. The metric values are stored in an SQLite database called eval_results.db.
See additional usage examples in the vignettes.
Installation
FlexEval is on PyPI as python-flexeval. See the Installation section in the Getting Started guide.
Using pip:
pip install python-flexeval
Basic functionality
FlexEval is designed to be "batteries included" for many basic use cases. It supports the following out-of-the-box:
- scoring historical conversations - useful for monitoring live systems.
- scoring LLMs:
- locally hosted and served via an endpoint using something like LM Studio
- LLMs accessible by a REST endpoint and accessible via a network call
- any OpenAI LLM
- a set of useful rubrics
- a set of useful Python functions
Evaluation results are saved in an SQLite database. See the Metric Analysis vignette for a sample analysis demonstrating the structure and utility of the data saved by FlexEval.
Read more in the Getting Started guide.
Cite this work
If this work is useful to you, please cite our EDM 2024 paper:
S. Thomas Christie, Baptiste Moreau-Pernet, Yu Tian, & John Whitmer. (2024). FlexEval: a customizable tool for chatbot performance evaluation and dialogue analysis. Proceedings of the 17th International Conference on Educational Data Mining, 903-908. Atlanta, Georgia, USA, July 2024. https://doi.org/10.5281/zenodo.12729993
Development
Pull requests are welcome. Feel free to contribute:
- New rubrics or functions
- Bug fixes
- New features
See DEVELOPMENT.md.
Project details
Verified details
These details have been verified by PyPIProject links
GitHub Statistics
Maintainers
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file python_flexeval-0.5.2.tar.gz.
File metadata
- Download URL: python_flexeval-0.5.2.tar.gz
- Upload date:
- Size: 806.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
34bd9c9c193e50c91e8573926a7119595f212278ab19ad2e4cd2424489b55345
|
|
| MD5 |
51c89a03c522f668fe1c1d63517f08b1
|
|
| BLAKE2b-256 |
137e965f0ceee05bf3b5fa6352770855f6e03bcf853fcd1b3ff7abdce3b12f06
|
Provenance
The following attestation bundles were made for python_flexeval-0.5.2.tar.gz:
Publisher:
deploy-to-pypi.yml on DigitalHarborFoundation/FlexEval
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
python_flexeval-0.5.2.tar.gz -
Subject digest:
34bd9c9c193e50c91e8573926a7119595f212278ab19ad2e4cd2424489b55345 - Sigstore transparency entry: 1806467214
- Sigstore integration time:
-
Permalink:
DigitalHarborFoundation/FlexEval@54dacbc9c20a306a4a8fe51d293d19a39c1df838 -
Branch / Tag:
refs/tags/v0.5.2 - Owner: https://github.com/DigitalHarborFoundation
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
deploy-to-pypi.yml@54dacbc9c20a306a4a8fe51d293d19a39c1df838 -
Trigger Event:
push
-
Statement type:
File details
Details for the file python_flexeval-0.5.2-py3-none-any.whl.
File metadata
- Download URL: python_flexeval-0.5.2-py3-none-any.whl
- Upload date:
- Size: 74.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
655d1342dbab99b687c2ad384e632c508483815c3701232b7ccb252985f0f865
|
|
| MD5 |
93ca5a0bb2d258e58e0b6da6b960a6f1
|
|
| BLAKE2b-256 |
6734cec8fc1cd8e32a0874f9461339b1319d08ce854dbf7b03a37c7d63f04fc1
|
Provenance
The following attestation bundles were made for python_flexeval-0.5.2-py3-none-any.whl:
Publisher:
deploy-to-pypi.yml on DigitalHarborFoundation/FlexEval
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
python_flexeval-0.5.2-py3-none-any.whl -
Subject digest:
655d1342dbab99b687c2ad384e632c508483815c3701232b7ccb252985f0f865 - Sigstore transparency entry: 1806467234
- Sigstore integration time:
-
Permalink:
DigitalHarborFoundation/FlexEval@54dacbc9c20a306a4a8fe51d293d19a39c1df838 -
Branch / Tag:
refs/tags/v0.5.2 - Owner: https://github.com/DigitalHarborFoundation
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
deploy-to-pypi.yml@54dacbc9c20a306a4a8fe51d293d19a39c1df838 -
Trigger Event:
push
-
Statement type:
