FlexEval
Flexible evaluation tool for language models. Easy to extend, highly customizable!
English | 日本語 |
With FlexEval, you can evaluate language models with:
- Zero/few-shot in-context learning tasks
- Open-ended text-generation benchmarks such as MT-Bench with automatic evaluation using GPT-4
- Log-probability-based multiple-choice tasks
- Computing perplexity of text data
For more use cases, see the documentation.
Key Features
- Flexibility:
flexevalis flexible in terms of the evaluation setup and the language model to be evaluated. - Modularity: The core components of
flexevalare easily extensible and replaceable. - Clarity: The results of evaluation are clear and all the details are saved.
- Reproducibility:
flexevalshould be reproducible, with the ability to save and load configurations and results.
Installation
pip install flexeval
Quick Start
The following minimal example evaluates the hugging face model sbintuitions/sarashina2.2-0.5b with the commonsense_qa task.
Run Command
flexeval_lm \
--language_model HuggingFaceLM \
--language_model.model "sbintuitions/sarashina2.2-0.5b" \
--eval_setup "commonsense_qa" \
--save_dir "results/commonsense_qa"
Output
...
2025-09-03 16:22:58.434 | INFO | flexeval.core.evaluate_generation:evaluate_generation:92 - {'exact_match': 0.3185913185913186, 'finish_reason_ratio-stop': 1.0, 'avg_output_length': 9.095004095004095, 'max_output_length': 69, 'min_output_length': 2}
...
The results saved in --saved_dir contain:
config.json: The configuration of the evaluation, which can be used to replicate the evaluation.metrics.json: The evaluation metrics.outputs.jsonl: The outputs of the language model that comes with instance-level metrics.
You can flexibly customize the evaluation by specifying command-line arguments or configuration files. Besides the Transformers model, you can also evaluate models via OpenAI ChatGPT and vLLM, and other models can be readily added!
Next Steps
- Run
flexeval_presetsto check the list of off-the-shelf presets in addition tocommonsense_qa. You can find the details in the Preset Configs section. - See Getting Started to check the tutorial examples for other kinds of tasks.
- See the Configuration Guide to set up your evaluation.
1 maintainer
Metadata
1 maintainer
Release files for flexeval 0.18.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| flexeval-0.18.1.tar.gz | 311.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| flexeval-0.18.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 726.5 kB
Release files / flexeval-0.18.1.tar.gz
| Download URL | flexeval-0.18.1.tar.gz |
|---|---|
| Size | 311.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8f09416448b9627bf9ce8a957189dd3f103cfe61b87affbf2a4963b9f18041e6
|
|
BLAKE2b-256 checksum How to use checksums |
7238ded6148c64707c8203123d27f8153d7171a71e0c2d4c2d9628bccbee49e2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.3 CPython/3.10.21 Linux/6.17.0-1022-azure
|
Release files / flexeval-0.18.1-py3-none-any.whl
| Download URL | flexeval-0.18.1-py3-none-any.whl |
|---|---|
| Size | 415.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6e6c66a835522cba6e08785b71650589ebac50acb697ac8a12c12f56506410a2
|
|
BLAKE2b-256 checksum How to use checksums |
4d9af9729cd339bec58a8e86d839f55e67b006cc6ec8a1555b0547dd315dbda9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.3 CPython/3.10.21 Linux/6.17.0-1022-azure
|
