FlexEval
Flexible evaluation tool for language models. Easy to extend, highly customizable!
English | 日本語 |
With FlexEval, you can evaluate language models with:
- Zero/few-shot in-context learning tasks
- Open-ended text-generation benchmarks such as MT-Bench with automatic evaluation using GPT-4
- Log-probability-based multiple-choice tasks
- Computing perplexity of text data
For more use cases, see the documentation.
Key Features
- Flexibility:
flexevalis flexible in terms of the evaluation setup and the language model to be evaluated. - Modularity: The core components of
flexevalare easily extensible and replaceable. - Clarity: The results of evaluation are clear and all the details are saved.
- Reproducibility:
flexevalshould be reproducible, with the ability to save and load configurations and results.
Installation
pip install flexeval
Quick Start
The following minimal example evaluates the hugging face model sbintuitions/sarashina2.2-0.5b with the commonsense_qa task.
Run Command
flexeval_lm \
--language_model HuggingFaceLM \
--language_model.model "sbintuitions/sarashina2.2-0.5b" \
--eval_setup "commonsense_qa" \
--save_dir "results/commonsense_qa"
Output
...
2025-09-03 16:22:58.434 | INFO | flexeval.core.evaluate_generation:evaluate_generation:92 - {'exact_match': 0.3185913185913186, 'finish_reason_ratio-stop': 1.0, 'avg_output_length': 9.095004095004095, 'max_output_length': 69, 'min_output_length': 2}
...
The results saved in --saved_dir contain:
config.json: The configuration of the evaluation, which can be used to replicate the evaluation.metrics.json: The evaluation metrics.outputs.jsonl: The outputs of the language model that comes with instance-level metrics.
You can flexibly customize the evaluation by specifying command-line arguments or configuration files. Besides the Transformers model, you can also evaluate models via OpenAI ChatGPT and vLLM, and other models can be readily added!
Next Steps
- Run
flexeval_presetsto check the list of off-the-shelf presets in addition tocommonsense_qa. You can find the details in the Preset Configs section. - See Getting Started to check the tutorial examples for other kinds of tasks.
- See the Configuration Guide to set up your evaluation.
1 maintainer
Metadata
1 maintainer
Release files for flexeval 0.17.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| flexeval-0.17.3.tar.gz | 303.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| flexeval-0.17.3-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 706.9 kB
Release files / flexeval-0.17.3.tar.gz
| Download URL | flexeval-0.17.3.tar.gz |
|---|---|
| Size | 303.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4237d1f5416d47824f8b11f8f38ac8c4320123255679bb72c09bae52a19512d7
|
|
BLAKE2b-256 checksum How to use checksums |
1dd4c9222633c039f507140afeb6d98d78f9dbec7e5a78b25e2c32605c8dca78
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.1 CPython/3.10.20 Linux/6.17.0-1020-azure
|
Release files / flexeval-0.17.3-py3-none-any.whl
| Download URL | flexeval-0.17.3-py3-none-any.whl |
|---|---|
| Size | 403.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
cdae5a61b3f1ad4f2d076e9dc01b623c04ca705c05068c1c322fe6d2c64781e0
|
|
BLAKE2b-256 checksum How to use checksums |
c21d2830beb84d0e544b23b2ab6a1add69d5177382b57e8cd0f838d1616f62ec
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.1 CPython/3.10.20 Linux/6.17.0-1020-azure
|
