FlexEval
Flexible evaluation tool for language models. Easy to extend, highly customizable!
English | 日本語 |
With FlexEval, you can evaluate language models with:
- Zero/few-shot in-context learning tasks
- Open-ended text-generation benchmarks such as MT-Bench with automatic evaluation using GPT-4
- Log-probability-based multiple-choice tasks
- Computing perplexity of text data
For more use cases, see the documentation.
Key Features
- Flexibility:
flexevalis flexible in terms of the evaluation setup and the language model to be evaluated. - Modularity: The core components of
flexevalare easily extensible and replaceable. - Clarity: The results of evaluation are clear and all the details are saved.
- Reproducibility:
flexevalshould be reproducible, with the ability to save and load configurations and results.
Installation
pip install flexeval
Quick Start
The following minimal example evaluates the hugging face model sbintuitions/sarashina2.2-0.5b with the commonsense_qa task.
Run Command
flexeval_lm \
--language_model HuggingFaceLM \
--language_model.model "sbintuitions/sarashina2.2-0.5b" \
--eval_setup "commonsense_qa" \
--save_dir "results/commonsense_qa"
Output
...
2025-09-03 16:22:58.434 | INFO | flexeval.core.evaluate_generation:evaluate_generation:92 - {'exact_match': 0.3185913185913186, 'finish_reason_ratio-stop': 1.0, 'avg_output_length': 9.095004095004095, 'max_output_length': 69, 'min_output_length': 2}
...
The results saved in --saved_dir contain:
config.json: The configuration of the evaluation, which can be used to replicate the evaluation.metrics.json: The evaluation metrics.outputs.jsonl: The outputs of the language model that comes with instance-level metrics.
You can flexibly customize the evaluation by specifying command-line arguments or configuration files. Besides the Transformers model, you can also evaluate models via OpenAI ChatGPT and vLLM, and other models can be readily added!
Next Steps
- Run
flexeval_presetsto check the list of off-the-shelf presets in addition tocommonsense_qa. You can find the details in the Preset Configs section. - See Getting Started to check the tutorial examples for other kinds of tasks.
- See the Configuration Guide to set up your evaluation.
1 maintainer
Metadata
1 maintainer
Release files for flexeval 0.18.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| flexeval-0.18.2.tar.gz | 316.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| flexeval-0.18.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 743.9 kB
Release files / flexeval-0.18.2.tar.gz
| Download URL | flexeval-0.18.2.tar.gz |
|---|---|
| Size | 316.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d09a5b76cb14557df0cf170e55528d6f945ff7c7d0486fe9fa6cdf9bfeec53ac
|
|
BLAKE2b-256 checksum How to use checksums |
d74c78aeba75cc6a52f797bd9eea369b6423c198ad9fefbeac059b62f88a3c8f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.5.1 CPython/3.10.22 Linux/6.17.0-1022-azure
|
Release files / flexeval-0.18.2-py3-none-any.whl
| Download URL | flexeval-0.18.2-py3-none-any.whl |
|---|---|
| Size | 427.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f22e213ece072953b90105e17a1ebbe20bb636399dad5b687b786e60d6f2f5e9
|
|
BLAKE2b-256 checksum How to use checksums |
7231a44deab697711cf9db7523c1e7f40ddab9011e3e7b8407333105ff4f3847
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.5.1 CPython/3.10.22 Linux/6.17.0-1022-azure
|
