FlexEval
Flexible evaluation tool for language models. Easy to extend, highly customizable!
English | 日本語 |
With FlexEval, you can evaluate language models with:
- Zero/few-shot in-context learning tasks
- Open-ended text-generation benchmarks such as MT-Bench with automatic evaluation using GPT-4
- Log-probability-based multiple-choice tasks
- Computing perplexity of text data
For more use cases, see the documentation.
Key Features
- Flexibility:
flexevalis flexible in terms of the evaluation setup and the language model to be evaluated. - Modularity: The core components of
flexevalare easily extensible and replaceable. - Clarity: The results of evaluation are clear and all the details are saved.
- Reproducibility:
flexevalshould be reproducible, with the ability to save and load configurations and results.
Installation
pip install flexeval
Quick Start
The following minimal example evaluates the hugging face model sbintuitions/sarashina2.2-0.5b with the commonsense_qa task.
Run Command
flexeval_lm \
--language_model HuggingFaceLM \
--language_model.model "sbintuitions/sarashina2.2-0.5b" \
--eval_setup "commonsense_qa" \
--save_dir "results/commonsense_qa"
Output
...
2025-09-03 16:22:58.434 | INFO | flexeval.core.evaluate_generation:evaluate_generation:92 - {'exact_match': 0.3185913185913186, 'finish_reason_ratio-stop': 1.0, 'avg_output_length': 9.095004095004095, 'max_output_length': 69, 'min_output_length': 2}
...
The results saved in --saved_dir contain:
config.json: The configuration of the evaluation, which can be used to replicate the evaluation.metrics.json: The evaluation metrics.outputs.jsonl: The outputs of the language model that comes with instance-level metrics.
You can flexibly customize the evaluation by specifying command-line arguments or configuration files. Besides the Transformers model, you can also evaluate models via OpenAI ChatGPT and vLLM, and other models can be readily added!
Next Steps
- Run
flexeval_presetsto check the list of off-the-shelf presets in addition tocommonsense_qa. You can find the details in the Preset Configs section. - See Getting Started to check the tutorial examples for other kinds of tasks.
- See the Configuration Guide to set up your evaluation.
1 maintainer
Metadata
1 maintainer
Release files for flexeval 0.18.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| flexeval-0.18.0.tar.gz | 304.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| flexeval-0.18.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 707.3 kB
Release files / flexeval-0.18.0.tar.gz
| Download URL | flexeval-0.18.0.tar.gz |
|---|---|
| Size | 304.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
395fe6a40287cf1db43a78af81013af41e677d30f7efb800df205806fa66a29a
|
|
BLAKE2b-256 checksum How to use checksums |
548f42e50bdc2ebbef07a8de34f948e5f8000b359e1cb7349bb6bf9be5715d5f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.1 CPython/3.10.21 Linux/6.17.0-1022-azure
|
Release files / flexeval-0.18.0-py3-none-any.whl
| Download URL | flexeval-0.18.0-py3-none-any.whl |
|---|---|
| Size | 403.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b272debaea482947ee18db631b53418af09fc7033cad52603e9ecf17197d8796
|
|
BLAKE2b-256 checksum How to use checksums |
1e200365b418c30c537593a232ad3a202238da032e548d7b5a59951d9bcf9004
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.1 CPython/3.10.21 Linux/6.17.0-1022-azure
|
