Skip to main content

FlexEval

logo

Flexible evaluation tool for language models. Easy to extend, highly customizable!

English | 日本語 |

With FlexEval, you can evaluate language models with:

  • Zero/few-shot in-context learning tasks
  • Open-ended text-generation benchmarks such as MT-Bench with automatic evaluation using GPT-4
  • Log-probability-based multiple-choice tasks
  • Computing perplexity of text data

For more use cases, see the documentation.

Key Features

  • Flexibility: flexeval is flexible in terms of the evaluation setup and the language model to be evaluated.
  • Modularity: The core components of flexeval are easily extensible and replaceable.
  • Clarity: The results of evaluation are clear and all the details are saved.
  • Reproducibility: flexeval should be reproducible, with the ability to save and load configurations and results.

Installation

pip install flexeval

Quick Start

The following minimal example evaluates the hugging face model sbintuitions/sarashina2.2-0.5b with the commonsense_qa task.

Run Command

flexeval_lm \
  --language_model HuggingFaceLM \
  --language_model.model "sbintuitions/sarashina2.2-0.5b" \
  --eval_setup "commonsense_qa" \
  --save_dir "results/commonsense_qa"

Output

...
2025-09-03 16:22:58.434 | INFO     | flexeval.core.evaluate_generation:evaluate_generation:92 - {'exact_match': 0.3185913185913186, 'finish_reason_ratio-stop': 1.0, 'avg_output_length': 9.095004095004095, 'max_output_length': 69, 'min_output_length': 2}
...

The results saved in --saved_dir contain:

  • config.json: The configuration of the evaluation, which can be used to replicate the evaluation.
  • metrics.json: The evaluation metrics.
  • outputs.jsonl: The outputs of the language model that comes with instance-level metrics.

You can flexibly customize the evaluation by specifying command-line arguments or configuration files. Besides the Transformers model, you can also evaluate models via OpenAI ChatGPT and vLLM, and other models can be readily added!

Next Steps

  • Run flexeval_presets to check the list of off-the-shelf presets in addition to commonsense_qa. You can find the details in the Preset Configs section.
  • See Getting Started to check the tutorial examples for other kinds of tasks.
  • See the Configuration Guide to set up your evaluation.

Metadata

Release files for flexeval 0.18.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for flexeval 0.18.4
File Size Uploaded Provenance Actions
flexeval-0.18.4.tar.gz 322.8 kB Without provenance Details for flexeval-0.18.4.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for flexeval 0.18.4
File Interpreter ABI Platform Actions
flexeval-0.18.4-py3-none-any.whl Python 3 none any Details for flexeval-0.18.4-py3-none-any.whl

Total release size: 756.6 kB

Release files / flexeval-0.18.4.tar.gz

Download URL flexeval-0.18.4.tar.gz
Size 322.8 kB
Tags Source
SHA-256 checksum
How to use checksums
24a3f187cc4add2d48fc814b508fc96cd9aaaa611dffbb970cf66520c731cf2f
BLAKE2b-256 checksum
How to use checksums
91a9ab824a307004d7768a4eef3d785a392d06e74c14524e246cbc63d97f10b0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.5.1 CPython/3.12.16 Linux/7.0.0-1019-azure

Release files / flexeval-0.18.4-py3-none-any.whl

Download URL flexeval-0.18.4-py3-none-any.whl
Size 433.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
22a0bca0d370944ece0f7eba11281bec053f6981674a98aec5b7ec734f1c5c10
BLAKE2b-256 checksum
How to use checksums
56f2b40a88f8dd2e2ef6557810327310aecc81638882ed93347e86bf92aa6fdd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.5.1 CPython/3.12.16 Linux/7.0.0-1019-azure

Security for flexeval 0.18.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.18.4 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page