The robust European language model benchmark
(formerly known as ScandEval)
Maintainer
- Dan Saattrup Smart (@saattrupdan, dan.smart@alexandra.dk)
Installation and usage
See the documentation for more information.
Reproducing the evaluation datasets
All datasets used in this project are generated using the scripts located in the src/scripts/dataset_creation folder. To reproduce a dataset, run the corresponding script with the following command
uv run src/scripts/dataset_creation/<name-of-script>.py
Replace with the specific script you wish to execute, e.g.,
uv run src/scripts/dataset_creation/create_allocine.py
Contributors 🙏
A huge thank you to all the contributors who have helped make this project a success!
Contribute to EuroEval
We welcome contributions to EuroEval! Whether you're fixing bugs, adding features, or contributing new datasets, your help makes this project better for everyone.
- General contributions: Check out our contribution guidelines for information on how to get started.
- Volunteer GPU workers: See the volunteer worker guide to run or operate a community evaluation worker.
- Adding datasets: If you're interested in adding a new dataset to EuroEval, we have a dedicated guide with step-by-step instructions.
Special thanks
- Thanks to Google for sponsoring Gemini credits as part of their Google Cloud for Researchers Program.
- Thanks @Mikeriess for evaluating many of the larger models on the leaderboards.
- Thanks to OpenAI for sponsoring OpenAI credits as part of their Researcher Access Program.
- Thanks to UWV and KU Leuven for sponsoring the Azure OpenAI credits used to evaluate GPT-4-turbo in Dutch.
- Thanks to Miðeind for sponsoring the OpenAI credits used to evaluate GPT-4-turbo in Icelandic and Faroese.
- Thanks to CHC for sponsoring the OpenAI credits used to evaluate GPT-4-turbo in German.
Citing EuroEval
If you want to cite the framework then feel free to use this:
@inproceedings{smart2025encoder,
title={Encoder vs decoder: Comparative analysis of encoder and decoder language models on multilingual NLU tasks},
author={Smart, Dan Saattrup and Enevoldsen, Kenneth and Schneider-Kamp, Peter},
booktitle={Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025)},
pages={561--572},
year={2025}
}
@inproceedings{smart2023scandeval,
author = {Smart, Dan Saattrup},
booktitle = {Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)},
month = may,
pages = {185--201},
title = {{ScandEval: A Benchmark for Scandinavian Natural Language Processing}},
year = {2023}
}
Metadata
Release files for ScandEval 18.4.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| scandeval-18.4.0.tar.gz | 3.5 MB | Without provenance | Details for scandeval-18.4.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| scandeval-18.4.0-py3-none-any.whl | Python 3 | none | any | Details for scandeval-18.4.0-py3-none-any.whl |
Total release size: 3.6 MB
Release files / scandeval-18.4.0.tar.gz
| Download URL | scandeval-18.4.0.tar.gz |
|---|---|
| Size | 3.5 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f4fdf296cd18af6aa54626e6668b0696dab5ac98723ae711f9e6e09733fe0d7b
|
|
BLAKE2b-256 checksum How to use checksums |
deccfff156adee048379173a5e0249ed150d2976a1cdaf93424656067509a5aa
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / scandeval-18.4.0-py3-none-any.whl
| Download URL | scandeval-18.4.0-py3-none-any.whl |
|---|---|
| Size | 64.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f44f9345b7b5df4fc6e12ff704cc403aed9709265cfc88d08e5d2e3a6ad6d155
|
|
BLAKE2b-256 checksum How to use checksums |
19a66ac94915c40c397db20216bf9fa3929ae16f5d43407bc51c72dfdebf320a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for ScandEval 18.4.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.