The robust European language model benchmark
(formerly known as ScandEval)
Maintainer
- Dan Saattrup Smart (@saattrupdan, dan.smart@alexandra.dk)
Installation and usage
See the documentation for more information.
Reproducing the evaluation datasets
All datasets used in this project are generated using the scripts located in the src/scripts/dataset_creation folder. To reproduce a dataset, run the corresponding script with the following command
uv run src/scripts/dataset_creation/<name-of-script>.py
Replace with the specific script you wish to execute, e.g.,
uv run src/scripts/dataset_creation/create_allocine.py
Contributors 🙏
A huge thank you to all the contributors who have helped make this project a success!
Contribute to EuroEval
We welcome contributions to EuroEval! Whether you're fixing bugs, adding features, or contributing new datasets, your help makes this project better for everyone.
- General contributions: Check out our contribution guidelines for information on how to get started.
- Volunteer GPU workers: See the volunteer worker guide to run or operate a community evaluation worker.
- Adding datasets: If you're interested in adding a new dataset to EuroEval, we have a dedicated guide with step-by-step instructions.
Special thanks
- Thanks to Google for sponsoring Gemini credits as part of their Google Cloud for Researchers Program.
- Thanks @Mikeriess for evaluating many of the larger models on the leaderboards.
- Thanks to OpenAI for sponsoring OpenAI credits as part of their Researcher Access Program.
- Thanks to UWV and KU Leuven for sponsoring the Azure OpenAI credits used to evaluate GPT-4-turbo in Dutch.
- Thanks to Miðeind for sponsoring the OpenAI credits used to evaluate GPT-4-turbo in Icelandic and Faroese.
- Thanks to CHC for sponsoring the OpenAI credits used to evaluate GPT-4-turbo in German.
Citing EuroEval
If you want to cite the framework then feel free to use this:
@inproceedings{smart2025encoder,
title={Encoder vs decoder: Comparative analysis of encoder and decoder language models on multilingual NLU tasks},
author={Smart, Dan Saattrup and Enevoldsen, Kenneth and Schneider-Kamp, Peter},
booktitle={Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025)},
pages={561--572},
year={2025}
}
@inproceedings{smart2023scandeval,
author = {Smart, Dan Saattrup},
booktitle = {Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)},
month = may,
pages = {185--201},
title = {{ScandEval: A Benchmark for Scandinavian Natural Language Processing}},
year = {2023}
}
1 maintainer
Metadata
1 maintainer
Release files for ScandEval 18.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scandeval-18.3.0.tar.gz | 12.1 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scandeval-18.3.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 12.2 MB
Release files / scandeval-18.3.0.tar.gz
| Download URL | scandeval-18.3.0.tar.gz |
|---|---|
| Size | 12.1 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7288dd327a17348620a5cb033b4b2ae093536140a929a550420ddba4dd729d0b
|
|
BLAKE2b-256 checksum How to use checksums |
222a2f5b96148571f7e8b0f621e744b6989adcf90d32cee4ed5ad84f8c98fb74
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / scandeval-18.3.0-py3-none-any.whl
| Download URL | scandeval-18.3.0-py3-none-any.whl |
|---|---|
| Size | 54.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7dc90705f4267cc1fe0556347cd847fce00d59cf1b6b83243526b09e11850562
|
|
BLAKE2b-256 checksum How to use checksums |
c8a89d6b65a91d1c5eeab94a5610bbe5acaeedee14fce9a9adf8fe8ad6af5f58
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
