Skip to main content

The robust European language model benchmark

(formerly known as ScandEval)


Documentation PyPI Status First paper Second paper License LastCommit Code Coverage Contributor Covenant

Maintainer

Installation and usage

See the documentation for more information.

Reproducing the evaluation datasets

All datasets used in this project are generated using the scripts located in the src/scripts/dataset_creation folder. To reproduce a dataset, run the corresponding script with the following command

uv run src/scripts/dataset_creation/<name-of-script>.py

Replace with the specific script you wish to execute, e.g.,

uv run src/scripts/dataset_creation/create_allocine.py

Contributors 🙏

A huge thank you to all the contributors who have helped make this project a success!

Contributor avatar for peter-sk Contributor avatar for AJDERS Contributor avatar for oliverkinch Contributor avatar for versae Contributor avatar for KennethEnevoldsen Contributor avatar for viggo-gascou Contributor avatar for mathiasesn Contributor avatar for Alkarex Contributor avatar for marksverdhei Contributor avatar for Mikeriess Contributor avatar for ThomasKluiters Contributor avatar for BramVanroy Contributor avatar for peregilk Contributor avatar for Rijgersberg Contributor avatar for duarteocarmo Contributor avatar for slowwavesleep Contributor avatar for mrkowalski Contributor avatar for sofiehb Contributor avatar for simonevanbruggen Contributor avatar for tvosch Contributor avatar for Touzen Contributor avatar for caldaibis Contributor avatar for SwekeR-463 Contributor avatar for N-essuno Contributor avatar for harderj Contributor avatar for lardinator Contributor avatar for lswiers Contributor avatar for avalyset Contributor avatar for pariidanDKE Contributor avatar for Biorrith Contributor avatar for FrejaThoresen Contributor avatar for rlrs Contributor avatar for jaideeppyne Contributor avatar for milos-plavsic Contributor avatar for Mr-Neutr0n Contributor avatar for djstrong Contributor avatar for SushantGautam

Contribute to EuroEval

We welcome contributions to EuroEval! Whether you're fixing bugs, adding features, or contributing new datasets, your help makes this project better for everyone.

  • General contributions: Check out our contribution guidelines for information on how to get started.
  • Volunteer GPU workers: See the volunteer worker guide to run or operate a community evaluation worker.
  • Adding datasets: If you're interested in adding a new dataset to EuroEval, we have a dedicated guide with step-by-step instructions.

Special thanks

  • Thanks to Google for sponsoring Gemini credits as part of their Google Cloud for Researchers Program.
  • Thanks @Mikeriess for evaluating many of the larger models on the leaderboards.
  • Thanks to OpenAI for sponsoring OpenAI credits as part of their Researcher Access Program.
  • Thanks to UWV and KU Leuven for sponsoring the Azure OpenAI credits used to evaluate GPT-4-turbo in Dutch.
  • Thanks to Miðeind for sponsoring the OpenAI credits used to evaluate GPT-4-turbo in Icelandic and Faroese.
  • Thanks to CHC for sponsoring the OpenAI credits used to evaluate GPT-4-turbo in German.

Citing EuroEval

If you want to cite the framework then feel free to use this:

@inproceedings{smart2025encoder,
  title={Encoder vs decoder: Comparative analysis of encoder and decoder language models on multilingual NLU tasks},
  author={Smart, Dan Saattrup and Enevoldsen, Kenneth and Schneider-Kamp, Peter},
  booktitle={Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025)},
  pages={561--572},
  year={2025}
}

@inproceedings{smart2023scandeval,
  author = {Smart, Dan Saattrup},
  booktitle = {Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)},
  month = may,
  pages = {185--201},
  title = {{ScandEval: A Benchmark for Scandinavian Natural Language Processing}},
  year = {2023}
}

Metadata

Release files for EuroEval 18.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for EuroEval 18.4.0
File Size Uploaded Provenance Actions
euroeval-18.4.0.tar.gz 3.5 MB Without provenance Details for euroeval-18.4.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for EuroEval 18.4.0
File Interpreter ABI Platform Actions
euroeval-18.4.0-py3-none-any.whl Python 3 none any Details for euroeval-18.4.0-py3-none-any.whl

Total release size: 4.1 MB

Release files / euroeval-18.4.0.tar.gz

Download URL euroeval-18.4.0.tar.gz
Size 3.5 MB
Tags Source
SHA-256 checksum
How to use checksums
2c472028a839a4316f4f4e566b3c64f0b22624704b0e07f3451eef4defe4855f
BLAKE2b-256 checksum
How to use checksums
9e0599cd647eb8623eabe5f08065e4673482ca6a72da8412e3383adbf2d81be3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / euroeval-18.4.0-py3-none-any.whl

Download URL euroeval-18.4.0-py3-none-any.whl
Size 529.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f9d593632756e41e053e0caf17baff35e46911c22316cb4922f8ffb6b95f800e
BLAKE2b-256 checksum
How to use checksums
e156fb2624f85bf0b910c0a8f711ae73ba33c841c7237bf65a6da36a9a74c60c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for EuroEval 18.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

18.4.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page