MTEB MTEB

Multimodal toolbox for evaluating embeddings and retrieval systems

GitHub release License Downloads

Installation | Usage | Leaderboard | Documentation | Citing

Installation

You can install mteb simply using pip or uv. For more on installation please see the documentation.

pip install mteb

For faster installation, you can also use uv:

uv add mteb

Example Usage

Below we present a simple use-case example. For more information, see the documentation.

import mteb
from sentence_transformers import SentenceTransformer

# Select model
model_name = "sentence-transformers/all-MiniLM-L6-v2"
model = mteb.get_model(
    model_name
)  # if the model is not implemented in MTEB it will be eq. to SentenceTransformer(model_name)

# Select tasks
tasks = mteb.get_tasks(tasks=["Banking77Classification.v2"])

# evaluate
results = mteb.evaluate(model, tasks=tasks)

You can also run it using the CLI:

mteb run \
    -m sentence-transformers/all-MiniLM-L6-v2 \
    -t "Banking77Classification.v2" \
    --output-folder results

For more on how to use the CLI check out the related documentation.

Overview

Overview
📈 Leaderboard The interactive leaderboard of the benchmark
Get Started.
🏃 Get Started Overview of how to use mteb
🤖 Defining Models How to use existing models and define custom ones
📋 Selecting tasks How to select tasks, benchmarks, splits etc.
🏭 Running Evaluation How to run the evaluations, including cache management, speeding up evaluations etc.
📊 Loading Results How to load and work with existing model results
Overview.
📋 Tasks Overview of available tasks
📐 Benchmarks Overview of available benchmarks
🤖 Models Overview of available Models
Contributing
🤖 Adding a model How to submit a model to MTEB and to the leaderboard
👩‍💻 Adding a dataset How to add a new task/dataset to MTEB
👩‍💻 Adding a benchmark How to add a new benchmark to MTEB and to the leaderboard
🤝 Contributing How to contribute to MTEB and set it up for development

Citing

MTEB was introduced in "MTEB: Massive Text Embedding Benchmark", and heavily expanded in "MMTEB: Massive Multilingual Text Embedding Benchmark". When using mteb, we recommend that you cite both articles.

Bibtex Citation (click to unfold) 
@article{muennighoff2022mteb,
  author = {Muennighoff, Niklas and Tazi, Nouamane and Magne, Loïc and Reimers, Nils},
  title = {MTEB: Massive Text Embedding Benchmark},
  publisher = {arXiv},
  journal={arXiv preprint arXiv:2210.07316},
  year = {2022},
  url = {https://arxiv.org/abs/2210.07316},
  doi = {10.48550/ARXIV.2210.07316},
}

@article{enevoldsen2025mmtebmassivemultilingualtext,
  title={MMTEB: Massive Multilingual Text Embedding Benchmark},
  author={Kenneth Enevoldsen and Isaac Chung and Imene Kerboua and Márton Kardos and Ashwin Mathur and David Stap and Jay Gala and Wissam Siblini and Dominik Krzemiński and Genta Indra Winata and Saba Sturua and Saiteja Utpala and Mathieu Ciancone and Marion Schaeffer and Gabriel Sequeira and Diganta Misra and Shreeya Dhakal and Jonathan Rystrøm and Roman Solomatin and Ömer Çağatan and Akash Kundu and Martin Bernstorff and Shitao Xiao and Akshita Sukhlecha and Bhavish Pahwa and Rafał Poświata and Kranthi Kiran GV and Shawon Ashraf and Daniel Auras and Björn Plüster and Jan Philipp Harries and Loïc Magne and Isabelle Mohr and Mariya Hendriksen and Dawei Zhu and Hippolyte Gisserot-Boukhlef and Tom Aarsen and Jan Kostkan and Konrad Wojtasik and Taemin Lee and Marek Šuppa and Crystina Zhang and Roberta Rocca and Mohammed Hamdy and Andrianos Michail and John Yang and Manuel Faysse and Aleksei Vatolin and Nandan Thakur and Manan Dey and Dipam Vasani and Pranjal Chitale and Simone Tedeschi and Nguyen Tai and Artem Snegirev and Michael Günther and Mengzhou Xia and Weijia Shi and Xing Han Lù and Jordan Clive and Gayatri Krishnakumar and Anna Maksimova and Silvan Wehrli and Maria Tikhonova and Henil Panchal and Aleksandr Abramov and Malte Ostendorff and Zheng Liu and Simon Clematide and Lester James Miranda and Alena Fenogenova and Guangyu Song and Ruqiya Bin Safi and Wen-Ding Li and Alessia Borghini and Federico Cassano and Hongjin Su and Jimmy Lin and Howard Yen and Lasse Hansen and Sara Hooker and Chenghao Xiao and Vaibhav Adlakha and Orion Weller and Siva Reddy and Niklas Muennighoff},
  publisher = {arXiv},
  journal={arXiv preprint arXiv:2502.13595},
  year={2025},
  url={https://arxiv.org/abs/2502.13595},
  doi = {10.48550/arXiv.2502.13595},
}

If you use any of the specific benchmarks, we also recommend that you cite the authors of both the benchmark and its tasks:

benchmark = mteb.get_benchmark("MTEB(eng, v2)")
benchmark.citation  # get citation for a specific benchmark

# you can also create a table of the task for the appendix using:
benchmark.tasks.to_latex()

Project links

Data verified by PyPI on Jul 30, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (2.18.16)

4 maintainers

PyPI data
Data sourced directly from PyPI's database.
Avatar for KennethEnevoldsen from gravatar.com KennethEnevoldsen Avatar for Muennighoff from gravatar.com Muennighoff Avatar for nouamanetazi from gravatar.com nouamanetazi
+1 more Avatar for nreimers from gravatar.com nreimers

Credits

Author: MTEB Contributors

Maintainer: Kenneth Enevoldsen

GitHub Statistics

Data verified by PyPI on Jul 30, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

Apache-2.0
View SPDX License List

Requires

Python <3.15, >=3.10

Provides Extra

image audio codecarbon leaderboard api og video github peft flagembedding jina jina-clip jina-v4 flash-attention colvec1-1 openai model2vec pylate msclap bm25s gritlm xformers blip2 voyageai voyage-v cohere vertexai google-genai octen llm2vec timm open-clip-torch nomic ark colpali-engine colqwen3 visrag-ret sauerkrautlm-colpali xet youtu llama-embed-nemotron llama-nemotron-colembed-vl nemotron-colembed-vl-v2 llama-nemotron-embed-vl-1b-v2 nemotron-3-embed multimodal-sbert faiss-cpu eager-embed speechbrain muq wav2clip fusion-embedding torch-vggish-yamnet vllm transformers-v4 siglip qwen-vl qwen-omni-utils embeddinggemma

Tags

deep learning text embeddings embeddings multimodal benchmark retrieval information retrieval

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mteb-2.18.10.tar.gz (3.9 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

mteb-2.18.10-py3-none-any.whl (6.1 MB view details)

Uploaded Python 3

File details

Details for the file mteb-2.18.10.tar.gz.

File metadata

  • Download URL: mteb-2.18.10.tar.gz
  • Upload date:
  • Size: 3.9 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for mteb-2.18.10.tar.gz
Algorithm Hash digest
SHA256 c61f03381c8a3fb86e86ba19dd870e36c5f6419ba6e12d83ecdb4804221badb9
MD5 4c069ad1a73e7b2035ee14826d86e680
BLAKE2b-256 bdefdc598abc805b6a5a9b803cf277e68f3c9749e8de50f9065345dca8b75233

See more details on using hashes here.

Provenance

The following attestation bundles were made for mteb-2.18.10.tar.gz:

Publisher: release.yml on embeddings-benchmark/mteb

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file mteb-2.18.10-py3-none-any.whl.

File metadata

  • Download URL: mteb-2.18.10-py3-none-any.whl
  • Upload date:
  • Size: 6.1 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for mteb-2.18.10-py3-none-any.whl
Algorithm Hash digest
SHA256 83dbc213447bf8b8c56553d6df2c7c6ef14d6649def465a55f5dfa9c83fbc73d
MD5 97bb63240eca34a62cbc5277ba87dd2a
BLAKE2b-256 dd7f0f6a40f05d090cb08fdff452e37dc3ec8a99ab1cc88526924e58f23cc820

See more details on using hashes here.

Provenance

The following attestation bundles were made for mteb-2.18.10-py3-none-any.whl:

Publisher: release.yml on embeddings-benchmark/mteb

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

2.18.16

2.18.15

2.18.14

2.18.13

2.18.12

2.18.11

This release

2.18.10

2.18.9

2.18.8

2.18.7

2.18.6

2.18.5

2.18.4

2.18.3

2.18.2

2.18.1

2.18.0

2.17.0

2.16.3

2.16.2

2.16.1

2.16.0

2.15.6

2.15.5

2.15.4

2.15.3

2.15.2

2.15.1

2.15.0

2.14.9

2.14.8

2.14.7

2.14.6

2.14.5

2.12.30

2.12.29

2.12.28

2.12.27

2.12.26

2.12.25

2.12.24

2.12.23

2.12.22

2.12.21

2.12.20

2.12.19

2.12.18

2.12.17

2.12.16

2.12.15

2.12.14

2.12.13

2.12.12

2.12.11

2.12.10

2.12.9

2.12.8

2.12.7

2.12.6

2.12.5

2.12.4

2.12.3

2.12.2

2.12.1

2.12.0

2.11.10

2.11.9

2.11.8

2.11.7

2.11.6

2.11.5

2.11.4

2.11.3

2.11.2

2.11.1

2.11.0

2.10.14

2.10.13

2.10.12

2.10.11

2.10.10

2.10.9

2.10.8

2.10.7

2.10.6

2.10.5

2.10.4

2.10.3

2.10.2

2.10.1

2.10.0

2.9.1

2.9.0

2.8.10

2.8.9

2.8.8

2.8.7

2.8.6

2.8.5

2.8.4

2.8.3

2.8.2

2.8.1

2.7.30

2.7.29

2.7.28

2.7.27

2.7.26

2.7.25

2.7.24

2.7.23

2.7.22

2.7.21

2.7.20

2.7.19

2.7.18

2.7.17

2.7.16

2.7.15

2.7.14

2.7.13

2.7.12

2.7.11

2.7.10

2.7.9

2.7.8

2.7.7

2.7.6

2.7.5

2.7.4

2.7.3

2.7.2

2.7.1

2.7.0

2.6.9

2.6.8

2.6.7

2.6.6

2.6.5

2.6.4

2.6.3

2.6.2

2.6.1

2.6.0

2.5.5

2.5.4

2.5.3

2.5.2

2.5.1

2.5.0

2.4.2

2.4.1

2.3.11

2.3.10

2.3.9

2.3.8

2.3.7

2.3.6

2.3.5

2.3.4

2.3.3

2.3.2

2.3.1

2.3.0

2.2.2

2.2.1

2.2.0

2.1.19

2.1.18

2.1.17

2.1.16

2.1.15

2.1.14

2.1.13

2.1.12

2.1.11

2.1.10

2.1.9

2.1.8

2.1.7

2.1.6

2.1.5

2.1.4

2.1.3

2.1.2

2.1.1

2.1.0

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

1.39.7

1.39.6

1.39.5

1.39.4

1.39.3

1.39.2

1.39.1

1.39.0

1.38.61

1.38.60

1.38.59

1.38.58

1.38.57

1.38.56

1.38.55

1.38.54

1.38.53

1.38.52

1.38.51

1.38.50

1.38.49

1.38.48

1.38.47

1.38.46

1.38.45

1.38.44

1.38.43

1.38.42

1.38.41

1.38.40

1.38.39

1.38.38

1.38.37

1.38.36

1.38.35

1.38.34

1.38.33

1.38.32

1.38.31

1.38.30

1.38.29

1.38.28

1.38.27

1.38.26

1.38.25

1.38.24

1.38.23

1.38.22

1.38.21

1.38.20

1.38.19

1.38.18

1.38.17

1.38.16

1.38.15

1.38.14

1.38.13

1.38.12

1.38.11

1.38.10

1.38.9

1.38.8

1.38.7

1.38.6

1.38.5

1.38.4

1.38.3

1.38.2

1.38.1

1.38.0

1.37.0

1.36.41

1.36.40

1.36.39

1.36.38

1.36.37

1.36.36

1.36.35

1.36.34

1.36.33

1.36.32

1.36.31

1.36.30

1.36.29

1.36.28

1.36.27

1.36.26

1.36.25

1.36.24

1.36.23

1.36.22

1.36.21

1.36.20

1.36.19

1.36.18

1.36.17

1.36.16

1.36.15

1.36.14

1.36.13

1.36.12

1.36.11

1.36.10

1.36.9

1.36.8

1.36.7

1.36.6

1.36.5

1.36.4

1.36.3

1.36.2

1.36.1

1.36.0

1.35.2

1.35.1

1.35.0

1.34.30

1.34.29

1.34.28

1.34.27

1.34.26

1.34.25

1.34.24

1.34.23

1.34.22

1.34.21

1.34.20

1.34.19

1.34.18

1.34.17

1.34.16

1.34.15

1.34.14

1.34.13

1.34.12

1.34.11

1.34.10

1.34.9

1.34.8

1.34.7

1.34.6

1.34.5

1.34.4

1.34.3

1.34.2

1.34.1

1.34.0

1.33.1

1.33.0

1.32.0

1.31.8

1.31.7

1.31.6

1.31.5

1.31.4

1.31.3

1.31.2

1.31.1

1.31.0

1.30.0

1.29.16

1.29.15

1.29.14

1.29.13

1.29.12

1.29.11

1.29.10

1.29.9

1.29.8

1.29.7

1.29.6

1.29.5

1.29.4

1.29.3

1.29.2

1.29.1

1.29.0

1.28.7

1.28.6

1.28.5

1.28.4

1.28.3

1.28.2

1.28.1

1.28.0

1.27.0

1.26.6

1.26.5

1.26.4

1.26.3

1.26.2

1.26.1

1.26.0

1.25.20

1.25.19

1.25.18

1.25.17

1.25.16

1.25.15

1.25.14

1.25.13

1.25.12

1.25.11

1.25.10

1.25.9

1.25.8

1.25.7

1.25.6

1.25.5

1.25.4

1.25.3

1.25.2

1.25.1

1.25.0

1.24.2

1.24.1

1.24.0

1.23.2

1.23.1

1.23.0

1.22.1

1.22.0

1.21.8

1.21.7

1.21.6

1.21.5

1.21.4

1.21.3

1.21.2

1.21.1

1.21.0

1.20.6

1.20.5

1.20.4

1.20.3

1.20.2

1.20.1

1.20.0

1.19.10

1.19.9

1.19.8

1.19.7

1.19.6

1.19.5

1.19.4

1.19.3

1.19.2

1.19.1

1.19.0

1.18.9

1.18.8

1.18.7

1.18.6

1.18.5

1.18.4

1.18.3

1.18.2

1.18.1

1.18.0

1.17.0

1.16.5

1.16.4

1.16.3

1.16.2

1.16.1

1.16.0

1.15.8

1.15.7

1.15.6

1.15.5

1.15.4

1.15.3

1.15.2

1.15.1

1.15.0

1.14.26

1.14.25

1.14.24

1.14.23

1.14.22

1.14.21

1.14.20

1.14.19

1.14.18

1.14.17

1.14.16

1.14.15

1.14.14

1.14.13

1.14.12

1.14.11

1.14.10

1.14.9

1.14.8

1.14.7

1.14.6

1.14.5

1.14.4

1.14.3

1.14.2

1.14.1

1.14.0

1.13.2

1.13.1

1.13.0

1.12.94

1.12.93

1.12.92

1.12.91

1.12.90

1.12.89

1.12.88

1.12.87

1.12.86

1.12.85

1.12.84

1.12.83

1.12.82

1.12.81

1.12.80

1.12.79

1.12.78

1.12.77

1.12.76

1.12.75

1.12.74

1.12.73

1.12.72

1.12.71

1.12.70

1.12.69

1.12.68

1.12.67

1.12.66

1.12.65

1.12.64

1.12.63

1.12.62

1.12.61

1.12.60

1.12.59

1.12.58

1.12.57

1.12.56

1.12.55

1.12.54

1.12.53

1.12.52

1.12.51

1.12.50

1.12.49

1.12.48

1.12.47

1.12.46

1.12.45

1.12.44

1.12.43

1.12.42

1.12.41

1.12.40

1.12.39

1.12.38

1.12.37

1.12.36

1.12.35

1.12.34

1.12.33

1.12.32

1.12.31

1.12.30

1.12.29

1.12.28

1.12.27

1.12.26

1.12.25

1.12.24

1.12.23

1.12.22

1.12.21

1.12.20

1.12.19

1.12.18

1.12.17

1.12.16

1.12.15

1.12.14

1.12.13

1.12.12

1.12.11

1.12.10

1.12.9

1.12.8

1.12.7

1.12.6

1.12.5

1.12.4

1.12.3

1.12.2

1.12.1

1.12.0

1.11.19

1.11.18

1.11.17

1.11.16

1.11.15

1.11.14

1.11.13

1.11.12

1.11.11

1.11.10

1.11.9

1.11.8

1.11.7

1.11.6

1.11.5

1.11.4

1.11.3

1.11.2

1.11.1

1.11.0

1.10.18

1.10.17

1.10.16

1.10.15

1.10.14

1.10.13

1.10.12

1.10.11

1.10.10

1.10.9

1.10.8

1.10.7

1.10.6

1.10.5

1.10.4

1.10.3

1.10.2

1.10.1

1.10.0

1.9.3

1.9.2

1.9.1

1.9.0

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.64

1.7.63

1.7.62

1.7.61

1.7.60

1.7.59

1.7.58

1.7.57

1.7.56

1.7.55

1.7.54

1.7.53

1.7.52

1.7.51

1.7.50

1.7.49

1.7.48

1.7.47

1.7.46

1.7.45

1.7.44

1.7.43

1.7.42

1.7.41

1.7.40

1.7.39

1.7.38

1.7.37

1.7.36

1.7.35

1.7.34

1.7.33

1.7.32

1.7.31

1.7.30

1.7.29

1.7.28

1.7.27

1.7.26

1.7.25

1.7.24

1.7.23

1.7.22

1.7.21

1.7.20

1.7.19

1.7.18

1.7.17

1.7.16

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.38

1.6.37

1.6.36

1.6.35

1.6.34

1.6.33

1.6.32

1.6.31

1.6.30

1.6.29

1.6.28

1.6.27

1.6.26

1.6.25

1.6.24

1.6.23

1.6.22

1.6.21

1.6.20

1.6.19

1.6.18

1.6.17

1.6.16

1.6.15

1.6.14

1.6.13

1.6.12

1.6.11

1.6.10

1.6.9

1.6.8

1.6.7

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.1

1.4.0

1.3.4

1.3.3

1.3.2

1.3.1

1.2.0

1.1.2

1.1.1

1.1.0

1.0.2

1.0.1

1.0.0

0.9.1

0.0.1

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page