MTEB MTEB

Multimodal toolbox for evaluating embeddings and retrieval systems

GitHub release License Downloads

Installation | Usage | Leaderboard | Documentation | Citing

Installation

You can install mteb simply using pip or uv. For more on installation please see the documentation.

pip install mteb

For faster installation, you can also use uv:

uv add mteb

Example Usage

Below we present a simple use-case example. For more information, see the documentation.

import mteb
from sentence_transformers import SentenceTransformer

# Select model
model_name = "sentence-transformers/all-MiniLM-L6-v2"
model = mteb.get_model(
    model_name
)  # if the model is not implemented in MTEB it will be eq. to SentenceTransformer(model_name)

# Select tasks
tasks = mteb.get_tasks(tasks=["Banking77Classification.v2"])

# evaluate
results = mteb.evaluate(model, tasks=tasks)

You can also run it using the CLI:

mteb run \
    -m sentence-transformers/all-MiniLM-L6-v2 \
    -t "Banking77Classification.v2" \
    --output-folder results

For more on how to use the CLI check out the related documentation.

Overview

Overview
📈 Leaderboard The interactive leaderboard of the benchmark
Get Started.
🏃 Get Started Overview of how to use mteb
🤖 Defining Models How to use existing models and define custom ones
📋 Selecting tasks How to select tasks, benchmarks, splits etc.
🏭 Running Evaluation How to run the evaluations, including cache management, speeding up evaluations etc.
📊 Loading Results How to load and work with existing model results
Overview.
📋 Tasks Overview of available tasks
📐 Benchmarks Overview of available benchmarks
🤖 Models Overview of available Models
Contributing
🤖 Adding a model How to submit a model to MTEB and to the leaderboard
👩‍💻 Adding a dataset How to add a new task/dataset to MTEB
👩‍💻 Adding a benchmark How to add a new benchmark to MTEB and to the leaderboard
🤝 Contributing How to contribute to MTEB and set it up for development

Citing

MTEB was introduced in "MTEB: Massive Text Embedding Benchmark", and heavily expanded in "MMTEB: Massive Multilingual Text Embedding Benchmark". When using mteb, we recommend that you cite both articles.

Bibtex Citation (click to unfold) 
@article{muennighoff2022mteb,
  author = {Muennighoff, Niklas and Tazi, Nouamane and Magne, Loïc and Reimers, Nils},
  title = {MTEB: Massive Text Embedding Benchmark},
  publisher = {arXiv},
  journal={arXiv preprint arXiv:2210.07316},
  year = {2022},
  url = {https://arxiv.org/abs/2210.07316},
  doi = {10.48550/ARXIV.2210.07316},
}

@article{enevoldsen2025mmtebmassivemultilingualtext,
  title={MMTEB: Massive Multilingual Text Embedding Benchmark},
  author={Kenneth Enevoldsen and Isaac Chung and Imene Kerboua and Márton Kardos and Ashwin Mathur and David Stap and Jay Gala and Wissam Siblini and Dominik Krzemiński and Genta Indra Winata and Saba Sturua and Saiteja Utpala and Mathieu Ciancone and Marion Schaeffer and Gabriel Sequeira and Diganta Misra and Shreeya Dhakal and Jonathan Rystrøm and Roman Solomatin and Ömer Çağatan and Akash Kundu and Martin Bernstorff and Shitao Xiao and Akshita Sukhlecha and Bhavish Pahwa and Rafał Poświata and Kranthi Kiran GV and Shawon Ashraf and Daniel Auras and Björn Plüster and Jan Philipp Harries and Loïc Magne and Isabelle Mohr and Mariya Hendriksen and Dawei Zhu and Hippolyte Gisserot-Boukhlef and Tom Aarsen and Jan Kostkan and Konrad Wojtasik and Taemin Lee and Marek Šuppa and Crystina Zhang and Roberta Rocca and Mohammed Hamdy and Andrianos Michail and John Yang and Manuel Faysse and Aleksei Vatolin and Nandan Thakur and Manan Dey and Dipam Vasani and Pranjal Chitale and Simone Tedeschi and Nguyen Tai and Artem Snegirev and Michael Günther and Mengzhou Xia and Weijia Shi and Xing Han Lù and Jordan Clive and Gayatri Krishnakumar and Anna Maksimova and Silvan Wehrli and Maria Tikhonova and Henil Panchal and Aleksandr Abramov and Malte Ostendorff and Zheng Liu and Simon Clematide and Lester James Miranda and Alena Fenogenova and Guangyu Song and Ruqiya Bin Safi and Wen-Ding Li and Alessia Borghini and Federico Cassano and Hongjin Su and Jimmy Lin and Howard Yen and Lasse Hansen and Sara Hooker and Chenghao Xiao and Vaibhav Adlakha and Orion Weller and Siva Reddy and Niklas Muennighoff},
  publisher = {arXiv},
  journal={arXiv preprint arXiv:2502.13595},
  year={2025},
  url={https://arxiv.org/abs/2502.13595},
  doi = {10.48550/arXiv.2502.13595},
}

If you use any of the specific benchmarks, we also recommend that you cite the authors of both the benchmark and its tasks:

benchmark = mteb.get_benchmark("MTEB(eng, v2)")
benchmark.citation  # get citation for a specific benchmark

# you can also create a table of the task for the appendix using:
benchmark.tasks.to_latex()

Project links

Data verified by PyPI on Jul 30, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (2.18.16)

4 maintainers

PyPI data
Data sourced directly from PyPI's database.
Avatar for KennethEnevoldsen from gravatar.com KennethEnevoldsen Avatar for Muennighoff from gravatar.com Muennighoff Avatar for nouamanetazi from gravatar.com nouamanetazi
+1 more Avatar for nreimers from gravatar.com nreimers

Credits

Author: MTEB Contributors

Maintainer: Kenneth Enevoldsen

GitHub Statistics

Data verified by PyPI on Jul 30, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

Apache-2.0
View SPDX License List

Requires

Python <3.15, >=3.10

Provides Extra

image audio codecarbon leaderboard api og video github peft flagembedding jina jina-clip jina-v4 flash-attention colvec1-1 openai model2vec pylate msclap bm25s gritlm xformers blip2 voyageai voyage-v cohere vertexai google-genai octen llm2vec timm open-clip-torch nomic ark colpali-engine colqwen3 visrag-ret sauerkrautlm-colpali xet youtu llama-embed-nemotron llama-nemotron-colembed-vl nemotron-colembed-vl-v2 llama-nemotron-embed-vl-1b-v2 nemotron-3-embed multimodal-sbert faiss-cpu eager-embed speechbrain muq wav2clip fusion-embedding torch-vggish-yamnet vllm transformers-v4 siglip qwen-vl qwen-omni-utils embeddinggemma

Tags

deep learning text embeddings embeddings multimodal benchmark retrieval information retrieval

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mteb-2.18.8.tar.gz (3.9 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

mteb-2.18.8-py3-none-any.whl (6.1 MB view details)

Uploaded Python 3

File details

Details for the file mteb-2.18.8.tar.gz.

File metadata

  • Download URL: mteb-2.18.8.tar.gz
  • Upload date:
  • Size: 3.9 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for mteb-2.18.8.tar.gz
Algorithm Hash digest
SHA256 b7066bc96f988913f44ce026e4e448e9fdf7f69d6c2188bbc39f35ecf1fd37a3
MD5 8ec1719540e6a6f77d35feee8ac8df3d
BLAKE2b-256 66cbb112bb4851db5e35451b669d2a593dba0361d62bf446427114596c8100b1

See more details on using hashes here.

Provenance

The following attestation bundles were made for mteb-2.18.8.tar.gz:

Publisher: release.yml on embeddings-benchmark/mteb

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file mteb-2.18.8-py3-none-any.whl.

File metadata

  • Download URL: mteb-2.18.8-py3-none-any.whl
  • Upload date:
  • Size: 6.1 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for mteb-2.18.8-py3-none-any.whl
Algorithm Hash digest
SHA256 7a63730bef54fff7dcc7a5848e0da8dd43911f3a47facc70a5fc4e05dcbda8cf
MD5 ac33ab47e3228117dfb2c650a8465a60
BLAKE2b-256 eb135fde09e438b85e7ecdfc494368a1a3a8b7832aef20f0af170301f542a6c6

See more details on using hashes here.

Provenance

The following attestation bundles were made for mteb-2.18.8-py3-none-any.whl:

Publisher: release.yml on embeddings-benchmark/mteb

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

2.18.16

2.18.15

2.18.14

2.18.13

2.18.12

2.18.11

2.18.10

2.18.9

This release

2.18.8

2.18.7

2.18.6

2.18.5

2.18.4

2.18.3

2.18.2

2.18.1

2.18.0

2.17.0

2.16.3

2.16.2

2.16.1

2.16.0

2.15.6

2.15.5

2.15.4

2.15.3

2.15.2

2.15.1

2.15.0

2.14.9

2.14.8

2.14.7

2.14.6

2.14.5

2.12.30

2.12.29

2.12.28

2.12.27

2.12.26

2.12.25

2.12.24

2.12.23

2.12.22

2.12.21

2.12.20

2.12.19

2.12.18

2.12.17

2.12.16

2.12.15

2.12.14

2.12.13

2.12.12

2.12.11

2.12.10

2.12.9

2.12.8

2.12.7

2.12.6

2.12.5

2.12.4

2.12.3

2.12.2

2.12.1

2.12.0

2.11.10

2.11.9

2.11.8

2.11.7

2.11.6

2.11.5

2.11.4

2.11.3

2.11.2

2.11.1

2.11.0

2.10.14

2.10.13

2.10.12

2.10.11

2.10.10

2.10.9

2.10.8

2.10.7

2.10.6

2.10.5

2.10.4

2.10.3

2.10.2

2.10.1

2.10.0

2.9.1

2.9.0

2.8.10

2.8.9

2.8.8

2.8.7

2.8.6

2.8.5

2.8.4

2.8.3

2.8.2

2.8.1

2.7.30

2.7.29

2.7.28

2.7.27

2.7.26

2.7.25

2.7.24

2.7.23

2.7.22

2.7.21

2.7.20

2.7.19

2.7.18

2.7.17

2.7.16

2.7.15

2.7.14

2.7.13

2.7.12

2.7.11

2.7.10

2.7.9

2.7.8

2.7.7

2.7.6

2.7.5

2.7.4

2.7.3

2.7.2

2.7.1

2.7.0

2.6.9

2.6.8

2.6.7

2.6.6

2.6.5

2.6.4

2.6.3

2.6.2

2.6.1

2.6.0

2.5.5

2.5.4

2.5.3

2.5.2

2.5.1

2.5.0

2.4.2

2.4.1

2.3.11

2.3.10

2.3.9

2.3.8

2.3.7

2.3.6

2.3.5

2.3.4

2.3.3

2.3.2

2.3.1

2.3.0

2.2.2

2.2.1

2.2.0

2.1.19

2.1.18

2.1.17

2.1.16

2.1.15

2.1.14

2.1.13

2.1.12

2.1.11

2.1.10

2.1.9

2.1.8

2.1.7

2.1.6

2.1.5

2.1.4

2.1.3

2.1.2

2.1.1

2.1.0

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

1.39.7

1.39.6

1.39.5

1.39.4

1.39.3

1.39.2

1.39.1

1.39.0

1.38.61

1.38.60

1.38.59

1.38.58

1.38.57

1.38.56

1.38.55

1.38.54

1.38.53

1.38.52

1.38.51

1.38.50

1.38.49

1.38.48

1.38.47

1.38.46

1.38.45

1.38.44

1.38.43

1.38.42

1.38.41

1.38.40

1.38.39

1.38.38

1.38.37

1.38.36

1.38.35

1.38.34

1.38.33

1.38.32

1.38.31

1.38.30

1.38.29

1.38.28

1.38.27

1.38.26

1.38.25

1.38.24

1.38.23

1.38.22

1.38.21

1.38.20

1.38.19

1.38.18

1.38.17

1.38.16

1.38.15

1.38.14

1.38.13

1.38.12

1.38.11

1.38.10

1.38.9

1.38.8

1.38.7

1.38.6

1.38.5

1.38.4

1.38.3

1.38.2

1.38.1

1.38.0

1.37.0

1.36.41

1.36.40

1.36.39

1.36.38

1.36.37

1.36.36

1.36.35

1.36.34

1.36.33

1.36.32

1.36.31

1.36.30

1.36.29

1.36.28

1.36.27

1.36.26

1.36.25

1.36.24

1.36.23

1.36.22

1.36.21

1.36.20

1.36.19

1.36.18

1.36.17

1.36.16

1.36.15

1.36.14

1.36.13

1.36.12

1.36.11

1.36.10

1.36.9

1.36.8

1.36.7

1.36.6

1.36.5

1.36.4

1.36.3

1.36.2

1.36.1

1.36.0

1.35.2

1.35.1

1.35.0

1.34.30

1.34.29

1.34.28

1.34.27

1.34.26

1.34.25

1.34.24

1.34.23

1.34.22

1.34.21

1.34.20

1.34.19

1.34.18

1.34.17

1.34.16

1.34.15

1.34.14

1.34.13

1.34.12

1.34.11

1.34.10

1.34.9

1.34.8

1.34.7

1.34.6

1.34.5

1.34.4

1.34.3

1.34.2

1.34.1

1.34.0

1.33.1

1.33.0

1.32.0

1.31.8

1.31.7

1.31.6

1.31.5

1.31.4

1.31.3

1.31.2

1.31.1

1.31.0

1.30.0

1.29.16

1.29.15

1.29.14

1.29.13

1.29.12

1.29.11

1.29.10

1.29.9

1.29.8

1.29.7

1.29.6

1.29.5

1.29.4

1.29.3

1.29.2

1.29.1

1.29.0

1.28.7

1.28.6

1.28.5

1.28.4

1.28.3

1.28.2

1.28.1

1.28.0

1.27.0

1.26.6

1.26.5

1.26.4

1.26.3

1.26.2

1.26.1

1.26.0

1.25.20

1.25.19

1.25.18

1.25.17

1.25.16

1.25.15

1.25.14

1.25.13

1.25.12

1.25.11

1.25.10

1.25.9

1.25.8

1.25.7

1.25.6

1.25.5

1.25.4

1.25.3

1.25.2

1.25.1

1.25.0

1.24.2

1.24.1

1.24.0

1.23.2

1.23.1

1.23.0

1.22.1

1.22.0

1.21.8

1.21.7

1.21.6

1.21.5

1.21.4

1.21.3

1.21.2

1.21.1

1.21.0

1.20.6

1.20.5

1.20.4

1.20.3

1.20.2

1.20.1

1.20.0

1.19.10

1.19.9

1.19.8

1.19.7

1.19.6

1.19.5

1.19.4

1.19.3

1.19.2

1.19.1

1.19.0

1.18.9

1.18.8

1.18.7

1.18.6

1.18.5

1.18.4

1.18.3

1.18.2

1.18.1

1.18.0

1.17.0

1.16.5

1.16.4

1.16.3

1.16.2

1.16.1

1.16.0

1.15.8

1.15.7

1.15.6

1.15.5

1.15.4

1.15.3

1.15.2

1.15.1

1.15.0

1.14.26

1.14.25

1.14.24

1.14.23

1.14.22

1.14.21

1.14.20

1.14.19

1.14.18

1.14.17

1.14.16

1.14.15

1.14.14

1.14.13

1.14.12

1.14.11

1.14.10

1.14.9

1.14.8

1.14.7

1.14.6

1.14.5

1.14.4

1.14.3

1.14.2

1.14.1

1.14.0

1.13.2

1.13.1

1.13.0

1.12.94

1.12.93

1.12.92

1.12.91

1.12.90

1.12.89

1.12.88

1.12.87

1.12.86

1.12.85

1.12.84

1.12.83

1.12.82

1.12.81

1.12.80

1.12.79

1.12.78

1.12.77

1.12.76

1.12.75

1.12.74

1.12.73

1.12.72

1.12.71

1.12.70

1.12.69

1.12.68

1.12.67

1.12.66

1.12.65

1.12.64

1.12.63

1.12.62

1.12.61

1.12.60

1.12.59

1.12.58

1.12.57

1.12.56

1.12.55

1.12.54

1.12.53

1.12.52

1.12.51

1.12.50

1.12.49

1.12.48

1.12.47

1.12.46

1.12.45

1.12.44

1.12.43

1.12.42

1.12.41

1.12.40

1.12.39

1.12.38

1.12.37

1.12.36

1.12.35

1.12.34

1.12.33

1.12.32

1.12.31

1.12.30

1.12.29

1.12.28

1.12.27

1.12.26

1.12.25

1.12.24

1.12.23

1.12.22

1.12.21

1.12.20

1.12.19

1.12.18

1.12.17

1.12.16

1.12.15

1.12.14

1.12.13

1.12.12

1.12.11

1.12.10

1.12.9

1.12.8

1.12.7

1.12.6

1.12.5

1.12.4

1.12.3

1.12.2

1.12.1

1.12.0

1.11.19

1.11.18

1.11.17

1.11.16

1.11.15

1.11.14

1.11.13

1.11.12

1.11.11

1.11.10

1.11.9

1.11.8

1.11.7

1.11.6

1.11.5

1.11.4

1.11.3

1.11.2

1.11.1

1.11.0

1.10.18

1.10.17

1.10.16

1.10.15

1.10.14

1.10.13

1.10.12

1.10.11

1.10.10

1.10.9

1.10.8

1.10.7

1.10.6

1.10.5

1.10.4

1.10.3

1.10.2

1.10.1

1.10.0

1.9.3

1.9.2

1.9.1

1.9.0

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.64

1.7.63

1.7.62

1.7.61

1.7.60

1.7.59

1.7.58

1.7.57

1.7.56

1.7.55

1.7.54

1.7.53

1.7.52

1.7.51

1.7.50

1.7.49

1.7.48

1.7.47

1.7.46

1.7.45

1.7.44

1.7.43

1.7.42

1.7.41

1.7.40

1.7.39

1.7.38

1.7.37

1.7.36

1.7.35

1.7.34

1.7.33

1.7.32

1.7.31

1.7.30

1.7.29

1.7.28

1.7.27

1.7.26

1.7.25

1.7.24

1.7.23

1.7.22

1.7.21

1.7.20

1.7.19

1.7.18

1.7.17

1.7.16

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.38

1.6.37

1.6.36

1.6.35

1.6.34

1.6.33

1.6.32

1.6.31

1.6.30

1.6.29

1.6.28

1.6.27

1.6.26

1.6.25

1.6.24

1.6.23

1.6.22

1.6.21

1.6.20

1.6.19

1.6.18

1.6.17

1.6.16

1.6.15

1.6.14

1.6.13

1.6.12

1.6.11

1.6.10

1.6.9

1.6.8

1.6.7

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.1

1.4.0

1.3.4

1.3.3

1.3.2

1.3.1

1.2.0

1.1.2

1.1.1

1.1.0

1.0.2

1.0.1

1.0.0

0.9.1

0.0.1

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page