beir 0.0.9

A Heterogeneous Benchmark for Information Retrieval

What is it?

BEIR consists a heterogeneous benchmark for diverse sentence or passage IR level tasks. It also provides a common and easy framework for evaluation of your NLP models on them.

The package takes care of the downloading, hosting, preprocessing datasets and providing you in a single easy to understand dataset zip folders. We take care of transforming the dataset and provide 15 diverse datasets used for IR in the both academia and industry, with more to add. Further the package provides an easy framework to evalaute your models against some competitive benchmarks including Sentence-Transformers (SBERT), Dense Passage Retrieval (DPR), Universal Sentence Encoder (USE-QA) and Elastic Search.

Worried about your dataset or model not present in the benchmark?

Worry not! You can easily add your dataset into the benchmark by following this data format (here) and also you are free to evaluate your own model and required to return a dictionary with mappings (here) and you can evaluate your IR model using our easy plugin code.

Want us to add a new dataset or a new model? feel free to post an issue here or make a pull request!

Installation

Install via pip:

pip install beir

If you want to build from source, use:

$ git clone https://github.com/benchmarkir/beir.git
$ pip install -e .

Tested with python versions 3.6 and 3.7

Getting Started

Try it out live with our Google Colab Example.

First download and unzip a dataset. Click here to view all datasets available.

from beir import util

url = "https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/trec-covid.zip"
out_path = "datasets/trec-covid.zip"
out_dir = "datasets"

util.download_url(url, out_path)
util.unzip(out_path, out_dir)

Then load the dataset using our Generic Data Loader, (Wonderful right)

from beir.datasets.data_loader import GenericDataLoader
from beir.retrieval.evaluation import EvaluateRetrieval

data_path = "datasets/trec-covid/"
corpus, queries, qrels = GenericDataLoader(data_path).load(split="test")

Now, you can use either Sentence-transformers, DPR or USE-QA as your dense retriever model. Format of results is identical to that of qrels.

from beir.retrieval.evaluation import EvaluateRetrieval

retriever = EvaluateRetrieval(model="sbert", model_name="distilroberta-base-msmarco-v2") 
# retriever = EvaluateRetrieval(model="dpr")
# retriever = EvaluateRetrieval(model="use-qa")

results = retriever.retrieve(corpus, queries, qrels)

Finally after retrieving, you can evaluate your IR performance using qrels and results. We find NDCG@10 score for all datasets, for more details on why check our upcoming paper.

ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)

for key, value in ndcg.items():
    print(key, value) 
# ndcg@1    0.3456
# ndcg@3    0.4567
# ...

Examples

For all examples, see below:

Retrieval

• Google Colab Example
• Exact Search Retrieval using SBERT
• Exact Search Retrieval using DPR
• Exact Search Retrieval using USE-QA
• Faiss Search Retrieval using SBERT

Datasets

Available datasets include:

• TREC-COVID
• NFCorpus
• NQ
• HotpotQA
• NewsQA
• FiQA
• ArguAna
• Touche-2020
• CQaDupstack
• Quora
• DBPedia-v2
• SCIDOCS
• FEVER
• Climate-FEVER
• Signal-1M (Optional)
• BioASQ (Optional)

Data Formats

from beir.datasets.data_loader import GenericDataLoader

data_path = "datasets/trec-covid/"
corpus, queries, qrels = GenericDataLoader(data_path).load(split="test")

# Corpus
for doc_id, doc_metadata in corpus.items():
    print(doc_id, doc_metadata)
# ug7v899j  {"title": "Clinical features of culture-proven Mycoplasma...", "text": "This retrospective chart review describes the epidemiology..."}
# 02tnwd4m  {"title": "Nitric oxide: a pro-inflammatory mediator in lung disease?, "text": "Inflammatory diseases of the respiratory tract are commonly associated..."}
# ...

# Queries
for query_id, query_text in query.items():
    print(query_id, query_text)
# 1     what is the origin of COVID-19?
# 2     how does the coronavirus respond to changes in the weather?
# ...

# Query Relevance Judgements (Qrels)
for query_id, metadata in qrels.items():
    for doc_id, gold_score in metadata.items():
        print(query_id, doc_id, gold_score)
# 1     005b2j4b    2
# 1     00fmeepz    1
# ...

Benchmarking

Domain	Dataset	BM25	SBERT	USE-QA	DPR
	TREC-COVID
Bio-Medical	BioASQ
	NFCorpus
Question	NQ
Answering	HotpotQA
News	NewsQA
Twitter	Signal-1M
Finance	FiQA-2018
Argument	ArguAna
	Touche-2020
Duplicate	CQaDupstack
Question	Quora
Entity	DBPedia-v2
Scientific	SCIDOCS
Claim	FEVER
Verification	Climate-FEVER

Citing & Authors

The main contributors of this repository are:

• Nandan Thakur

Contact person: Nandan Thakur, [e-mail]

https://www.ukp.tu-darmstadt.de/

Don't hesitate to send us an e-mail or report an issue, if something is broken (and it shouldn't be) or if you have further questions.

This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.