Skip to main content

Various BM25 algorithms for document ranking

Project description

Rank_BM25: A two line search engine

A collection of algorithms for querying a set of documents and returning the ones most relevant to the query. The most common use case for these algorithms is, as you might have guessed, to create search engines.

So far the algorithms that have been implemented are:

  • Okapi BM25
  • BM25L
  • BM25+

Todo:

  • BM25-Adpt
  • BM25T

These algorithms were taken from this paper, which gives a nice overview of each method, and also benchmarks them against each other. A nice inclusion is that they compare different kinds of preprocessing like stemming vs no-stemming, stopword removal or not, etc. Great read if you're new to the subject.

Usage

For this example we'll be using the BM25Okapi algorithm, but the others are used in pretty much the same way.

Initalizing

First thing to do is create an instance of the BM25 class, which reads in a corpus of text and does some indexing on it:

from rank_bm25 import BM25Okapi

corpus = [
    "Hello there good man!",
    "It is quite windy in London",
    "How is the weather today?"
]

tokenized_corpus = [doc.split(" ") for doc in corpus]

bm25 = BM25Okapi(corpus)
# <rank_bm25.BM25Okapi at 0x1047881d0>

Note that this package doesn't do any text preprocessing. If you want to do things like lowercasing, stopword removal, stemming, etc, you need to do it yourself.

The only requirements is that the class receives a list of lists of strings, which are the document tokens.

Ranking of documents

Now that we've created our document indexes, we can give it queries and see which documents are the most relevant:

query = "windy London"
tokenized_query = query.split(" ")

doc_scores = bm25.get_scores(tokenized_query)
# array([0.        , 0.93729472, 0.        ])

Good to note that we also need to tokenize our query, and apply the same preprocessing steps we did to the documents in order to have an apples-to-apples comparison

Instead of getting the document scores, you can also just retrieve the best documents with

bm25.get_top_n(tokenized_query, corpus, n=1)
# ['It is quite windy in London']

And that's pretty much it!

Installation

The easiest way to install this package is through pip, using

pip install rank_bm25

If you want to be sure you're getting the newest version, you can install it directly from github wth

pip install git+ssh://git@github.com/dorianbrown/rank_bm25.git

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

rank_bm25-0.1.tar.gz (3.9 kB view details)

Uploaded Source

Built Distribution

rank_bm25-0.1-py3-none-any.whl (8.1 kB view details)

Uploaded Python 3

File details

Details for the file rank_bm25-0.1.tar.gz.

File metadata

  • Download URL: rank_bm25-0.1.tar.gz
  • Upload date:
  • Size: 3.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/39.1.0 requests-toolbelt/0.8.0 tqdm/4.29.1 CPython/3.7.2

File hashes

Hashes for rank_bm25-0.1.tar.gz
Algorithm Hash digest
SHA256 53c19803f6a0a52134b2e5dbf4c5252e41a90a30bac49ff15dc0285da05d6b2a
MD5 53d19e7570566dbbdb3a63abb2ce0739
BLAKE2b-256 8fb14d56d4ba2194e33a7fe81985fc0b7d8b07984596cefb9a88703c1fbe5a90

See more details on using hashes here.

File details

Details for the file rank_bm25-0.1-py3-none-any.whl.

File metadata

  • Download URL: rank_bm25-0.1-py3-none-any.whl
  • Upload date:
  • Size: 8.1 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/39.1.0 requests-toolbelt/0.8.0 tqdm/4.29.1 CPython/3.7.2

File hashes

Hashes for rank_bm25-0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 594a1d1f98a9818ddcc6b11210b868945dc9c034f7c8f0ce75097de8f84a506f
MD5 c8d95a60640e3a7de9181c31729059a7
BLAKE2b-256 065f23ac059dbc81f3e7a6ae7e25e4203407a9246bd12894f89d70efe408cd26

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page