bm25q 0.0.1

Faster, API-compatible BM25 with quality-bounded quantized retrieval.

bm25q

Fast BM25 retrieval with quality-bounded quantization.

Website · PyPI · Source

bm25q is a faster alternative to bm25s that keeps the same indexing, tokenization, retrieval, save/load, and Hugging Face APIs. Exact retrieval is still the default. Quantized retrieval is opt-in.

The adaptive mode uses one-byte posting impacts and accumulators only when a per-query upper bound proves that the accumulator cannot overflow. Queries that do not satisfy that proof automatically use the higher-precision path.

Install

pip install "bm25q[core]"

The base package only requires NumPy. The recommended core extra adds the stemmer, progress utilities, fast JSON support, and Numba retrieval backend.

Quick start

import bm25q
import Stemmer

corpus = [
    "a cat likes to purr",
    "a dog loves to play",
    "a fish lives in water",
]

stemmer = Stemmer.Stemmer("english")
corpus_tokens = bm25q.tokenize(corpus, stopwords="en", stemmer=stemmer)

retriever = bm25q.BM25(
    corpus=corpus,
    backend="numba",
    quantize="adaptive",
)
retriever.index(corpus_tokens)

query_tokens = bm25q.tokenize(
    "where does a fish live?",
    stopwords="en",
    stemmer=stemmer,
)
documents, scores = retriever.retrieve(query_tokens, k=2)

Existing code only needs an import change:

import bm25q

retriever = bm25q.BM25()
tokens = bm25q.tokenize(["document one", "document two"])
retriever.index(tokens)

Retrieval modes

Setting	Behavior
quantize=False	Exact floating-point retrieval; default
quantize=True	Fast 8-bit impacts with a 16-bit accumulator
quantize="adaptive"	Quality-bounded one-byte retrieval with automatic higher-precision fallback

Quantization can slightly reorder documents with nearly identical scores. Across the full 15-dataset BEIR validation, the largest relative change in NDCG@10 or Recall@1000 was 0.474%.

Performance

Single-threaded, full-query local measurements with k=1000:

Dataset	Quantized QPS	Adaptive QPS	Speedup
Natural Questions	421.5	558.2	1.32x
MSMARCO	130.8	268.9	2.06x
11-dataset aggregate	412.8	624.4	1.51x

The paired all-BEIR Kaggle sweep measured 205.77 aggregate QPS for the regular quantized mode and 243.76 QPS for adaptive mode (1.18x), with all 53,359 qrels queries and the same benchmark settings for both modes.

Save and load

retriever.save("index", corpus=corpus)
reloaded = bm25q.BM25.load("index", load_corpus=True)

Optional extras

pip install "bm25q[hf]"         # Hugging Face Hub integration
pip install "bm25q[cli]"        # terminal interface
pip install "bm25q[evaluation]" # BEIR-style evaluation helpers
pip install "bm25q[full]"       # everything

The command-line entry point is bm25q:

bm25q index documents.jsonl -c text -o index
bm25q search -i index "search terms"

License

MIT