retrievalx

The complete BM25 engine for Python — all major scoring variants, multiple retrieval strategies, Rust-native performance.

48-92x faster than rank-bm25 with equal or better retrieval quality on BEIR benchmarks.

Installation

uv add retrievalx

or with pip:

pip install retrievalx

Pre-built wheels available for:

Supports Python 3.9 through 3.15 (including 3.14 and 3.15 pre-releases). Zero dependencies.

Quickstart

from retrievalx import BM25Index

# Index documents
index = BM25Index.from_documents([
    "rust and python",
    "information retrieval with bm25",
    "search engine internals",
])

# Search
for hit in index.search("rust retrieval", top_k=2):
    print(f"{hit.doc_id}: {hit.score:.4f}")

Features

8 Scoring Variants

BM25 Okapi, Plus, L, Adpt, F (field-weighted), T (term-specific k1), Atire, and Tf-Idf.

from retrievalx import BM25Config, ScoringVariant

config = BM25Config(scoring=ScoringVariant.plus(k1=1.5, b=0.8, delta=1.0))
index = BM25Index.from_documents(docs, config=config)

5 Retrieval Strategies

Exhaustive DAAT/TAAT, WAND, Block-Max WAND, and MaxScore.

from retrievalx import RetrievalStrategy

# Exact retrieval
config = BM25Config(retrieval=RetrievalStrategy.exhaustive_taat())

# Fast approximate top-k
config = BM25Config(retrieval=RetrievalStrategy.block_max_wand())

Advanced Query Types

from retrievalx import BooleanQuery, PhraseQuery, WeightedQuery

# Boolean: must/should/must_not
index.search_boolean(BooleanQuery(must=["python"], should=["fast"], must_not=["slow"]))

# Phrase with proximity window
index.search_phrase(PhraseQuery(terms=["information", "retrieval"], window=2))

# Weighted terms
index.search_weighted(WeightedQuery(weights={"python": 2.0, "search": 1.0}))

Persistence & Crash Recovery

# Save and load
index.save("index.bin")
loaded = BM25Index.load("index.bin")              # in-memory
loaded = BM25Index.load("index.bin", mode="mmap")  # memory-mapped

# Write-ahead log for crash recovery
index.enable_wal("index.wal")
index.insert_batch(new_docs)
index.compact_and_flush("index.bin")

Incremental Updates

# Add documents (with optional IDs)
index.insert_batch(["new document text"])
index.insert_batch([("doc-id", "document with custom ID")])

# Delete and compact
index.delete("doc-id")
index.compact()

Score Fusion

Combine BM25 with dense retrieval or other signals:

from retrievalx import rrf, linear_combination, min_max_normalize

fused = rrf(bm25_results, dense_results, k=60)
fused = linear_combination(bm25_results, dense_results, alpha=0.7)
normalized = min_max_normalize(scores)

Evaluation Metrics

Built-in IR metrics with native acceleration:

from retrievalx import ndcg_at_k, recall_at_k, precision_at_k, mrr, average_precision_at_k

ndcg = ndcg_at_k(ranked_ids, relevant_ids, k=10)

Custom Tokenization

from retrievalx import BM25Config, TokenizerConfig, Tokenizer, Filter, Stemmer

config = BM25Config(
    tokenizer=TokenizerConfig(
        tokenizer=Tokenizer.UNICODE,
        filters=[Filter.LOWERCASE, Filter.stopwords("en"), Filter.length(min_len=2)],
        stemmer=Stemmer.snowball("en"),
    )
)

Benchmarks

On BEIR SciFact (5,183 documents, 300 queries):

All retrieval strategies produce identical quality metrics — no accuracy tradeoff.

Full 40-configuration matrix (8 scorers x 5 strategies): docs/benchmarks.md

Architecture

Rust workspace with five crates:

Details: docs/architecture.md | docs/algorithms.md

Examples

Development

# Setup with uv
uv sync --extra dev

# Or with pip
python -m venv .venv && source .venv/bin/activate
pip install -e .[dev]

# Run all checks (mirrors CI)
./scripts/check_all.sh

# Run benchmarks
uv sync --extra bench
uv run python examples/benchmark_retrievalx_vs_rank_bm25.py --dataset scifact

See CONTRIBUTING.md for guidelines.

License

Apache-2.0