lecore-bm25

Okapi BM25 + Reciprocal Rank Fusion, pure NumPy/stdlib, deterministic.

The reason to use this instead of the usual pip BM25 is the tokenizer ships with it. That turns out to be the whole ballgame — see the numbers, which are stated with the decomposition that makes them honest.

pip install lecore-bm25

Credit where it's due

This is leCore's holographic/semantic_router/holographic_bm25.py, vendored and packaged. The algorithm, the tokenizer, the API and the docstrings are AnOversizedMooseWithSocks', MIT licensed, shipped here with his LICENSE verbatim. The only change is a doc-major postings build that replaces an O(vocab x N) loop that didn't terminate at BEIR-NQ scale; the original is kept beside it as _build_postings_vocab_major and a test asserts the two are bit-identical.

If you want the full library — holographic memory, semantic routing, the rest — go upstream. This package is just the lexical half, for people who want pip install and a good BM25.

Quickstart

from lecore_bm25 import BM25, tokenize, reciprocal_rank_fusion

docs = [
    "smooth out the bumpy surface of a mesh",
    "denoise a grainy image with a median filter",
    "subdivide a polygon mesh into smaller pieces",
]

bm = BM25(docs)              # k1=1.5, b=0.75 (Robertson defaults)
bm.rank("bumpy surface")     # -> [(0, 4.19...), (2, 0.71...), (1, 0.0)]
bm.scores("bumpy surface")   # -> np.ndarray, one score per doc

# fuse with any other ranker (no score calibration needed)
reciprocal_rank_fusion([[0, 2, 1], [2, 0, 1]], k=60)

tokenize is the part that matters and it's exported on purpose — stoplist plus light inflectional and derivational stemming. You can hand it to any other retriever.

The numbers

BEIR via the mteb/* HuggingFace datasets, scored with pytrec_eval ndcg_cut.10 — the same scorer mteb uses underneath — 1000-doc scoring pool, ignore_identical_ids on ArguAna.

nDCG@10:

Read the third row before you quote the second. Against rank_bm25 with the tokenization its README actually demonstrates (doc.lower().split(), since it ships no tokenizer at all), this wins by 10.8, 5.1 and 8.5 points. But hand rank_bm25 this package's tokenize and the gap evaporates. So:

The scoring math is not better. The tokenizer is the entire advantage.

That's still a real advantage — it's the difference between what you get out of the box and what you get after you go build a stoplist and a stemmer yourself — but it is a packaging win, not an algorithmic one, and anyone telling you otherwise is selling something.

Where this loses: long queries

On ArguAna the third row doesn't just match us, it beats us by 5.4 points (0.4835 vs 0.4300). That is a real limitation and here is the mechanism, so you can decide if it applies to you:

for t in sorted(set(q_terms)):   # lecore-bm25: query terms DEDUPED
for q in query:                  # rank_bm25:   every occurrence counts

This implementation drops query-term frequency — a word repeated five times in your query scores the same as a word appearing once. For keyword-length queries that is invisible (terms rarely repeat) and it buys reproducibility. On ArguAna, where every "query" is a full argument passage, it throws away real signal and costs 5.4 points.

So: if your queries are short, use this. If your queries are passage-length, use rank_bm25's scoring with this package's tokenize — which is three lines and strictly better:

from rank_bm25 import BM25Okapi
from lecore_bm25 import tokenize
bm = BM25Okapi([tokenize(d) for d in docs])
bm.get_scores(tokenize(query))

Two more findings worth recording:

• The expand=True knob is noise. +0.0026 SciFact, −0.0014 NFCorpus, +0.0008 ArguAna. It is off by default and you should leave it off.
• Nothing here is "holographic." It's Robertson/Sparck-Jones BM25 with a good tokenizer.

These were independently reproduced from a fresh clone on different hardware by a tester in Moose's Telegram, matching to four decimals, before being re-run here.

Reproducing

The bench harness lives in the supercontext bench campaign (bm25_vs_pip_bench.py). It re-runs all three tasks against pip rank_bm25 and writes the table above.

API

• BM25(docs, k1=1.5, b=0.75) — docs is a list of raw strings
  • .scores(query, expand=False) → np.ndarray of length N
  • .rank(query, top=None, expand=False) → [(doc_index, score), ...] descending
• tokenize(text) → list[str]
• reciprocal_rank_fusion(ranked_lists, k=60, top=None, weights=None) → [(doc, score), ...]

License

MIT — Copyright (c) 2026 AnOversizedMooseWithSocks. See LICENSE.