lecore-bm25

Okapi BM25 + Reciprocal Rank Fusion, pure NumPy/stdlib, deterministic.

The reason to use this instead of the usual pip BM25 is the tokenizer ships with it. That turns out to be the whole ballgame — see the numbers, which are stated with the decomposition that makes them honest.

pip install lecore-bm25

Credit where it's due

This is leCore's holographic/semantic_router/holographic_bm25.py, vendored and packaged. The algorithm, the tokenizer, the API and the docstrings are AnOversizedMooseWithSocks', MIT licensed, shipped here with his LICENSE verbatim. The only change is a doc-major postings build that replaces an O(vocab x N) loop that didn't terminate at BEIR-NQ scale; the original is kept beside it as _build_postings_vocab_major and a test asserts the two are bit-identical.

If you want the full library — holographic memory, semantic routing, the rest — go upstream. This package is just the lexical half, for people who want pip install and a good BM25.

Quickstart

from lecore_bm25 import BM25, tokenize, reciprocal_rank_fusion

docs = [
    "smooth out the bumpy surface of a mesh",
    "denoise a grainy image with a median filter",
    "subdivide a polygon mesh into smaller pieces",
]

bm = BM25(docs)              # k1=1.5, b=0.75 (Robertson defaults)
bm.rank("bumpy surface")     # -> [(0, 4.19...), (2, 0.71...), (1, 0.0)]
bm.scores("bumpy surface")   # -> np.ndarray, one score per doc

# fuse with any other ranker (no score calibration needed)
reciprocal_rank_fusion([[0, 2, 1], [2, 0, 1]], k=60)

tokenize is the part that matters and it's exported on purpose — stoplist plus light inflectional and derivational stemming. You can hand it to any other retriever.

The numbers

BEIR via the mteb/* HuggingFace datasets, scored with pytrec_eval ndcg_cut.10 — the same scorer mteb uses underneath — 1000-doc scoring pool, ignore_identical_ids on ArguAna.

nDCG@10:

Read rows four through six before you quote row three. Against rank_bm25 with the tokenization its README actually demonstrates (doc.lower().split(), since it ships no tokenizer at all), this wins by 10.9, 5.1 and 14.2 points. That number is real, and it is also the least impressive thing you could measure, because it is a comparison against the weakest baseline available. Two things collapse it:

• Hand rank_bm25 this package's tokenize and the gap evaporates.
• Compare against bm25s, which is maintained and whose documented default tokenization already lowercases and drops English stopwords, and the margin is +0.7, +1.2 and +0.3 points. On ArguAna, bm25s out of the box beat our 0.1.0 outright — that was the query-term-frequency bug below, and it took fixing it to pull level.

So:

The scoring math is not better. The tokenizer is the entire advantage — and against a library that already tokenizes sensibly, that advantage is worth about a point.

That's still a real advantage over the most-installed option, and it's the difference between what you get out of the box and what you get after you go build a stoplist and a stemmer yourself. But it is a packaging win, not an algorithmic one, and anyone telling you otherwise is selling something. If you are already using bm25s and happy, there is no numerical reason to switch.

One comparison we deliberately do not make: the published BEIR Elasticsearch BM25 baselines (SciFact 0.620, NFCorpus 0.297, ArguAna 0.441). Those are full-corpus; every number in the table above uses a 1000-document scoring pool, which is an easier problem. Putting them in the same table would flatter us dishonestly.

Where this used to lose: long queries (fixed in 0.2.0)

Through 0.1.0 this section documented a defeat: on ArguAna, rank_bm25 with our own tokenizer beat us by 5.4 points (0.4835 vs 0.4300). The mechanism was one line —

for t in sorted(set(q_terms)):   # 0.1.0: query terms DEDUPED
for q in query:                  # rank_bm25: every occurrence counts

— and 0.2.0 fixes it by counting query terms instead of deduping them, so a term repeated q times contributes q x its weight, which is what BM25's query-side term frequency has always meant.

We measured the fix across seven BEIR tasks before shipping it, because a change that helps one task and quietly costs the other six is not a fix:

The effect is entirely explained by how often query terms repeat. Six of seven BEIR tasks have keyword-length queries that repeat almost nothing, so deduping was invisible there — every delta is under 0.002, which is noise. ArguAna's "queries" are whole argument passages, and there it was throwing away real signal.

Be careful what you conclude from that correlation: it looks near-perfect (r = 0.998 against query length) but it is carried by a single point. Drop ArguAna and it collapses to r = 0.155. The honest reading is a bound, not a law — harmless up to ~9.5 tokens / 0.028 repeat rate, worth 5.7 points at 121.6 tokens / 0.230. Nothing we measured says where in between it starts to matter.

With the fix, this now edges rank_bm25-with-our-tokenizer on ArguAna too (0.4867 vs 0.4835), so the earlier advice to switch libraries for passage-length queries no longer applies.

Two more findings worth recording:

• The expand=True knob is noise. +0.0026 SciFact, −0.0014 NFCorpus, +0.0008 ArguAna. It is off by default and you should leave it off.
• Nothing here is "holographic." It's Robertson/Sparck-Jones BM25 with a good tokenizer.

These were independently reproduced from a fresh clone on different hardware by a tester in Moose's Telegram, matching to four decimals, before being re-run here.

Reproducing

The bench harness lives in the supercontext bench campaign. bm25_vs_pip_bench.py re-runs the three tasks against pip rank_bm25; tokenizer_tax_bench.py adds bm25s, scikit-learn TF-IDF and Whoosh, each in both its own default tokenization and this package's, which is where the "+0.7 against bm25s" number above comes from. Both write the tables above and gate on reproducing previously published values before reporting anything new.

API

• BM25(docs, k1=1.5, b=0.75) — docs is a list of raw strings
  • .scores(query, expand=False) → np.ndarray of length N
  • .rank(query, top=None, expand=False) → [(doc_index, score), ...] descending
• tokenize(text) → list[str]
• reciprocal_rank_fusion(ranked_lists, k=60, top=None, weights=None) → [(doc, score), ...]

License

MIT — Copyright (c) 2026 AnOversizedMooseWithSocks. See LICENSE.