lecore-bm25

Okapi BM25 + Reciprocal Rank Fusion, pure NumPy/stdlib, deterministic.

The reason to use this instead of the usual pip BM25 is the tokenizer ships with it. That turns out to be the whole ballgame — see the numbers, which are stated with the decomposition that makes them honest.

pip install lecore-bm25

Credit where it's due

This is leCore's holographic/semantic_router/holographic_bm25.py, vendored and packaged. The algorithm, the tokenizer, the API and the docstrings are AnOversizedMooseWithSocks', MIT licensed, shipped here with his LICENSE verbatim. The only change is a doc-major postings build that replaces an O(vocab x N) loop that didn't terminate at BEIR-NQ scale; the original is kept beside it as _build_postings_vocab_major and a test asserts the two are bit-identical.

If you want the full library — holographic memory, semantic routing, the rest — go upstream. This package is just the lexical half, for people who want pip install and a good BM25.

Quickstart

from lecore_bm25 import BM25, tokenize, reciprocal_rank_fusion

docs = [
    "smooth out the bumpy surface of a mesh",
    "denoise a grainy image with a median filter",
    "subdivide a polygon mesh into smaller pieces",
]

bm = BM25(docs)              # k1=1.5, b=0.75 (Robertson defaults)
bm.rank("bumpy surface")     # -> [(0, 4.19...), (2, 0.71...), (1, 0.0)]
bm.scores("bumpy surface")   # -> np.ndarray, one score per doc

# fuse with any other ranker (no score calibration needed)
reciprocal_rank_fusion([[0, 2, 1], [2, 0, 1]], k=60)

tokenize is the part that matters and it's exported on purpose — stoplist plus light inflectional and derivational stemming. You can hand it to any other retriever.

The numbers

BEIR via the mteb/* HuggingFace datasets, scored with pytrec_eval ndcg_cut.10 — the same scorer mteb uses underneath — 1000-doc scoring pool, ignore_identical_ids on ArguAna.

nDCG@10:

Read the fourth row before you quote the third. Against rank_bm25 with the tokenization its README actually demonstrates (doc.lower().split(), since it ships no tokenizer at all), this wins by 10.8, 5.1 and 8.5 points. But hand rank_bm25 this package's tokenize and the gap evaporates. So:

The scoring math is not better. The tokenizer is the entire advantage.

That's still a real advantage — it's the difference between what you get out of the box and what you get after you go build a stoplist and a stemmer yourself — but it is a packaging win, not an algorithmic one, and anyone telling you otherwise is selling something.

Where this used to lose: long queries (fixed in 0.2.0)

Through 0.1.0 this section documented a defeat: on ArguAna, rank_bm25 with our own tokenizer beat us by 5.4 points (0.4835 vs 0.4300). The mechanism was one line —

for t in sorted(set(q_terms)):   # 0.1.0: query terms DEDUPED
for q in query:                  # rank_bm25: every occurrence counts

— and 0.2.0 fixes it by counting query terms instead of deduping them, so a term repeated q times contributes q x its weight, which is what BM25's query-side term frequency has always meant.

We measured the fix across seven BEIR tasks before shipping it, because a change that helps one task and quietly costs the other six is not a fix:

The effect is entirely explained by how often query terms repeat. Six of seven BEIR tasks have keyword-length queries that repeat almost nothing, so deduping was invisible there — every delta is under 0.002, which is noise. ArguAna's "queries" are whole argument passages, and there it was throwing away real signal.

Be careful what you conclude from that correlation: it looks near-perfect (r = 0.998 against query length) but it is carried by a single point. Drop ArguAna and it collapses to r = 0.155. The honest reading is a bound, not a law — harmless up to ~9.5 tokens / 0.028 repeat rate, worth 5.7 points at 121.6 tokens / 0.230. Nothing we measured says where in between it starts to matter.

With the fix, this now edges rank_bm25-with-our-tokenizer on ArguAna too (0.4867 vs 0.4835), so the earlier advice to switch libraries for passage-length queries no longer applies.

Two more findings worth recording:

• The expand=True knob is noise. +0.0026 SciFact, −0.0014 NFCorpus, +0.0008 ArguAna. It is off by default and you should leave it off.
• Nothing here is "holographic." It's Robertson/Sparck-Jones BM25 with a good tokenizer.

These were independently reproduced from a fresh clone on different hardware by a tester in Moose's Telegram, matching to four decimals, before being re-run here.

Reproducing

The bench harness lives in the supercontext bench campaign (bm25_vs_pip_bench.py). It re-runs all three tasks against pip rank_bm25 and writes the table above.

API

• BM25(docs, k1=1.5, b=0.75) — docs is a list of raw strings
  • .scores(query, expand=False) → np.ndarray of length N
  • .rank(query, top=None, expand=False) → [(doc_index, score), ...] descending
• tokenize(text) → list[str]
• reciprocal_rank_fusion(ranked_lists, k=60, top=None, weights=None) → [(doc, score), ...]

License

MIT — Copyright (c) 2026 AnOversizedMooseWithSocks. See LICENSE.