YaBM25 - Python BM25 Search Engine

Fast, scalable BM25 search engine implementation in Python with both in-memory and disk-based indexing. Perfect for RAG (Retrieval Augmented Generation), information retrieval, and search applications.

Key Features

• 🚀 High Performance: Optimized implementation with vectorized operations
• 💾 Memory Efficient: Optional disk-based indexing for large datasets
• 🔄 rank_bm25 Compatible: Drop-in replacement for rank_bm25 with extended features
• 📊 Multiple Variants: Supports BM25, BM25L, BM25Adpt
• 🛠 Production Ready: Thread-safe with proper resource management
• 📦 Easy Integration: Works with LangChain, LlamaIndex, and other RAG frameworks

Benchmarks

Installation

pip install yabm25

Quick Start

Simple In-Memory Usage

from yabm25 import BM25Indexer

# Initialize with corpus
corpus = [
    "Hello there good man!",
    "It is quite windy in London",
    "How is the weather today?"
]
tokenized_corpus = [doc.split(" ") for doc in corpus]
bm25 = BM25Indexer(tokenized_corpus)

# Search
query = "windy London"
doc_scores = bm25.get_scores(query.split(" "))
print(doc_scores)  # array([0., 0.93729472, 0.])

# Get top document
top_docs = bm25.get_top_n(query.split(" "), corpus, n=1)
print(top_docs)  # ['It is quite windy in London']

Large-Scale Usage

from yabm25 import BM25Indexer, BM25Config

# Configure disk-based index
config = BM25Config(
    index_dir="my_index",
    doc_chunk_size=500_000,
    compression="ZSTD"
)

# Build index
indexer = BM25Indexer(config)
indexer.build_index(large_corpus)

# Search
results = indexer.query(["term1", "term2"])

Documentation

• Examples
• API Reference
• Performance Guide

Use Cases

• 🤖 RAG Applications: Enhance LLM responses with relevant context
• 🔍 Search Systems: Build powerful document search engines
• 📚 Information Retrieval: Academic and research applications
• 📊 Text Analysis: Document similarity and ranking

Comparison with Alternatives

Citation

@software{yabm25,
  title = {YaBM25: Yet Another BM25 Implementation},
  author = {Muhammad, Ali},
  year = {2025},
  url = {https://github.com/alimuhammadofficial/yabm25}
}

Contributing

Contributions welcome! See CONTRIBUTING.md for guidelines.

License

MIT License. See LICENSE for details.