Skip to main content
Yanked

This release has been yanked by its maintainers, and will be ignored by installers, except when explicitly specified.
Reason given by maintainers: bm25-chroma is correct name

Enhanced Hybrid Retriever

A fast, memory-efficient hybrid search system combining optimized BM25 and vector search with Reciprocal Rank Fusion (RRF).

Features

  • Optimized BM25: Memory-efficient with integer indices and pre-sorted postings
  • Vector Search: Semantic similarity using ChromaDB and sentence transformers
  • Hybrid Fusion: Industry-standard Reciprocal Rank Fusion (RRF)
  • Dual Processing Modes: Sequential or unified batch processing
  • State Persistence: Automatic save/load of BM25 index

Quick Start

import sys
import os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from hybrid_retriever import HybridRetriever as HybridRetriever

# Initialize
retriever = HybridRetriever(
    chroma_path="./my_db",
    collection_name="my_docs"
)

# Add documents
documents = [
    "Machine learning helps analyze data patterns.",
    "Natural language processing understands human text.",
    "Deep learning uses neural networks for complex tasks."
]

retriever.add_documents_batch(
    documents,
    mode="unified",  # or "sequential"
    show_progress=True
)

# Search
results = retriever.hybrid_search("machine learning", top_k=5)
for doc_id, score, metadata in results:
    print(f"{doc_id}: {score:.3f} - {metadata['text'][:100]}...")

Installation

pip install -r requirements.txt

Testing

Run tests to verify functionality:

pytest tests/

Or run directly:

python tests/test_examples.py

Examples

  • examples/basic_usage.py - Simple example with custom documents
  • examples/brown_corpus_demo.py - Full demo with Brown corpus

Processing Modes

Unified Mode (Recommended)

  • Processes both BM25 and ChromaDB together
  • Usually faster for large datasets
  • Better for production use

Sequential Mode

  • Processes ChromaDB first, then BM25
  • Better for debugging and optimization
  • Separate timing for each system

Performance

The system is designed for efficiency through incremental operations:

No Full Recalculation: Adding or removing documents updates only affected components:

  • Vocabulary set adds/removes only new/orphaned terms
  • Inverted index updates only posting lists for changed terms
  • Document statistics incrementally adjust averages and counts

Python Native Libraries: Heavy lifting handled by optimized built-ins:

  • Counter.most_common() provides pre-sorted frequency lists
  • heapq.merge() efficiently combines sorted posting lists
  • set operations for O(1) vocabulary lookups and updates
  • defaultdict(Counter) for sparse term-document matrices

Batch Processing: Configurable batch sizes balance memory usage and processing speed:

  • Pending additions buffer reduces index update frequency
  • Automatic flush mechanism maintains data consistency
  • Progress tracking for large document collections

API

Main Classes

  • BM25: Fast BM25 implementation
  • HybridRetriever: Main hybrid search interface

Key Methods

  • add_documents_batch(): Add documents in batches
  • search_bm25(): BM25-only search
  • search_vector(): Vector-only search
  • hybrid_search(): Combined search with RRF
  • get_system_stats(): Performance statistics

License

MIT

Metadata

Release files for hybrid-retriever 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hybrid-retriever 0.1.0
File Size Uploaded Provenance Actions
hybrid_retriever-0.1.0.tar.gz 7.6 kB Without provenance Details for hybrid_retriever-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for hybrid-retriever 0.1.0
File Interpreter ABI Platform Actions
hybrid_retriever-0.1.0-py3-none-any.whl Python 3 none any Details for hybrid_retriever-0.1.0-py3-none-any.whl

Total release size: 13.3 kB

Release files / hybrid_retriever-0.1.0.tar.gz

Download URL hybrid_retriever-0.1.0.tar.gz
Size 7.6 kB
Tags Source
SHA-256 checksum
How to use checksums
e921fe1850891bc304de4840ac74a761ee9ac8ad85d65d739e257119ae38b44d
BLAKE2b-256 checksum
How to use checksums
654e8e7ce1001fa362fa61fe20e016984dec9b04398f51f5e797785cf6df3eb0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.10.0

Release files / hybrid_retriever-0.1.0-py3-none-any.whl

Download URL hybrid_retriever-0.1.0-py3-none-any.whl
Size 5.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
2c7a64e484fddb69751ab59a43a725a0697ff9a35c748374dc6b8d218cb8215e
BLAKE2b-256 checksum
How to use checksums
f3f0f2b6f89b859334fc7d6d8759223b72da083643abc793ea012a714f902a51
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.10.0

Security for hybrid-retriever 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page