This release has been yanked by its maintainers, and will be ignored by installers, except when explicitly specified.
Reason given by maintainers: bm25-chroma is correct name
Enhanced Hybrid Retriever
A fast, memory-efficient hybrid search system combining optimized BM25 and vector search with Reciprocal Rank Fusion (RRF).
Features
- Optimized BM25: Memory-efficient with integer indices and pre-sorted postings
- Vector Search: Semantic similarity using ChromaDB and sentence transformers
- Hybrid Fusion: Industry-standard Reciprocal Rank Fusion (RRF)
- Dual Processing Modes: Sequential or unified batch processing
- State Persistence: Automatic save/load of BM25 index
Quick Start
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from hybrid_retriever import HybridRetriever as HybridRetriever
# Initialize
retriever = HybridRetriever(
chroma_path="./my_db",
collection_name="my_docs"
)
# Add documents
documents = [
"Machine learning helps analyze data patterns.",
"Natural language processing understands human text.",
"Deep learning uses neural networks for complex tasks."
]
retriever.add_documents_batch(
documents,
mode="unified", # or "sequential"
show_progress=True
)
# Search
results = retriever.hybrid_search("machine learning", top_k=5)
for doc_id, score, metadata in results:
print(f"{doc_id}: {score:.3f} - {metadata['text'][:100]}...")
Installation
pip install -r requirements.txt
Testing
Run tests to verify functionality:
pytest tests/
Or run directly:
python tests/test_examples.py
Examples
examples/basic_usage.py- Simple example with custom documentsexamples/brown_corpus_demo.py- Full demo with Brown corpus
Processing Modes
Unified Mode (Recommended)
- Processes both BM25 and ChromaDB together
- Usually faster for large datasets
- Better for production use
Sequential Mode
- Processes ChromaDB first, then BM25
- Better for debugging and optimization
- Separate timing for each system
Performance
The system is designed for efficiency through incremental operations:
No Full Recalculation: Adding or removing documents updates only affected components:
- Vocabulary set adds/removes only new/orphaned terms
- Inverted index updates only posting lists for changed terms
- Document statistics incrementally adjust averages and counts
Python Native Libraries: Heavy lifting handled by optimized built-ins:
Counter.most_common()provides pre-sorted frequency listsheapq.merge()efficiently combines sorted posting listssetoperations for O(1) vocabulary lookups and updatesdefaultdict(Counter)for sparse term-document matrices
Batch Processing: Configurable batch sizes balance memory usage and processing speed:
- Pending additions buffer reduces index update frequency
- Automatic flush mechanism maintains data consistency
- Progress tracking for large document collections
API
Main Classes
BM25: Fast BM25 implementationHybridRetriever: Main hybrid search interface
Key Methods
add_documents_batch(): Add documents in batchessearch_bm25(): BM25-only searchsearch_vector(): Vector-only searchhybrid_search(): Combined search with RRFget_system_stats(): Performance statistics
License
MIT
Metadata
Release files for hybrid-retriever 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| hybrid_retriever-0.1.0.tar.gz | 7.6 kB | Without provenance | Details for hybrid_retriever-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| hybrid_retriever-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for hybrid_retriever-0.1.0-py3-none-any.whl |
Total release size: 13.3 kB
Release files / hybrid_retriever-0.1.0.tar.gz
| Download URL | hybrid_retriever-0.1.0.tar.gz |
|---|---|
| Size | 7.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e921fe1850891bc304de4840ac74a761ee9ac8ad85d65d739e257119ae38b44d
|
|
BLAKE2b-256 checksum How to use checksums |
654e8e7ce1001fa362fa61fe20e016984dec9b04398f51f5e797785cf6df3eb0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.10.0
|
Release files / hybrid_retriever-0.1.0-py3-none-any.whl
| Download URL | hybrid_retriever-0.1.0-py3-none-any.whl |
|---|---|
| Size | 5.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2c7a64e484fddb69751ab59a43a725a0697ff9a35c748374dc6b8d218cb8215e
|
|
BLAKE2b-256 checksum How to use checksums |
f3f0f2b6f89b859334fc7d6d8759223b72da083643abc793ea012a714f902a51
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.10.0
|
Security for hybrid-retriever 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.