zvec-db 0.17.0

Utility suite for sparse vectorization and document reranking using zvec

zvec-db


Sparse/dense vectorization and document reranking toolkit for zvec.

Requires: zvec >= 0.5.1

zvec-db provides:

• 🚀 Hybrid search - Combine sparse (BM25) and dense (semantic) vectors with full-text search
• 🔄 Score fusion - Weighted fusion and Reciprocal Rank Fusion (RRF)
• 🎯 Cross-encoder reranking - Two-stage retrieval with local or API-based rerankers
• 📦 6 sparse embedders - BM25, BM25L, BM25+, TF-IDF, Count, DisMax
• 🔧 Normalization - Bayesian, minmax, percentile, and arctan-based normalization
• ⚡ Native fast path - C++ acceleration for fusion rerankers

---

Quick Start

pip install zvec-db

Complete Example: Hybrid Search with RerankPipeline

from zvec_db import search, RerankPipeline, ScoreNormalizer, WeightedReranker
from zvec_db.rerankers import SentenceTransformerReranker
from zvec_db.embedders import BM25Embedder, SentenceTransformersEmbedder
import zvec

# 1. Open collection and initialize embedders
collection = zvec.open("./my_collection")
bm25 = BM25Embedder(max_features=4096).load("./bm25.joblib")
dense = SentenceTransformersEmbedder(model_name="all-MiniLM-L6-v2")

# 2. Create rerank pipeline with auto-detection from schema
reranker = RerankPipeline(
    normalizer=ScoreNormalizer(
        method="bayes",
        schema=collection.schema,  # Auto-detect metrics (COSINE, L2, IP)
    ),
    fusion=WeightedReranker(
        weights={"sparse": 0.5, "dense": 0.3, "text": 0.2},
        topk=16,  # Top 16 after fusion
    ),
    cross_encoder=SentenceTransformerReranker(
        model_name="cross-encoder/ms-marco-MiniLM-L-6-v2",
        topk=8,  # Top 8 after cross-encoder
    ),
)

# 3. Search with full pipeline
results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    fts_fields="text",
    reranker=reranker,
    topk=32,  # Fetch 32 candidates
)

for doc in results:
    print(f"{doc.fields['text']} (score: {doc.score:.4f})")

Simple Search (No Reranking)

from zvec_db import search

results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    topk=10,
)

Search with Filter

from zvec_db import search

# Filter with SQL-like WHERE clause
results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    filter="publish_year >= 2020 AND category == 'research'",
    topk=10,
)

Custom Fusion Only

from zvec_db.rerankers import WeightedReranker

# List weights enable C++ fast path (2-5x faster)
reranker = WeightedReranker(weights=[0.7, 0.3])

results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    reranker=reranker,
)

Search with Minimum Score

from zvec_db import search

# Filter out results with score below 0.5
results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    min_score=0.5,  # Only keep results with score >= 0.5
    topk=10,
)

---

API Reference

search() Parameters

Parameter	Type	Description
collection	zvec.Collection	Collection to search
query	str	Query text
sparse_fields	dict[str, Embedder]	Sparse fields (e.g., {"sparse": bm25})
dense_fields	dict[str, Embedder]	Dense fields (e.g., {"dense": dense})
fts_fields	str | list[str]	FTS fields
reranker	Reranker	Reranker instance
topk	int	Number of candidates to fetch (before filtering)
min_score	float	Minimum score threshold (filter out low scores)
output_fields	list[str]	Fields to include in results
filter	str	Conditional filter (SQL-like WHERE clause)

Reranker Types

Type	Description
WeightedReranker(weights={...})	Weighted fusion
RrfReranker()	Reciprocal Rank Fusion
SentenceTransformerReranker()	Cross-encoder
RerankPipeline(normalizer=..., fusion=..., cross_encoder=...)	Full pipeline

---

Embedders

Sparse (lexical): BM25Embedder (recommended), BM25LEmbedder, BM25PlusEmbedder, TfidfEmbedder, CountEmbedder, DisMaxEmbedder

Dense (semantic): SentenceTransformersEmbedder, OpenAIEmbedder

from zvec_db import BM25Embedder, SentenceTransformersEmbedder

bm25 = BM25Embedder(max_features=4096).load("./bm25.joblib")
dense = SentenceTransformersEmbedder(model_name="all-MiniLM-L6-v2")

bm25.save("bm25.joblib")

Rerankers

Fusion: WeightedReranker (weighted sum), RrfReranker (rank-based)

Cross-Encoder: SentenceTransformerReranker, OpenAIReranker

Pipeline: RerankPipeline (normalization + fusion + cross-encoder)

from zvec_db import RerankPipeline, ScoreNormalizer, WeightedReranker
from zvec_db.rerankers import SentenceTransformerReranker

# Full pipeline with schema auto-detection
pipeline = RerankPipeline(
    normalizer=ScoreNormalizer(method="bayes", schema=collection.schema),
    fusion=WeightedReranker(weights={"sparse": 0.7, "dense": 0.3}),
    cross_encoder=SentenceTransformerReranker(topk=10),
)

Performance tip: Use list weights for C++ fast path: WeightedReranker(weights=[0.7, 0.3])

---

Documentation

• User Guide — Complete examples
• API Reference — Full API docs

---

License

MIT License