zvec-db 0.10.0

Utility suite for sparse vectorization and document reranking using zvec

zvec-db

Sparse/dense vectorization and document reranking toolkit for zvec.

---

Quick Start

pip install zvec-db

1. Basic hybrid search with search()

from zvec_db import search, BM25Embedder, SentenceTransformersEmbedder
import zvec

collection = zvec.open("./my_collection")
bm25 = BM25Embedder(max_features=4096).load("./bm25.joblib")
dense = SentenceTransformersEmbedder(model_name="all-MiniLM-L6-v2")

# Hybrid search: sparse + dense + FTS
results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    fts_fields="text",
    fusion="auto",
    topk=10,
)

2. Two-stage retrieval with cross-encoder

from zvec_db.rerankers import SentenceTransformerReranker

reranker = SentenceTransformerReranker(
    model_name="cross-encoder/ms-marco-MiniLM-L-6-v2",
    topk=10,
)

results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    fts_fields="text",
    fusion="auto",
    reranker=reranker,
    topk=50,  # Fetch 50 candidates, rerank to 10
)

3. Custom fusion reranker (e.g., RRF)

from zvec_db.rerankers import RrfReranker

fusion_model = RrfReranker(topk=50, rank_constant=60)

results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    fusion=fusion_model,  # Pre-instantiated fusion reranker
    topk=50,
)

Parameters: collection, query, sparse_fields, dense_fields, fts_fields, fusion ("auto" | dict | reranker instance | None), weights, reranker, topk, output_fields.

Note: Configure topk and cross_encoder_weight directly on the reranker instance.

---

Embedders

Sparse (lexical search)

Embedder	Description
BM25Embedder	Recommended — Standard BM25 scoring
BM25LEmbedder	BM25L for documents with variable lengths
BM25PlusEmbedder	BM25+ with delta smoothing
TfidfEmbedder	TF-IDF weighting
CountEmbedder	Simple term counts
DisMaxEmbedder	Multi-field search (maximum score)

Example:

from zvec_db import BM25Embedder

bm25 = BM25Embedder(max_features=4096, k1=1.2, b=0.75)
bm25.fit(documents)

# Embed query
query_vector = bm25.embed("machine learning")

# Save/load
bm25.save("bm25.joblib")
bm25_loaded = BM25Embedder().load("bm25.joblib")

Dense (semantic search)

Embedder	Description
SentenceTransformersEmbedder	Local models (e.g., all-MiniLM-L6-v2)
OpenAIEmbedder	OpenAI API (text-embedding-3-small)

Example:

from zvec_db import SentenceTransformersEmbedder

dense = SentenceTransformersEmbedder(model_name="all-MiniLM-L6-v2")
vector = dense.embed("machine learning")

---

Rerankers

Fusion (combine multiple sources)

Reranker	Description
WeightedReranker	Weighted score fusion with normalization
RrfReranker	Reciprocal Rank Fusion (rank-based)
MultiFieldWeightedReranker	Field-based weighted fusion

Example — WeightedReranker:

from zvec_db.rerankers import WeightedReranker

reranker = WeightedReranker(
    schema=collection.schema,
    weights={"sparse": 0.4, "dense": 0.6},
    normalize="auto",
    topk=10,
)
results = reranker.rerank({"sparse": sparse_docs, "dense": dense_docs})

Example — RrfReranker:

from zvec_db.rerankers import RrfReranker

reranker = RrfReranker(topk=10, rank_constant=60)
results = reranker.rerank({"sparse": sparse_docs, "dense": dense_docs})

Cross-Encoder (query + document scoring)

Reranker	Description
SentenceTransformerReranker	Local cross-encoder models
ClassificationReranker	Multi-class classification
OpenAIReranker	OpenAI API-based reranking

Example:

from zvec_db.rerankers import SentenceTransformerReranker

reranker = SentenceTransformerReranker(
    model_name="cross-encoder/ms-marco-MiniLM-L-6-v2",
    topk=10,
    cross_encoder_weight=0.8,  # Blend: 80% cross-encoder, 20% fusion
)
results = reranker.rerank({"bm25": docs}, query="machine learning")

Pipeline (chain multiple rerankers)

from zvec_db.rerankers import PipelineReranker, RrfReranker, SentenceTransformerReranker

pipeline = PipelineReranker(
    rerankers=[
        RrfReranker(topk=32, rank_constant=60),  # RRF fusion
        SentenceTransformerReranker(topk=10),    # Cross-encoder
    ],
    topk=10,
    query="neural networks",
)
results = pipeline.rerank({"sparse": sparse_docs, "dense": dense_docs})

---

Preprocessing

Improve sparse embedding quality with text normalization:

from zvec_db.embedders import BM25Embedder
from zvec_db.preprocessing import NormalizationConfig

config = NormalizationConfig.aggressive(language="english")
bm25 = BM25Embedder(max_features=4096, preprocessing_config=config)
bm25.fit(documents)

Install: pip install "zvec-db[preprocessing]"

Utilities: normalize_text(), stem_word(), remove_stopwords()

---

Advanced Usage

Manual query building

For full control, use multi_field_queries():

from zvec_db import multi_field_queries

queries = multi_field_queries(
    query_text="neural networks",
    sparse_fields={"sparse_title": bm25, "sparse_content": bm25},
    dense_fields={"dense_title": dense, "dense_content": dense},
    fts_fields=["title", "content"],
)

results = collection.query(queries=queries, topk=10, output_fields=["text"])

FTS with phrase matching

from zvec_db import fts_query

q = fts_query(field_name="title", query="deep learning", match_string="deep learning")
results = collection.query(queries=[q], topk=10)

---

Documentation

For detailed examples and API reference:

• User Guide — Hybrid search, FTS, reranking
• API Reference — Complete API docs

---

Development

git clone https://github.com/ccdv-ai/zvec-db.git
cd zvec-db
pip install -e ".[dev,test,docs,preprocessing]"

make test   # Run tests
make lint   # black, isort, flake8, mypy
make docs   # Build documentation

---

License

MIT License