zvec-db


Sparse/dense vectorization and document reranking toolkit for zvec.

Requires: zvec >= 0.7.0

zvec-db provides:

• 🚀 Hybrid search - Combine sparse (BM25) and dense (semantic) vectors with full-text search
• 🔄 Score fusion - Weighted fusion and Reciprocal Rank Fusion (RRF)
• 🎯 Cross-encoder reranking - Two-stage retrieval with local or API-based rerankers
• 📦 6 sparse embedders - BM25, BM25L, BM25+, TF-IDF, Count, DisMax
• 🔧 Normalization - Bayesian, minmax, percentile, and arctan-based normalization
• ⚡ Native fast path - C++ acceleration for fusion rerankers

Quick Start

pip install zvec-db

Complete Example: Hybrid Search with RerankPipeline

from zvec_db import search, RerankPipeline, ScoreNormalizer, WeightedReranker
from zvec_db.rerankers import SentenceTransformerReranker
from zvec_db.embedders import BM25Embedder, SentenceTransformersEmbedder
import zvec

# 1. Open collection and initialize embedders
collection = zvec.open("./my_collection")
bm25 = BM25Embedder(max_features=4096).load("./bm25.joblib")
dense = SentenceTransformersEmbedder(model_name="all-MiniLM-L6-v2")

# 2. Create rerank pipeline with auto-detection from schema
reranker = RerankPipeline(
    normalizer=ScoreNormalizer(
        method="bayes",
        schema=collection.schema,  # Auto-detect metrics (COSINE, L2, IP)
    ),
    fusion=WeightedReranker(
        weights={"sparse": 0.5, "dense": 0.3, "text": 0.2},
        topk=16,  # Top 16 after fusion
    ),
    cross_encoder=SentenceTransformerReranker(
        model_name="cross-encoder/ms-marco-MiniLM-L-6-v2",
        topk=8,  # Top 8 after cross-encoder
    ),
)

# 3. Search with full pipeline
results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    fts_fields="text",
    reranker=reranker,
    topk=32,  # Fetch 32 candidates
)

for doc in results:
    print(f"{doc.fields['text']} (score: {doc.score:.4f})")

Simple Search (No Reranking)

from zvec_db import search

results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    topk=10,
)

Search with Filter

from zvec_db import search

# Filter with SQL-like WHERE clause
results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    filter="publish_year >= 2020 AND category == 'research'",
    topk=10,
)

Custom Fusion Only

from zvec_db.rerankers import WeightedReranker

# List weights enable C++ fast path (2-5x faster)
reranker = WeightedReranker(weights=[0.7, 0.3])

results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    reranker=reranker,
)

Search with Minimum Score

from zvec_db import search

# Filter out results with score below 0.5
results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    min_score=0.5,  # Only keep results with score >= 0.5
    topk=10,
)

Full-Text Search with the Native ngram Tokenizer

zvec ships native C++ tokenizers for its full-text search (FTS) indexes: standard (default), ngram, jieba, and whitespace. The ngram tokenizer indexes every substring of n characters, enabling partial-word and typo-tolerant matching (e.g. searching "vector" matches "vectors", "vectorize") and works on any language without a dictionary.

Configure it on the FTS field when creating the collection with FtsIndexParam:

import zvec

schema = zvec.CollectionSchema(
    name="my_collection",
    fields=[
        zvec.FieldSchema(
            name="text",
            data_type=zvec.DataType.STRING,
            index_param=zvec.FtsIndexParam(
                tokenizer_name="ngram",
                filters=["lowercase"],
                extra_params='{"ngram_min": 2, "ngram_max": 3, "token_chars": ["letter", "digit"]}',
            ),
        )
    ],
)
collection = zvec.create_and_open("./my_collection", schema)

Then search the field with fts_fields="text" as usual — the query is tokenized with the same ngram pipeline at query time:

results = search(
    collection=collection,
    query="vector",
    fts_fields="text",
    topk=10,
)

extra_params keys for the ngram tokenizer:

Note: Larger ngram_max values increase index size. For most use cases {"ngram_min": 2, "ngram_max": 3} is a good trade-off between recall and index footprint. The other native tokenizers (standard, jieba for Chinese, whitespace) accept their own extra_params — see the zvec FtsIndexParam documentation.

API Reference

search() Parameters

Reranker Types

Embedders

Sparse (lexical): BM25Embedder (recommended), BM25LEmbedder, BM25PlusEmbedder, TfidfEmbedder, CountEmbedder, DisMaxEmbedder

Dense (semantic): SentenceTransformersEmbedder, OpenAIEmbedder

from zvec_db import BM25Embedder, SentenceTransformersEmbedder

bm25 = BM25Embedder(max_features=4096).load("./bm25.joblib")
dense = SentenceTransformersEmbedder(model_name="all-MiniLM-L6-v2")

bm25.save("bm25.joblib")

Rerankers

Fusion: WeightedReranker (weighted sum), RrfReranker (rank-based)

Cross-Encoder: SentenceTransformerReranker, OpenAIReranker

Pipeline: RerankPipeline (normalization + fusion + cross-encoder)

from zvec_db import RerankPipeline, ScoreNormalizer, WeightedReranker
from zvec_db.rerankers import SentenceTransformerReranker

# Full pipeline with schema auto-detection
pipeline = RerankPipeline(
    normalizer=ScoreNormalizer(method="bayes", schema=collection.schema),
    fusion=WeightedReranker(weights={"sparse": 0.7, "dense": 0.3}),
    cross_encoder=SentenceTransformerReranker(topk=10),
)

Performance tip: Use list weights for C++ fast path: WeightedReranker(weights=[0.7, 0.3])

Documentation

• User Guide — Complete examples
• API Reference — Full API docs

License

MIT License