zvec-db 0.8.0

Utility suite for sparse vectorization and document reranking using zvec

zvec-db

Sparse/dense vectorization and document reranking toolkit for zvec.

---

Quick Start

pip install zvec-db

---

Available Embedders & Rerankers

Embedders

Sparse (lexical search):

Embedder	Description
BM25Embedder	Standard BM25 scoring (recommended for most use cases)
BM25LEmbedder	BM25L variant for documents with highly variable lengths
BM25PlusEmbedder	BM25+ with delta smoothing to avoid zero scores
TfidfEmbedder	TF-IDF weighting with optional sublinear TF
CountEmbedder	Simple term counts (binary option available)
DisMaxEmbedder	Multi-field search using maximum score

Dense (semantic search):

Embedder	Description
SentenceTransformersEmbedder	Local sentence-transformers models (e.g., all-MiniLM-L6-v2)
OpenAIEmbedder	OpenAI API embeddings (e.g., text-embedding-3-small)

Rerankers

Fusion (combine multiple sources):

Reranker	Description
WeightedReranker	Weighted score fusion with automatic metric handling
RrfReranker	Reciprocal Rank Fusion (rank-based, robust to score scale differences)
MultiFieldWeightedReranker	Weighted fusion for multi-field schemas

Cross-Encoder (query + document scoring):

Reranker	Description
SentenceTransformerReranker	Local cross-encoder models (e.g., ms-marco-MiniLM-L-6-v2)
ClassificationReranker	Multi-class classification reranking
OpenAIReranker	OpenAI API-based reranking
OpenAIDecoderReranker	OpenAI decoder-style reranking

Utilities:

Utility	Description
PipelineReranker	Chain multiple rerankers in sequence
Normalize	Score normalization (bayes, minmax, rank, percentile)

---

Quick Start (5 minutes)

1. Index documents

from zvec_db.embedders import BM25Embedder, SentenceTransformersEmbedder

documents = [
    "Machine learning is a subset of AI",
    "Deep learning uses neural networks",
    "NLP helps computers understand text",
]

# Create embedders
bm25 = BM25Embedder(max_features=4096)
bm25.fit(documents)

dense = SentenceTransformersEmbedder(model_name="all-MiniLM-L6-v2")

# Encode documents
for doc in documents:
    sparse_vec = bm25.embed(doc)   # dict: {index: score}
    dense_vec = dense.embed(doc)   # numpy array

BM25 variants and CountVectorizer parameters

Standard BM25 (unigrams only - FTS):

bm25 = BM25Embedder(
    max_features=4096,
    k1=1.2,  # Term frequency saturation
    b=0.75,  # Length normalization
)

BM25 with trigrams (character n-grams for fuzzy matching):

bm25_trigram = BM25Embedder(
    max_features=8192,
    k1=1.2,
    b=0.75,
    analyzer="char_wb",      # Character-level n-grams (or "char")
    ngram_range=(3, 3),      # Trigrams only
    min_df=2,                # Minimum document frequency
)

BM25 with mixed unigrams + trigrams:

bm25_mixed = BM25Embedder(
    max_features=16384,
    analyzer="word",
    ngram_range=(1, 1),      # Word unigrams only
)

# Or combine word + char n-grams by using char_wb:
bm25_fuzzy = BM25Embedder(
    max_features=16384,
    analyzer="char_wb",      # Character n-grams at word boundaries
    ngram_range=(2, 4),      # Bi-grams, trigrams, 4-grams
)

Other CountVectorizer parameters you can use:

bm25 = BM25Embedder(
    max_features=4096,
    # Tokenization
    tokenizer=lambda x: x.split(),  # Custom tokenizer
    token_pattern=r"(?u)\b\w+\b",   # Regex pattern for tokens
    # Vocabulary filtering
    min_df=2,            # Minimum document frequency
    max_df=0.8,          # Maximum document frequency (removes common terms)
    max_features=10000,  # Max vocabulary size
    # N-grams
    ngram_range=(1, 2),  # Unigrams + bigrams
    analyzer="word",     # "word", "char", or "char_wb"
    # Preprocessing
    lowercase=True,      # Convert to lowercase
    stop_words="english",# Remove English stopwords
)

2. Search with hybrid + reranking

from zvec.model.doc import Doc
from zvec_db.rerankers import WeightedReranker
from zvec.typing import MetricType

query = "neural networks"

# Simulated search results from different retrievers
bm25_results = [
    Doc(id="0", score=1.2),
    Doc(id="1", score=0.9),
    Doc(id="2", score=0.6),
]
dense_results = [
    Doc(id="1", score=0.85),
    Doc(id="0", score=0.75),
    Doc(id="3", score=0.65),
]

# Option 1: With explicit metrics
reranker = WeightedReranker(
    weights={"bm25": 0.4, "dense": 0.6},
    metrics={"bm25": MetricType.IP, "dense": MetricType.COSINE},
    normalize=True,  # Smart default: COSINE->/2, others->bayes
)

# Option 2: With schema auto-detection (recommended with zvec)
# import zvec
# collection = zvec.open("./my_collection")
# reranker = WeightedReranker(
#     schema=collection.schema,  # Auto-detect metrics from schema
#     weights={"bm25": 0.4, "dense": 0.6},
#     normalize=True,
# )

results = reranker.rerank({
    "bm25": bm25_results,
    "dense": dense_results,
})

print(results[0].id)  # Most relevant document

---

Table of Contents

• Key Concepts
• Advanced Example
• Reranking
• Pipeline Example
• Cross-Encoder Rerankers
• Preprocessing
• Model Persistence
• Development

---

Key Concepts

Distance vs Similarity Metrics

Vector databases store distances (smaller = more similar), but fusion algorithms assume similarities (larger = more relevant). The metrics parameter handles conversion automatically:

Metric	Type	Range	Conversion	Usage
COSINE	Distance	[0, 2]	(2 - score) / 2	Normalized embeddings (Qdrant, zvec)
L2	Distance	[0, ∞)	-score	Euclidean distance
IP	Similarity	(-∞, ∞)	None	Inner product, BM25 scores (already similar)

Default: MetricType.COSINE (main use case with zvec/Qdrant).

Choosing a Sparse Embedder

Embedder	Use case
BM25Embedder	Recommended - standard lexical search
TfidfEmbedder	TF-IDF weighting with sublinear TF option
CountEmbedder	Simple term counts (binary option available)
BM25LEmbedder	Documents with highly variable lengths
BM25PlusEmbedder	Avoid zero scores with delta smoothing
DisMaxEmbedder	Multi-field search (takes maximum score)

---

Advanced Example: Hybrid Search with zvec

import zvec
from zvec.model.doc import Doc
from zvec_db.embedders import BM25Embedder, SentenceTransformersEmbedder
from zvec_db.rerankers import WeightedReranker

documents = [
    "Machine learning is a subset of AI",
    "Deep learning uses neural networks",
    "NLP helps computers understand text",
]

# Create embedders
bm25 = BM25Embedder(max_features=4096)
bm25.fit(documents)
dense = SentenceTransformersEmbedder(model_name="all-MiniLM-L6-v2")

# Create collection
schema = zvec.CollectionSchema(
    name="docs",
    fields=[zvec.FieldSchema("text", zvec.DataType.STRING)],
    vectors=[
        zvec.VectorSchema(name="sparse", data_type=zvec.DataType.SPARSE_VECTOR_FP32, dimension=4096),
        zvec.VectorSchema(
            name="dense", 
            data_type=zvec.DataType.VECTOR_FP32, 
            dimension=384,
            index_param=zvec.FlatIndexParam(metric_type=zvec.MetricType.COSINE)
        ),
    ]
)
collection = zvec.create_and_open("./my_db", schema)

# Index documents
for i, doc in enumerate(documents):
    collection.insert(Doc(
        id=str(i),
        fields={"text": doc},
        vectors={
            "sparse": bm25.embed(doc),
            "dense": dense.embed(doc),
        }
    ))

# Search with auto-detected metrics from schema
reranker = WeightedReranker(
    topn=3,
    schema=collection.schema,  # Auto-detect metrics: sparse->None, dense->COSINE
    weights={"sparse": 0.4, "dense": 0.6},
    normalize=True,  # Smart default: sparse->bayes, dense->/2
)

query = "neural networks"
results = collection.query(
    vectors=[
        zvec.VectorQuery(field_name="sparse", vector=bm25.embed(query)),
        zvec.VectorQuery(field_name="dense", vector=dense.embed(query)),
    ],
    topk=10,
    reranker=reranker,
)

print("Top results:")
for i, doc in enumerate(results[:3]):
    print(f"  {i+1}. {doc.fields['text']} (score: {doc.score:.4f})")

---

Reranking

Normalization

The normalize parameter controls score normalization:

Value	Effect
True	Smart default: COSINE → no-op, others → "bayes"
"bayes"	Bayesian sigmoid calibration (robust to outliers)
"minmax"	Min-max: (x - min) / (max - min)
"rank" / "percentile"	Rank-based (very robust to outliers)
"cosine"	No-op (identity). COSINE scores already in [0, 1]
{"sparse": "bayes", "dense": "cosine"}	Per-source configuration
None / False	No normalization

Note: normalize=True requires schema or metrics to auto-detect the metric per source.

COSINE is already normalized to [0, 1] by the conversion formula (2 - score) / 2, so normalize="cosine" is a no-op (identity). Use it for explicit API consistency when you want to document that no additional normalization is applied.

WeightedReranker

Weighted fusion of multiple sources:

from zvec_db.rerankers import WeightedReranker
from zvec.typing import MetricType

# With explicit metrics
reranker = WeightedReranker(
    weights={"bm25": 0.4, "dense": 0.6},
    metrics={"bm25": MetricType.IP, "dense": MetricType.COSINE},
    normalize="bayes",
)

# With schema auto-detection (recommended)
import zvec
collection = zvec.open("./my_collection")
reranker = WeightedReranker(
    schema=collection.schema,
    weights={"sparse": 0.4, "dense": 0.6},
    normalize=True,
)

results = reranker.rerank({"bm25": bm25_docs, "dense": dense_docs})

RrfReranker (Reciprocal Rank Fusion)

Rank-based fusion (robust to score scale differences):

from zvec_db.rerankers import RrfReranker
from zvec.model.doc import Doc
import zvec

# Search results from different retrievers
bm25_results = [Doc(id="1", score=1.5), Doc(id="2", score=1.2)]
dense_results = [Doc(id="2", score=0.9), Doc(id="3", score=0.8)]

# With schema auto-detection (recommended)
collection = zvec.open("./my_collection")
reranker = RrfReranker(
    topn=10,
    rank_constant=60,
    schema=collection.schema,  # Auto-detect metrics
)
results = reranker.rerank({
    "bm25": bm25_results,
    "dense": dense_results,
})

# With custom weights
reranker = RrfReranker(
    topn=10,
    rank_constant=60,
    weights={"dense": 0.7, "bm25": 0.3},
    schema=collection.schema,
)

Note: RRF uses ranks, not scores. The normalize parameter has no effect.

---

Cross-Encoder Rerankers

Cross-encoders recalculate scores using both query and document. Require a query parameter.

from zvec_db.rerankers import SentenceTransformerReranker

reranker = SentenceTransformerReranker(
    query="machine learning",
    model_name="cross-encoder/ms-marco-MiniLM-L-6-v2",
    topn=10,
)
results = reranker.rerank({"bm25": docs})

Other cross-encoders: ClassificationReranker (multi-class), OpenAIReranker (API).

---

Pipeline Example

Chain multiple rerankers (RRF fusion + Cross-Encoder refinement):

from zvec_db.rerankers import PipelineReranker, RrfReranker, SentenceTransformerReranker

pipeline = PipelineReranker(
    rerankers=[
        RrfReranker(
            topn=32, 
            rank_constant=60, 
            schema=collection.schema
        ),  # First: RRF fusion
        SentenceTransformerReranker(
            topn=16,
            model_name="cross-encoder/ms-marco-MiniLM-L-6-v2"
        )  # Then: Cross-Encoder re-scoring
    ],
    topn=10
)

results = pipeline.rerank({
    "bm25": bm25_results,
    "dense": dense_results,
}, query="neural networks")

---

Preprocessing

Preprocessing improves sparse embedding quality:

from zvec_db.embedders import BM25Embedder
from zvec_db.preprocessing import NormalizationConfig

config = NormalizationConfig.aggressive(language="english")
bm25 = BM25Embedder(max_features=4096, preprocessing_config=config)
bm25.fit(documents)

# Utility functions
from zvec_db.preprocessing import normalize_text, stem_word, remove_stopwords
normalize_text("  HELLO WORLD  ", lowercase=True, stem=True)  # "hello world"

Install: pip install "zvec-db[preprocessing]"

---

Model Persistence

from zvec_db.embedders import BM25Embedder

# Save
bm25 = BM25Embedder(max_features=4096, preprocessing_config=config)
bm25.fit(documents)
bm25.save("models/bm25_model.joblib")

# Load
bm25_loaded = BM25Embedder()
bm25_loaded.load("models/bm25_model.joblib")

---

Development

# Clone and install
git clone https://github.com/ccdv-ai/zvec-db.git
cd zvec-db
pip install -e ".[dev,test,docs,preprocessing]"

# Run tests
make test

# Lint
make lint      # black, isort, flake8, mypy

# Build docs
make docs

---

License

MIT License