VectorBench-FAISS

A benchmarking and experimentation framework for FAISS vector search.

VectorBench-FAISS lets you build and compare vector search systems using recall@k, latency, and memory, with support for vector, BM25, hybrid retrieval, and metadata filtering.

The goal is to make the components of vector search inspectable and measurable rather than hiding everything behind a single similarity_search() call.

Features

• FAISS vector search — IVF, HNSW, and PQ indexes
• BM25 retrieval — keyword-based search without embeddings
• Hybrid retrieval — vector + BM25 with Reciprocal Rank Fusion
• Metadata filtering — filter results using document metadata
• Benchmarking — recall@k, latency, and memory measurements
• Dataset utilities — dataset downloading, chunking, and embedding preparation
• Consistent interface — experiment with different index types through the same search API

---

Installation

Install the latest release from PyPI:

pip install vectorbench_faiss

Then import the main search engine:

from vectorbench_faiss.engine import VectorSearchEngine

---

Quick Start

from vectorbench_faiss.engine import VectorSearchEngine

documents = [
    "Python is a programming language.",
    "FAISS is a library for efficient similarity search.",
    "Vector databases are commonly used in RAG systems.",
]

metadatas = [
    {"source": "python"},
    {"source": "faiss"},
    {"source": "rag"},
]

engine = VectorSearchEngine(index_type="hnsw")

engine.add_documents(
    documents,
    metadatas=metadatas
)

results = engine.search(
    "What is FAISS?",
    k=5,
    mode="vector"
)

for result in results:
    print(result["score"], result["text"])

---

Search Modes

Vector Search

Embedding-based nearest-neighbor retrieval using the selected FAISS index.

results = engine.search(
    "What is vector search?",
    k=5,
    mode="vector"
)

BM25 Search

Keyword-based retrieval without embeddings.

results = engine.search(
    "vector search",
    k=5,
    mode="bm25"
)

Hybrid Search

Combines vector and BM25 retrieval using Reciprocal Rank Fusion (RRF).

results = engine.search(
    "What is vector search?",
    k=5,
    mode="hybrid"
)

---

Metadata Filtering

Attach metadata to documents and filter results during retrieval:

results = engine.search(
    "search algorithms",
    k=5,
    mode="vector",
    filter={"topic": "vector-search"}
)

Metadata filtering can be used with vector, BM25, and hybrid search.

---

Benchmarking

Compare multiple search engines using BenchmarkRunner:

from vectorbench_faiss.benchmark import BenchmarkRunner

engines = {
    "ivf": ivf_engine,
    "hnsw": hnsw_engine,
    "pq": pq_engine,
}

runner = BenchmarkRunner(
    engines,
    ground_truth_key="ivf",
)

results = runner.run_all(
    documents,
    queries
)

Benchmarks measure:

• Recall@k — retrieval quality
• Latency — search performance
• Memory — index memory usage

A reference index can be used as ground truth when evaluating approximate indexes.

---

Dataset Utilities

VectorBench includes utilities for preparing datasets from Hugging Face for experiments:

from vectorbench_faiss.data.download import download_dataset
from vectorbench_faiss.data.chunk import chunk_dataset

download_dataset(
    dataset_name="rajpurkar/squad",
    split="train",
    output_dir="data/raw/squad",
)

chunk_dataset(
    input_path="data/raw/squad/raw_data.jsonl",
    output_path="data/raw/squad/chunked_data.jsonl",
)

The resulting documents and metadata can then be passed to VectorSearchEngine.

---

Embeddings

VectorBench uses sentence-transformers for generating embeddings. Models are cached locally after the first download, allowing subsequent runs to load them from the local cache.

---

Example Workflow

Dataset
   ↓
Chunking
   ↓
Embeddings
   ↓
FAISS Index
   ↓
Vector / BM25 / Hybrid Retrieval
   ↓
Benchmark
   ↓
Recall · Latency · Memory

VectorBench is designed primarily for learning, experimentation, and benchmarking of vector retrieval systems.

---

Requirements

VectorBench uses libraries including:

• FAISS
• sentence-transformers
• BM25 retrieval

Dependencies are installed automatically when installing the package through PyPI.

---

Project Status

VectorBench-FAISS is currently focused on vector retrieval experimentation and benchmarking.

Upcoming: comparative dashboard for visualizing benchmark results across datasets, index types, and configurations.

---

Links

Source code: https://github.com/SitanshuA091/VectorBench