milvus-beir 1.0.0

A Python library that integrates Milvus vector database with BEIR (Benchmarking IR) for efficient information retrieval and evaluation.

Milvus BEIR

A Python library that integrates Milvus vector database with BEIR (Benchmarking IR) for efficient information retrieval and evaluation. This library provides various search strategies including dense retrieval, sparse retrieval, and hybrid search approaches.

Features

• Multiple search strategies:
  • Dense Vector Search
  • Sparse Vector Search
  • BM25 Search
  • Hybrid Search (BM25 + Dense, Sparse + Dense)
  • Multi-Match Search
• Seamless integration with BEIR datasets and evaluation metrics
• Easy-to-use API for retrieval and evaluation
• Built-in performance measurement (QPS)
• Compatible with Milvus 2.5.x

Installation

pip install milvus-beir

Prerequisites

• Python >= 3.10
• Running Milvus instance (2.5.0 or higher)

Quick Start

Here's a comprehensive example of how to use milvus-beir for search and evaluation:

from beir import util
from beir.datasets.data_loader import GenericDataLoader
from beir.retrieval.evaluation import EvaluateRetrieval
from milvus_beir.retrieval.search.dense.dense_search import MilvusDenseSearch

# Download and load BEIR dataset
dataset = "scifact"
url = f"https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/{dataset}.zip"
data_path = util.download_and_unzip(url, "datasets")
corpus, queries, qrels = GenericDataLoader(data_folder=data_path).load(split="test")

# Initialize Milvus search model
uri = "http://localhost:19530"
model = MilvusDenseSearch(
    uri, 
    token=None,  # Optional authentication token
    collection_name="milvus_beir_demo",
    nq=100,  # Number of queries to process in parallel
    nb=1000  # Number of documents to process in parallel
)

# Perform retrieval and evaluation
retriever = EvaluateRetrieval(model)
results = retriever.retrieve(corpus, queries)

# Calculate standard metrics
ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)

# Measure search performance (QPS)
qps = model.measure_search_qps(
    corpus, 
    queries, 
    top_k=1000,
    concurrency_levels=[1, 2],
    test_duration=60
)

Search Strategies

Dense Vector Search

Uses dense embeddings for semantic search.

from milvus_beir.retrieval.search.dense.dense_search import MilvusDenseSearch

Sparse Vector Search

Implements sparse vector retrieval.

from milvus_beir.retrieval.search.sparse.sparse_search import MilvusSparseSearch

BM25 Search

Traditional lexical search using BM25 algorithm.

from milvus_beir.retrieval.search.lexical.bm25_search import MilvusBM25Search

Multi-Match Search

Implements a multi-match search strategy similar to Elasticsearch's multi-match with best_fields type.

from milvus_beir.retrieval.search.multi_match.multi_match_search import MilvusMultiMatchSearch

Hybrid Search

Combines different search strategies for better results.

from milvus_beir.retrieval.search.hybrid.bm25_hybrid_search import MilvusBM25DenseHybridSearch
from milvus_beir.retrieval.search.hybrid.sparse_hybrid_search import MilvusSparseDenseHybridSearch

Command Line Interface

The package includes a powerful command-line interface for evaluating different search methods:

# Basic usage
milvus-beir --dataset nfcorpus --search-method sparse

# Full options
milvus-beir \
    --dataset nfcorpus \
    --uri "http://localhost:19530" \
    --token "your_token" \
    --search-method bm25 \
    --collection-name "my_collection" \
    --nq 100 \
    --nb 1000 \
    --concurrency-levels "1,2" \
    --measure-qps

Available options:

• --dataset, -d: Dataset name to evaluate on (required)
  • Supported datasets: climate-fever, dbpedia-entity, fever, fiqa, hotpotqa, nfcorpus, nq, quora, scidocs, scifact, webis-touche2020, trec-covid, mmarco, cqadupstack/android, cqadupstack/english
• --uri, -u: Milvus server URI (default: http://localhost:19530)
• --token, -t: Authentication token for Milvus (optional)
• --search-method, -m: Search method to use (required)
  • Available methods: dense, sparse, sparse_hybrid, bm25_hybrid, multi_match, bm25
• --collection-name, -c: Milvus collection name (optional)
• --nq: Number of queries to process in parallel (default: 100)
• --nb: Number of documents to process in parallel (default: 1000)
• --concurrency-levels: Comma-separated list of concurrency levels for QPS measurement (default: "1,2")
• --measure-qps: Flag to enable QPS measurement (default: True)

The CLI tool will:

1. Download and load the specified BEIR dataset
2. Initialize the selected search method
3. Perform retrieval and evaluation
4. Calculate and display standard metrics (NDCG@k, MAP@k, Recall@k, Precision@k)
5. Measure and report search performance in QPS (Queries Per Second)

Development

Setup Development Environment

# Clone the repository
git clone https://github.com/your-org/milvus-beir.git
cd milvus-beir

# Install dependencies using PDM
pdm install
pre-commit install

Running Tests

pdm run pytest tests/

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

We welcome contributions! Please feel free to submit a Pull Request.