isage-benchmark 0.1.0.6

SAGE Benchmark - RAG and experimental benchmarks for SAGE framework

SAGE Benchmark

Comprehensive benchmarking tools and RAG examples for the SAGE framework

📋 Overview

SAGE Benchmark provides a comprehensive suite of benchmarking tools and RAG (Retrieval-Augmented Generation) examples for evaluating SAGE framework performance. This package enables researchers and developers to:

• Benchmark RAG pipelines with multiple retrieval strategies (dense, sparse, hybrid)
• Compare vector databases (Milvus, ChromaDB, FAISS) for RAG applications
• Evaluate multimodal retrieval with text, image, and video data
• Run reproducible experiments with standardized configurations and metrics

This package is designed for both research experiments and production system evaluation.

✨ Key Features

• Multiple RAG Implementations: Dense, sparse, hybrid, and multimodal retrieval
• Vector Database Support: Milvus, ChromaDB, FAISS integration
• Experiment Framework: Automated benchmarking with configurable experiments
• Evaluation Metrics: Comprehensive metrics for RAG performance
• Sample Data: Included test data for quick start
• Extensible Design: Easy to add new benchmarks and retrieval methods

📦 Package Structure

sage-benchmark/
├── src/
│   └── sage/
│       └── benchmark/
│           ├── __init__.py
│           └── benchmark_rag/           # RAG benchmarking
│               ├── __init__.py
│               ├── implementations/     # RAG implementations
│               │   ├── pipelines/      # RAG pipeline scripts
│               │   │   ├── qa_dense_retrieval_milvus.py
│               │   │   ├── qa_sparse_retrieval_milvus.py
│               │   │   ├── qa_multimodal_fusion.py
│               │   │   └── ...
│               │   └── tools/          # Supporting tools
│               │       ├── build_chroma_index.py
│               │       ├── build_milvus_dense_index.py
│               │       └── loaders/
│               ├── evaluation/          # Experiment framework
│               │   ├── pipeline_experiment.py
│               │   ├── evaluate_results.py
│               │   └── config/
│               ├── config/              # RAG configurations
│               └── data/                # Test data
│           # Future benchmarks:
│           # ├── benchmark_agent/      # Agent benchmarking
│           # └── benchmark_anns/       # ANNS benchmarking
├── tests/
├── pyproject.toml
└── README.md

🚀 Installation

Quick Start (Recommended)

Clone the repository with submodules and set up development environment:

# 1. Clone repository
git clone --recurse-submodules https://github.com/intellistream/sage-benchmark.git
cd sage-benchmark

# Or if already cloned, initialize submodules
./quickstart.sh

# 2. Install package with development dependencies
pip install -e ".[dev]"

# 3. Install pre-commit hooks (IMPORTANT for contributors)
pre-commit install

The quickstart.sh script will automatically:

• Initialize all Git submodules (LibAMM, SAGE-DB-Bench, sageData)
• Check environment and dependencies
• Display submodule status

Why install pre-commit? Pre-commit hooks automatically check code quality (formatting, import sorting, linting) before each commit, preventing CI/CD failures.

Manual Installation

If you prefer manual setup:

# Clone repository
git clone https://github.com/intellistream/sage-benchmark.git
cd sage-benchmark

# Initialize submodules (direct level only, not recursive)
git submodule update --init

# Install package
pip install -e .

Or with development dependencies:

pip install -e ".[dev]"

Git Submodules

This repository uses Git submodules for external components:

• benchmark_amm (src/sage/benchmark/benchmark_amm/) → LibAMM
• benchmark_anns (src/sage/benchmark/benchmark_anns/) → SAGE-DB-Bench
• sage.data (src/sage/data/) → sageData

All submodules track the main-dev branch and must be initialized before use.

📊 RAG Benchmarking

The benchmark_rag module provides comprehensive RAG benchmarking capabilities:

RAG Implementations

Various RAG approaches for performance comparison:

Vector Databases:

• Milvus: Dense, sparse, and hybrid retrieval
• ChromaDB: Local vector database with simple setup
• FAISS: Efficient similarity search

Retrieval Methods:

• Dense retrieval (embeddings-based)
• Sparse retrieval (BM25, sparse vectors)
• Hybrid retrieval (combining dense + sparse)
• Multimodal fusion (text + image + video)

Quick Start

1. Build Vector Index

First, prepare your vector index:

# Build ChromaDB index (simplest)
python -m sage.benchmark.benchmark_rag.implementations.tools.build_chroma_index

# Or build Milvus dense index
python -m sage.benchmark.benchmark_rag.implementations.tools.build_milvus_dense_index

2. Run a RAG Pipeline

Test individual RAG pipelines:

# Dense retrieval with Milvus
python -m sage.benchmark.benchmark_rag.implementations.pipelines.qa_dense_retrieval_milvus

# Sparse retrieval
python -m sage.benchmark.benchmark_rag.implementations.pipelines.qa_sparse_retrieval_milvus

# Hybrid retrieval (dense + sparse)
python -m sage.benchmark.benchmark_rag.implementations.pipelines.qa_hybrid_retrieval_milvus

3. Run Benchmark Experiments

Execute full benchmark suite:

# Run comprehensive benchmark
python -m sage.benchmark.benchmark_rag.evaluation.pipeline_experiment

# Evaluate and generate reports
python -m sage.benchmark.benchmark_rag.evaluation.evaluate_results

4. View Results

Results are saved in benchmark_results/:

• experiment_TIMESTAMP/ - Individual experiment runs
• metrics.json - Performance metrics
• comparison_report.md - Comparison report

📖 Quick Start

Basic Example

from sage.benchmark.benchmark_rag.implementations.pipelines import (
    qa_dense_retrieval_milvus,
)
from sage.benchmark.benchmark_rag.config import load_config

# Load configuration
config = load_config("config_dense_milvus.yaml")

# Run RAG pipeline
results = qa_dense_retrieval_milvus.run_pipeline(query="What is SAGE?", config=config)

# View results
print(f"Retrieved {len(results)} documents")
for doc in results:
    print(f"- {doc.content[:100]}...")

Run Custom Benchmark

from sage.benchmark.benchmark_rag.evaluation import PipelineExperiment

# Define experiment configuration
experiment = PipelineExperiment(
    name="custom_rag_benchmark",
    pipelines=["dense", "sparse", "hybrid"],
    queries=["query1.txt", "query2.txt"],
    metrics=["precision", "recall", "latency"],
)

# Run experiment
results = experiment.run()

# Generate report
experiment.generate_report(results)

Configuration

Configuration files are located in sage/benchmark/benchmark_rag/config/:

• config_dense_milvus.yaml - Dense retrieval configuration
• config_sparse_milvus.yaml - Sparse retrieval configuration
• config_hybrid_milvus.yaml - Hybrid retrieval configuration
• config_qa_chroma.yaml - ChromaDB configuration

Experiment configurations in sage/benchmark/benchmark_rag/evaluation/config/:

• experiment_config.yaml - Benchmark experiment settings

📖 Data

Test data is included in the package:

• Benchmark Data (benchmark_rag/data/):

  • queries.jsonl - Sample queries for testing
  • qa_knowledge_base.* - Knowledge base in multiple formats (txt, md, pdf, docx)
  • sample/ - Additional sample documents for testing
  • sample/ - Additional sample documents

• Benchmark Config (benchmark_rag/config/):

  • experiment_config.yaml - RAG benchmark configurations

🔧 Development

Running Tests

pytest packages/sage-benchmark/

Code Formatting

# Format code
black packages/sage-benchmark/

# Lint code
ruff check packages/sage-benchmark/

📚 Documentation

For detailed documentation on each component:

• See src/sage/benchmark/rag/README.md for RAG examples
• See src/sage/benchmark/benchmark_rag/README.md for benchmark details

🔮 Future Components

• benchmark_agent: Agent system performance benchmarking
• benchmark_anns: Approximate Nearest Neighbor Search benchmarking
• benchmark_llm: LLM inference performance benchmarking

🤝 Contributing

This package follows the same contribution guidelines as the main SAGE project. See the main repository's CONTRIBUTING.md.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🔗 Related Packages

• sage-kernel: Core computation engine for running benchmarks
• sage-libs: RAG components and utilities
• sage-middleware: Vector database services (Milvus, ChromaDB)
• sage-common: Common utilities and data types

📮 Support

• Documentation: https://intellistream.github.io/SAGE-Pub/guides/packages/sage-benchmark/
• Issues: https://github.com/intellistream/SAGE/issues
• Discussions: https://github.com/intellistream/SAGE/discussions

---

Part of the SAGE Framework | Main Repository