SAGE Benchmark
Comprehensive benchmarking tools and RAG examples for the SAGE framework
📋 Overview
SAGE Benchmark provides a comprehensive suite of benchmarking tools and RAG (Retrieval-Augmented Generation) examples for evaluating SAGE framework performance. This package enables researchers and developers to:
- Benchmark RAG pipelines with multiple retrieval strategies (dense, sparse, hybrid)
- Compare vector databases (Milvus, ChromaDB, FAISS) for RAG applications
- Evaluate multimodal retrieval with text, image, and video data
- Run reproducible experiments with standardized configurations and metrics
This package is designed for both research experiments and production system evaluation.
✨ Key Features
- Multiple RAG Implementations: Dense, sparse, hybrid, and multimodal retrieval
- Vector Database Support: Milvus, ChromaDB, FAISS integration
- Experiment Framework: Automated benchmarking with configurable experiments
- Evaluation Metrics: Comprehensive metrics for RAG performance
- Sample Data: Included test data for quick start
- Extensible Design: Easy to add new benchmarks and retrieval methods
📦 Package Structure
sage-benchmark/
├── src/
│ └── sage/
│ └── benchmark/
│ ├── __init__.py
│ └── benchmark_rag/ # RAG benchmarking
│ ├── __init__.py
│ ├── implementations/ # RAG implementations
│ │ ├── pipelines/ # RAG pipeline scripts
│ │ │ ├── qa_dense_retrieval_milvus.py
│ │ │ ├── qa_sparse_retrieval_milvus.py
│ │ │ ├── qa_multimodal_fusion.py
│ │ │ └── ...
│ │ └── tools/ # Supporting tools
│ │ ├── build_chroma_index.py
│ │ ├── build_milvus_dense_index.py
│ │ └── loaders/
│ ├── evaluation/ # Experiment framework
│ │ ├── pipeline_experiment.py
│ │ ├── evaluate_results.py
│ │ └── config/
│ ├── config/ # RAG configurations
│ └── data/ # Test data
│ # Future benchmarks:
│ # ├── benchmark_agent/ # Agent benchmarking
│ # └── benchmark_anns/ # ANNS benchmarking
├── tests/
├── pyproject.toml
└── README.md
🚀 Installation
Install the benchmark package:
pip install -e packages/sage-benchmark
Or with development dependencies:
pip install -e "packages/sage-benchmark[dev]"
Note: The sage.data module is included as a submodule in the package and will be installed
automatically. It contains datasets for various benchmarks including LibAMM datasets.
📊 RAG Benchmarking
The benchmark_rag module provides comprehensive RAG benchmarking capabilities:
RAG Implementations
Various RAG approaches for performance comparison:
Vector Databases:
- Milvus: Dense, sparse, and hybrid retrieval
- ChromaDB: Local vector database with simple setup
- FAISS: Efficient similarity search
Retrieval Methods:
- Dense retrieval (embeddings-based)
- Sparse retrieval (BM25, sparse vectors)
- Hybrid retrieval (combining dense + sparse)
- Multimodal fusion (text + image + video)
Quick Start
1. Build Vector Index
First, prepare your vector index:
# Build ChromaDB index (simplest)
python -m sage.benchmark.benchmark_rag.implementations.tools.build_chroma_index
# Or build Milvus dense index
python -m sage.benchmark.benchmark_rag.implementations.tools.build_milvus_dense_index
2. Run a RAG Pipeline
Test individual RAG pipelines:
# Dense retrieval with Milvus
python -m sage.benchmark.benchmark_rag.implementations.pipelines.qa_dense_retrieval_milvus
# Sparse retrieval
python -m sage.benchmark.benchmark_rag.implementations.pipelines.qa_sparse_retrieval_milvus
# Hybrid retrieval (dense + sparse)
python -m sage.benchmark.benchmark_rag.implementations.pipelines.qa_hybrid_retrieval_milvus
3. Run Benchmark Experiments
Execute full benchmark suite:
# Run comprehensive benchmark
python -m sage.benchmark.benchmark_rag.evaluation.pipeline_experiment
# Evaluate and generate reports
python -m sage.benchmark.benchmark_rag.evaluation.evaluate_results
4. View Results
Results are saved in benchmark_results/:
experiment_TIMESTAMP/- Individual experiment runsmetrics.json- Performance metricscomparison_report.md- Comparison report
📖 Quick Start
Basic Example
from sage.benchmark.benchmark_rag.implementations.pipelines import (
qa_dense_retrieval_milvus,
)
from sage.benchmark.benchmark_rag.config import load_config
# Load configuration
config = load_config("config_dense_milvus.yaml")
# Run RAG pipeline
results = qa_dense_retrieval_milvus.run_pipeline(query="What is SAGE?", config=config)
# View results
print(f"Retrieved {len(results)} documents")
for doc in results:
print(f"- {doc.content[:100]}...")
Run Custom Benchmark
from sage.benchmark.benchmark_rag.evaluation import PipelineExperiment
# Define experiment configuration
experiment = PipelineExperiment(
name="custom_rag_benchmark",
pipelines=["dense", "sparse", "hybrid"],
queries=["query1.txt", "query2.txt"],
metrics=["precision", "recall", "latency"],
)
# Run experiment
results = experiment.run()
# Generate report
experiment.generate_report(results)
Configuration
Configuration files are located in sage/benchmark/benchmark_rag/config/:
config_dense_milvus.yaml- Dense retrieval configurationconfig_sparse_milvus.yaml- Sparse retrieval configurationconfig_hybrid_milvus.yaml- Hybrid retrieval configurationconfig_qa_chroma.yaml- ChromaDB configuration
Experiment configurations in sage/benchmark/benchmark_rag/evaluation/config/:
experiment_config.yaml- Benchmark experiment settings
📖 Data
Test data is included in the package:
-
Benchmark Data (
benchmark_rag/data/):queries.jsonl- Sample queries for testingqa_knowledge_base.*- Knowledge base in multiple formats (txt, md, pdf, docx)sample/- Additional sample documents for testingsample/- Additional sample documents
-
Benchmark Config (
benchmark_rag/config/):experiment_config.yaml- RAG benchmark configurations
🔧 Development
Running Tests
pytest packages/sage-benchmark/
Code Formatting
# Format code
black packages/sage-benchmark/
# Lint code
ruff check packages/sage-benchmark/
📚 Documentation
For detailed documentation on each component:
- See
src/sage/benchmark/rag/README.mdfor RAG examples - See
src/sage/benchmark/benchmark_rag/README.mdfor benchmark details
🔮 Future Components
- benchmark_agent: Agent system performance benchmarking
- benchmark_anns: Approximate Nearest Neighbor Search benchmarking
- benchmark_llm: LLM inference performance benchmarking
🤝 Contributing
This package follows the same contribution guidelines as the main SAGE project. See the main
repository's CONTRIBUTING.md.
📄 License
This project is licensed under the MIT License - see the LICENSE file for details.
🔗 Related Packages
- sage-kernel: Core computation engine for running benchmarks
- sage-libs: RAG components and utilities
- sage-middleware: Vector database services (Milvus, ChromaDB)
- sage-common: Common utilities and data types
📮 Support
- Documentation: https://intellistream.github.io/SAGE-Pub/guides/packages/sage-benchmark/
- Issues: https://github.com/intellistream/SAGE/issues
- Discussions: https://github.com/intellistream/SAGE/discussions
Part of the SAGE Framework | Main Repository
Metadata
Release files for isage-benchmark 0.2.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| isage_benchmark-0.2.4.tar.gz | 1.2 MB | Without provenance | Details for isage_benchmark-0.2.4.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| isage_benchmark-0.2.4-py3-none-any.whl | Python 3 | none | any | Details for isage_benchmark-0.2.4-py3-none-any.whl |
Total release size: 3.0 MB
Release files / isage_benchmark-0.2.4.tar.gz
| Download URL | isage_benchmark-0.2.4.tar.gz |
|---|---|
| Size | 1.2 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
535c4157baacbad74f81a33d08fef823e26d51e689c569501f63cdb1f1eb0264
|
|
BLAKE2b-256 checksum How to use checksums |
6eeb18094adfdba2f66a3ad2a4c1e78babe3726162107057d1dab2a88a4bbfbd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.13
|
Release files / isage_benchmark-0.2.4-py3-none-any.whl
| Download URL | isage_benchmark-0.2.4-py3-none-any.whl |
|---|---|
| Size | 1.8 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
072ea17da5eed15354e45b39819deb2b937a27db8cba2c678f8358defeacdf58
|
|
BLAKE2b-256 checksum How to use checksums |
9c8116c357ac28785cc5822a5074921acea0ade294490797d88ba5737d8d0e5e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.13
|
Security for isage-benchmark 0.2.4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.