suiteeval 0.1.5

Tools for running IR Evaluation Suites

🍬 SuiteEval

Tools for running IR evaluation suites with PyTerrier.
SuiteEval helps you define, run, and aggregate evaluations across datasets while managing temporary indices and memory footprint.

📘 Overview

SuiteEval provides:

• Declaration of pipelines (BM25, dense, re-ranking chains).
• Execution of evaluation suites (e.g., BEIR-style benchmarks).
• DatasetContext utilities for temporary paths and text loading.
• DataFrame outputs for downstream analysis.

Workflow:

1. Implement pipelines(context) that yields one or more PyTerrier pipelines (optionally named).
2. Pass it to a suite (e.g., BEIR).
3. Analyse the returned DataFrame.

🚀 Getting Started

Install from PyPI

pip install suiteeval

Install from source

git clone https://github.com/Parry-Parry/suiteeval.git
cd suiteeval
pip install -e .

⚙️ Defining Pipelines

Write a callable that accepts a DatasetContext and returns or yields pipelines.

• Return a list/tuple of pipelines or (pipeline, name) pairs; or
• Yield pipelines to keep only one large model resident in memory.

DatasetContext provides:

• context.path — temporary working directory for indices/artifacts.
• context.get_corpus_iter() — iterator suitable for indexing.
• context.text_loader() — attaches document text for re-ranking.

Example

from suiteeval import BEIR
from pyterrier_pisa import PisaIndex
from pyterrier_dr import ElectraScorer
from pyterrier_t5 import MonoT5ReRanker

def pipelines(context):
    index = PisaIndex(context.path + "/index.pisa")
    index.index(context.get_corpus_iter())

    bm25 = index.bm25()
    yield bm25 >> context.text_loader() >> MonoT5ReRanker(), "BM25 >> monoT5"
    yield bm25 >> context.text_loader() >> ElectraScorer(), "BM25 >> monoELECTRA"

results = BEIR(pipelines)

🧪 Running Suites

Entry points (e.g., BEIR) accept your pipeline factory and return a DataFrame:

results = BEIR(pipelines)  # per-dataset metrics and system names (if provided)

📦 Reproducibility & Resource Management

• Temporary indices live under context.path and are cleaned up.
• Prefer yielding pipelines when using large models.
• Name systems via (pipeline, "<name>") for clear result tables and logs.

Persistent Index Storage

By default, indices are stored in temporary directories. To persist indices across runs, use the index_dir parameter:

# Indices will be stored in ./indices/<corpus-name>/
# Run files will be stored in ./results/<dataset-name>/
results = BEIR(
    pipelines,
    save_dir="./results",   # Where to save run files (per-dataset)
    index_dir="./indices"   # Where to store indices (per-corpus)
)

Key differences:

• save_dir creates per-dataset subdirectories (e.g., ./results/beir-arguana/)
• index_dir creates per-corpus subdirectories (e.g., ./indices/beir-arguana/)
• Multiple datasets sharing a corpus will reuse the same index directory

🛠️ Compatibility

Works with modern PyTerrier and common extensions
(e.g., pyterrier_pisa, pyterrier_dr, pyterrier_t5).
For older environments, ensure standard PyTerrier transformer interfaces.

👥 Authors

• Andrew Parry
• Sean MacAvaney

🧾 Version History

Version	Date	Changes
0.1	2025-11-03	Initial README

License

This project is licensed under the MIT License — see the LICENSE file for details.