🍬 SuiteEval

Tools for running IR evaluation suites with PyTerrier.
SuiteEval helps you define, run, and aggregate evaluations across datasets while managing temporary indices and memory footprint.

📘 Overview

SuiteEval provides:

• Declaration of pipelines (BM25, dense, re-ranking chains).
• Execution of evaluation suites (e.g., BEIR-style benchmarks).
• DatasetContext utilities for temporary paths and text loading.
• DataFrame outputs for downstream analysis.

Workflow:

1. Implement pipelines(context) that yields one or more PyTerrier pipelines (optionally named).
2. Pass it to a suite (e.g., BEIR).
3. Analyse the returned DataFrame.

🚀 Getting Started

Install from PyPI

pip install suiteeval

Install from source

git clone https://github.com/Parry-Parry/suiteeval.git
cd suiteeval
pip install -e .

⚙️ Defining Pipelines

Write a callable that accepts a DatasetContext and returns or yields pipelines.

• Return a list/tuple of pipelines or (pipeline, name) pairs; or
• Yield pipelines to keep only one large model resident in memory.

DatasetContext provides:

• context.path — temporary working directory for indices/artifacts.
• context.get_corpus_iter() — iterator suitable for indexing.
• context.text_loader() — attaches document text for re-ranking.

Example

from suiteeval import NanoBEIR
from pyterrier_pisa import PisaIndex
from pyterrier_dr import ElectraScorer
from pyterrier_t5 import MonoT5ReRanker

def pipelines(context):
    index = PisaIndex(context.path + "/index.pisa")
    index.index(context.get_corpus_iter())

    bm25 = index.bm25(num_results=10)
    yield bm25 >> context.text_loader() >> MonoT5ReRanker(), "BM25 >> monoT5"
    yield bm25 >> context.text_loader() >> ElectraScorer(), "BM25 >> monoELECTRA"

results = NanoBEIR(pipelines)

This would product a table as follows:

🧪 Running Suites

Entry points (e.g., BEIR) accept your pipeline factory and return a DataFrame:

results = BEIR(pipelines)  # per-dataset metrics and system names (if provided)

📦 Reproducibility & Resource Management

• Temporary indices live under context.path and are cleaned up.
• Prefer yielding pipelines when using large models.
• Name systems via (pipeline, "<name>") for clear result tables and logs.

Persistent Index Storage

By default, indices are stored in temporary directories. To persist indices across runs, use the index_dir parameter:

# Indices will be stored in ./indices/<corpus-name>/
# Run files will be stored in ./results/<dataset-name>/
results = BEIR(
    pipelines,
    save_dir="./results",   # Where to save run files (per-dataset)
    index_dir="./indices"   # Where to store indices (per-corpus)
)

Key differences:

• save_dir creates per-dataset subdirectories (e.g., ./results/beir-arguana/)
• index_dir creates per-corpus subdirectories (e.g., ./indices/beir-arguana/)
• Multiple datasets sharing a corpus will reuse the same index directory

Automatic Result Caching

When using save_dir, SuiteEval automatically skips inference for pipelines that already have saved run files. If a {pipeline_name}.res.gz file exists for all datasets in a corpus, the suite loads results from disk instead of re-running the pipeline.

# First run: executes inference and saves results
results = BEIR(pipelines, save_dir="./results")

# Second run: automatically loads from ./results/{dataset}/{name}.res.gz
results = BEIR(pipelines, save_dir="./results")

To force re-running inference, use save_mode="overwrite":

# Always re-run inference, even if files exist
results = BEIR(pipelines, save_dir="./results", save_mode="overwrite")

🔧 Customising a Suite

A suite run is a fixed sequence of small, overridable steps. To change one part of the computation, override the hook that covers it rather than reimplementing __call__:

__call__
  └─ resolve_config          split call arguments into a RunConfig
  └─ run
     └─ iter_corpus_groups   group datasets by shared corpus
        └─ select_members    pick datasets to evaluate in this group
        └─ build_context     build the shared DatasetContext (indexes once)
        └─ iter_pipeline_batches
           └─ wrap_pipeline          decorate each pipeline
           └─ prepare_topics_qrels   fetch topics and qrels
           └─ evaluate_batch
              ├─ has_cached_run / run_file_path / load_cached_run
              └─ run_experiment      the pt.Experiment call
                 └─ measures_for     pick metrics for a dataset
           └─ annotate_results       tag rows with their dataset
           └─ release_pipelines      free memory between batches
  └─ postprocess_results     aggregate the concatenated table
     └─ compute_overall_mean

Wrap every pipeline — applies to both sequential and grouped execution:

class MySuite(Suite):
    _datasets = ["my-corpus/test"]

    def wrap_pipeline(self, pipeline, context):
        return pipeline >> MyResultFilter(context.dataset.get_qrels())

Reshape the results table before the Overall rows are added:

def postprocess_results(self, results, config):
    results = merge_sub_collections(results)
    return super().postprocess_results(results, config)

Change where runs are cached:

def run_file_path(self, save_dir, dataset_name, pipeline_name):
    return os.path.join(save_dir, f"{dataset_name}--{pipeline_name}.res.gz")

🛠️ Compatibility

Works with modern PyTerrier and common extensions
(e.g., pyterrier_pisa, pyterrier_dr, pyterrier_t5).
For older environments, ensure standard PyTerrier transformer interfaces.

👥 Authors

• Andrew Parry
• Sean MacAvaney

🧾 Version History

License

This project is licensed under the MIT License — see the LICENSE file for details.