pyterrier-tar

Replay-only technology-assisted-review (TAR) stopping extensions for PyTerrier.

This is an independent package, not a PyTerrier fork. It depends only on documented public pyterrier APIs so that the dependency can be upgraded and compatibility tested independently. It does not vendor PyTerrier code.

Status

Alpha (0.1.0). Each stopping rule replays a recorded, labelled ranking and reports where it would have stopped, why, and at what review cost. Rules whose authors released code or results are checked against them, and the rest against their published definitions; PUBLISHED_EVIDENCE.md lists every check and what it does not establish.

The package also registers the public CLEF eHealth TAR collections as tar:clef2017, tar:clef2018, and tar:clef2019. These dataset adapters hash-check the public archives and pinned released AutoTAR rankings before using them for replay validation.

Stopping rules

For resumable experiments, uncertainty and failure reports, ranking comparisons, TARexp/CSV history import, and diagnostic plots, see BENCHMARKING.md.

Every rule is a PyTerrier transformer over a ranked, labelled trajectory (qid, rank, label, in review order). transform returns the reviewed prefix; stop_report gives the stop and its cost; stop_trace explains the decision; calculation_trace shows every checkpoint the rule evaluated. A rule sees labels only up to the checkpoint it is judging, which the test suite enforces for all of them.

A rule that aims at a recall target takes target_recall as its first argument, with no default, and a rule that needs the collection size takes collection_size next: tar.IPHyperbolic(.8, 3000), tar.CMHHeuristic(.8, 3000). Control-set rules take their screened frame first: tar.QBCB(control, .8).

Rules and metrics that need the collection size either require collection_size or accept it optionally. When it is optional (SAFE, BetaBinomial, stopping_metrics) and omitted, the trajectory is taken to be the whole collection, so pass it whenever the ranking is truncated. CLEFTARDataset.complete_ranking appends a topic's unranked documents. A qid is matched across frames as text, so 1 and '1' are the same query.

Choosing one

tar.catalogue()                        # every rule: family, promise, what it needs, source
tar.catalogue(promise='certificate')   # only the rules that carry a guarantee

Start from what you can supply. With labels alone, you can run any trajectory rule or the checkpoint rules that also take a collection size. With calibrated probabilities from your classifier, the estimation rules become available. If you can screen an independent random sample, QBCB and TargetRecapture are the only rules here that give a guarantee rather than an estimate, and they charge that sample as review cost.

What a rule promises

The four groups below say what a rule reads. What it promises is a different split, and decides how it should be evaluated: heuristics promise nothing about recall, estimators promise it if their model holds, and certificates (QBCB, TargetRecapture) promise it with a stated probability over the draw of their control sample. Use reliability_summary for an estimator and coverage_summary for a certificate; see EVALUATION.md.

Trajectory rules — labels only

Checkpoint rules — a statistical test at fixed intervals

Control-set rules — an independently screened sample

Estimation rules — recorded probabilities or sampling metadata

AnytimeCMH, BetaBinomial, and the EVPI family are original to this package rather than replays of published methods, so they carry no parity claim: they are covered by formula tests and the shared leakage and trace checks only.

Learned rule

GRLStop trains a PPO policy over 100 review windows, where the state is the relevance rate in each reviewed window, a logistic-regression prediction for each unreviewed one, the current window, and the target recall. It needs the grl extra. Source: Bin-Hezam and Stevenson (2025).

Datasets and helpers

CLEFTARDataset backs pt.get_dataset('tar:clef2017') and its 2018/2019 siblings: hash-pinned public archives, one candidate pool per topic, complete_ranking() to append unretrieved documents, and label_ranking() to attach qrels only after a ranking exists.

Metrics

stopping_metrics reports recall, cost, review fraction, reliability, oracle depth, and the CLEF loss_e/loss_r/loss_er components, charging any separately screened controls once. stopping_frontier summarises several targets, and reliability and review_fraction are ir-measures metrics for pt.Experiment.

reliability_summary gives the share of topics reaching the target with a Wilson interval, which is the check for an estimator. coverage_summary gives the share of repeated control draws that certified it, which is the check for a certificate.

Adding a rule

One file per approach, inside the folder for what the rule reads: rules/trajectory/ for labels only, rules/checkpoint/ for a test at fixed checkpoints, rules/control/ for an independently screened sample, and rules/estimation/ for recorded probabilities or sampling metadata.

# src/pyterrier_tar/rules/trajectory/my_rule.py
from .._base import _TrajectoryStoppingRule, _batch_positions


class MyRule(_TrajectoryStoppingRule):
    """Stops once a batch holds no relevant documents."""

    _equation = 'fire if the batch ending at n has no relevant document'

    def __init__(self, batch_size: int = 200, initial_documents: int = 1):
        if batch_size < 1 or initial_documents < 1:
            raise ValueError('batch sizes must be positive')
        self.batch_size = batch_size
        self.initial_documents = initial_documents

    def _checkpoint_rows(self, ranked, labels):
        for stop in _batch_positions(len(labels), self.batch_size, self.initial_documents):
            batch = labels[max(0, stop - self.batch_size):stop]
            yield {'index': stop, 'relevant_in_batch': int((batch > 0).sum()),
                   'fired': not (batch > 0).any()}

_checkpoint_rows yields one dict per checkpoint, with at least index and fired; the base class turns it into the stop, stop_report, stop_trace, and calculation_trace, so those cannot disagree. A rule that is not checkpoint-shaped implements _stop_position(labels) instead, and one that needs other columns implements _stop_position_for_results(ranked, labels). Add _trace_details for the columns stop_trace should carry.

The row for checkpoint k must use only labels[:k]. Export the class from the group's __init__.py, from rules/__init__.py, and from pyterrier_tar/__init__.py; the shared tests then pick it up, including the leakage check and the report/trace consistency check. New rules are expected to come with a test that re-derives the stop from the rule's own definition, as tests/test_rule_validity.py does for the others.

Scope and safety boundary

pyterrier-tar replays recorded, labelled trajectories. It is not a live active-learning or screening controller, and it does not certify target recall in deployment. Labels beyond the reviewed prefix must not be exposed to a stopping rule. Separately sampled controls must be independently screened and accounted for in review cost.

Install

python -m pip install pyterrier-tar
python -m pip install 'pyterrier-tar[grl]'

The grl extra installs gymnasium, scikit-learn, and stable-baselines3 for GRLStop; tutorial adds PyTerrier's Java support and matplotlib for the notebooks that build Terrier indexes.

Quick start

import numpy as np
import pandas as pd
from pyterrier import tar

# A recorded review: ranked documents, with the label revealed as each is read.
# Relevance thins out down the ranking, as it does in a real screening run.
rng = np.random.default_rng(0)
size = 6000
prevalence = .7 * np.exp(-np.arange(size) / 300) + .002
trajectory = pd.DataFrame({
    'qid': 'CD008081',
    'docno': [f'd{position}' for position in range(size)],
    'rank': range(size),
    'label': (rng.random(size) < prevalence).astype(int),   # 1 for relevant
})

rule = tar.Kneedle()
reviewed = rule.transform(trajectory)          # the prefix the rule would have read
report = rule.stop_report(trajectory)          # stop, fired, review cost
print(rule.stop_trace(trajectory).iloc[0].reason)
print(rule.calculation_trace(trajectory, index=int(report.stop.iloc[0])).tail())

qrels = trajectory.loc[trajectory.label > 0, ['qid', 'docno', 'label']]
print(tar.stopping_metrics(trajectory, reviewed, qrels, target_recall=.95, report=report))

On this trajectory Kneedle stops after 2,001 of 6,000 documents, having found 94.7% of the relevant ones; stop_trace says the knee slope ratio reached its dynamic threshold, and calculation_trace shows the ratio at every checkpoint it tested.

Development

python -m pip install -e '.[test,grl,tutorial]'
pytest

test runs the suite, grl adds the GRLStop tests, and tutorial the notebooks. On a CUDA-free machine, UV_TORCH_BACKEND=cpu keeps grl from pulling the GPU wheels.

Results on CLEF

RESULTS.md shows how each rule performs on the CLEF 2017, 2018, and 2019 TAR collections: reliability and review cost at recall targets 0.8, 0.9, and 0.95, per collection, with rules grouped by what they promise rather than ranked. It describes one ranker, the released AutoTAR runs; a rule that reads the ranking can do much worse behind a weaker one. RESULTS_UNCERTAINTY.md gives the intervals behind each row and the largest individual shortfalls. examples/results_table.py regenerates both.

Published Evidence

The exact public checks and remaining non-parity boundaries are listed in PUBLISHED_EVIDENCE.md. The examples under examples/ are intentionally separate from CI when they require network access, large public archives, or locally held trajectories.

For real-data or external-source smoke checks:

python examples/published_cmh_buscar.py
python examples/published_clef_tar_eval_metrics.py
python examples/published_ip_h_clef.py
python examples/published_kneedle_clef2017.py
python examples/published_point_process_clef2017.py
python examples/published_autostop_clef2017.py
PYTHONPATH=examples python examples/published_baselines_clef.py

Run them from a scratch directory: each caches its inputs in data/ relative to the working directory, about 1 GB in total.

examples/notebooks/parity_published.ipynb runs all of them and shows one parity table. Every notebook is either a demo_ (how the package behaves) or a parity_ (a published result reproduced); see examples/notebooks.md.

published_kneedle_clef2017.py checks Kneedle's released 86,243 effort. The same seven scripts run weekly in CI's published-parity job. The notebooks that build Terrier indexes need the tutorial extra, which installs PyTerrier's Java support.

Only the metrics asserted by each evidence check should be treated as validated. For example, the CLEF metric checker asserts only the official recall, total cost, and loss fields exposed by stopping_metrics(), while the IP-H CLEF checker asserts recall, cost, reliability, loss, and relative-error parity across all nine CLEF 2017--19 settings.

Known gaps

GAPS.md lists what this package does not establish: the replay-versus-live question, published results whose inputs were never released, methods named but not implemented, and release chores. New findings about limits belong there rather than in a commit message.

Compatibility policy

The supported PyTerrier range is declared in pyproject.toml. New code must import only public pyterrier names; imports from private modules such as pyterrier._ops are prohibited. Each supported PyTerrier release will receive a focused compatibility test before its range is widened.

License and notices

This source is licensed under MPL-2.0; see LICENSE.txt. PyTerrier-derived source and the IP-H implementation boundary are recorded in THIRD_PARTY_NOTICES.md.