Models for highlight-based explainable AI research.

A highlight-based model commits to a subset of the document, the highlight, and predicts from that subset alone. The highlight is therefore not a story about the prediction, it is the input to it. A highlight that omits what mattered produces a worse prediction, which is measurable rather than arguable.

The methods are many and the benchmarks are many, and no standard way of running them exists. Every paper brings its own corpora, its own splits and its own metrics, so two published numbers rarely answer the same question. Reproducing a result is hard and comparing two methods fairly is harder.

pyhighlights exists to remove that obstacle. It runs any model it ships on any corpus it ships, under one configuration system and one set of metrics. A comparison is then a change of key rather than a rewrite.

The library ships the select-then-predict family, where a selector picks the highlight and a predictor reads only what the selector kept. Other families, dual-head architectures among them, come next. The corpora, the metrics and the experiment machinery belong to the library rather than to a family. Adding a family is therefore an architecture and its configurations, and nothing else.

Documentation · Tutorial · Contributing

Installation

The library requires Python 3.10 or later.

pip install pyhighlights

Transformer backbones are an extra, so a GRU run does not pull in transformers:

pip install "pyhighlights[transformers]"

Quickstart

A task is one experiment start to finish: a corpus, its preprocessing, a model, its metrics, and a list of seeds. Write the script below to a file and run it from any directory you can write to. The results land beneath the save_path it names.

from pathlib import Path

import pyhighlights
from cinnamon.registry import Registry
from pyhighlights.components.analyzers import MetricsAnalyzer, PredictionAnalyzer
from pyhighlights.configurations.keys import TOY_TASK

Registry.build(directory=Path(pyhighlights.__file__).parent)

task = Registry.from_key(
    TOY_TASK,
    save_path="results",
    seeds=[0, 1],
    store_predictions=True,
)
task.run()

python quickstart.py

Each seed trains from scratch, restores the checkpoint that scored best on validation, and is scored on validation and test. Seeds are a list rather than a number because one run says very little. The selector is trained through a discrete choice, which makes the optimization unstable. The spread across seeds is therefore part of the result.

What lands on disk:

results/toy/2026-09-19T16-41-53/
├── results.json               # every seed's metrics, and their summary
├── manifest.json              # the key, the overrides, the whole
│                              #   configuration tree, and the versions
├── predictions-seed=0.pkl
├── predictions-seed=1.pkl
├── seed=0/epoch=1-step=16.ckpt
└── seed=0/lightning_logs/version_0/metrics.csv

A run never overwrites an earlier one, so two runs of a task are two results to compare. The manifest names the key and the arguments the run was launched with. It also resolves every nested key, so it states the hidden size and the learning rate themselves rather than the name they came from.

Read the results back as frames, so the same analyzer serves a notebook, a test and a LaTeX table:

MetricsAnalyzer(directory="results", metrics=["accuracy", "highlight_f1"]).run()

task                 run  seeds          accuracy      highlight_f1
 toy 2026-09-19T16-41-53      2 0.7188 +/- 0.2812 0.3489 +/- 0.3063

TOY_TASK is a smoke test rather than a result. It trains FR on a GRU backbone for two epochs on CPU over 64 generated documents, which checks that a run holds together and teaches nothing else. Still, the spread is worth reading: two seeds of one configuration disagree by more than a quarter of the highlight score.

A stored prediction is token ids and masks. PredictionAnalyzer rebuilds the corpus from the key in the manifest and joins it on sample_id, so a row says which words the model kept:

PredictionAnalyzer(directory="results").analyze()[
    ["seed", "sample_id", "label", "predicted", "selected_text"]
]

Both analyzers report the same frame, and they differ in what they do with it: run prints it and returns it, while analyze only returns it.

Architectures

Selection is a discrete choice inside a differentiable model, and each architecture answers that differently. All eight are registered for a GRU and for a Transformer backbone. No algorithm mentions either, since a backbone is anything implementing encode, pool and output_size.

Corpora

Three of the five corpora leak as distributed: annotated evaluation rows also appear in training. Every Hotel aspect leaks its whole annotated split, and a highlight score published on those splits is measured on seen data. The table below reports the repaired splits, which is what LeakageRemover produces and what LeakageDetector().check() accepts. The distributed splits stay reachable, since a reproduction of published numbers needs them. The datasets page gives the per-aspect counts and the digests.

Beer and Hotel carry three aspects each, and the rows below are beer0 and hotel_Location. HateXplain is aggregated before the repair, so its three annotators per document become one label and one highlight. That aggregation drops no row. The toy corpus is generated by ToyLoader, whose triggers, document length, alphabet and split sizes are all parameters. The same loader reads a published corpus when given a url and a digest, which is how the released GenSPP corpus is read (10.5281/zenodo.22711448).

Metrics

Classification metrics are registered per class count. Highlight metrics score only the positions a corpus annotated, so an unannotated position counts as nothing rather than as a negative.

Sufficiency and comprehensiveness are the faithfulness pair of DeYoung et al., ACL 2020, computed over the test split. They are not two views of one number. A model can highlight three words that suffice while ten others would have sufficed too, which is sufficient and not comprehensive.

Reproductions

A reproduction lives in a repository of its own.

The GenSPP reproduction runs two corpora against the five architectures that paper compares, which are FR, MCD, MGR, G-RAT and GenSPP, rather than against all eight the library ships.

Contact

pyhighlights ships tools rather than an experiment. Which metrics to log, which aspect of Beer to train on and which sparsity target to aim at all depend on a study. Each stays a configuration the study writes.

Questions about a component or a result are best raised as an issue. For anything else, write to Federico Ruggeri, federico.ruggeri6@unibo.it.