PairCtrl

Evaluation code of PairCtrl, a benchmark for genomic language models on gene pairs. Gene position alone, or a close relative in the training data, predicts the labels of many gene-pair tasks. PairCtrl uses experimental labels, scores every model next to a position-only baseline (row zero) and reports how much of a score remains with gene position held fixed. The data, the per-item scores of the leaderboard models and the leaderboard tables are in the dataset repository https://huggingface.co/datasets/tstomar/pairctrl.

Install

pip install pairctrl          # Python >= 3.9, installs seqleak (formerly posbench) for the position model
pip install -e ".[test]"      # from a clone of this repository, with the tests
pytest

Score a model

1. Download the dataset into a directory, DATA_DIR. Each task has a folder with items.parquet.
2. Score every item of a task. For a zero-shot score, embed the genes listed in genomes/genes.parquet and take the cosine similarity of the two genes of each item: python -m pairctrl.zeroshot DATA_DIR/ppi_exp/items.parquet embeddings.npz scores.parquet. Here embeddings.npz holds one array per genome record, keyed by accession, with one row per gene in gene_index order. The option --standardise z-scores each dimension within a genome first. Supervised scores must be out-of-fold predictions under the task's frozen folds.
3. Write a parquet or CSV file with the columns item_id and score and run

python -m pairctrl.evaluate --task ppi_exp --data DATA_DIR --submission scores.parquet --out metrics.json

Add --oof for out-of-fold predictions, which are then compared only within folds. The output has one row per view and evaluation unit (data set, genome or screen class) for the submission, row zero and the untrained distance rule. Each row gives the AUROC, the AUROC within position strata, the AUPRC, the gain over the out-of-fold position model and an interval for the share of the AUROC above 0.5 that is due to gene position, each with a 95% bootstrap interval. Items without a score are left out, and the command reports the coverage.

Tasks

Row zero is the position-only model, a logistic regression on spline bases of the position features that is fitted out of fold on the task's frozen folds. The distance rule is an untrained baseline. For gene pairs it scores 1[same strand] x exp(-gap / 500 bp), with a score of 0 above a gap of 500 bp, and for element-gene pairs it is the negative distance. A view is a label column with a row filter, and the strict subset of ppi_exp is one such view. pairctrl/specs.py defines the views of each task.

Labels and inputs that you rebuild

The dataset leaves out the RegulonDB labels of operon_regulondb_exp and the hg19 sequences of etgp_nodist. python -m pairctrl.joins.regulondb TUSet.txt GeneProductSet.txt items.parquet labels_regulondb.parquet writes the labels from your own RegulonDB v14.5 files. Place the output next to the task's items.parquet. python -m pairctrl.joins.etgp_inputs hg19.fa items.parquet inputs.parquet writes the model inputs from your copy of hg19.

Leaderboard submissions

Open an issue with a link to per-item scores for every item of a task and a short model card. The card names the model, the layer or readout, whether the scores are zero-shot or supervised, and how the training data relate to the test genomes. The maintainer evaluates the scores with this package and adds the rows. Numbers without per-item scores are not added.

Metrics

pairctrl/metrics.py defines every reported quantity. pairctrl/boot.py computes the bootstrap intervals exactly from Mann-Whitney sufficient statistics, with two-way resampling of gene blocks for gene pairs and of clusters otherwise.

Citation

See CITATION.cff. The release is archived at https://doi.org/10.5281/zenodo.23605245.

License

Code: Apache License 2.0 (LICENSE). The dataset is released under CC BY 4.0, except the per-item scores of six models with non-commercial licences, which are under CC BY-NC-SA 4.0. Its NOTICE.md lists the licence of every source.