tcr-pmhc-analyzer

Analyzer for TCR-pMHC binding predictor outputs. Merges predictions from multiple models into a unified table, detects data leakage against bundled training sets, identifies seen/unseen peptides, and benchmarks model performance via ROC curves.

Installation

pip install tcr-pmhc-analyzer

For development:

git clone https://github.com/qbic-pipelines/tcr-pmhc-analyzer.git
cd tcr-pmhc-analyzer
pip install -e ".[dev]"

Input format

Both commands accept a TSV configuration file with two required columns (model, file) and one optional column (training_data):

Example input.tsv:

model	file
ergo2	results/ergo2_predictions.csv
mixtcrpred	results/mixtcrpred_predictions.csv
t2pmhc-gcn	results/t2pmhc_gcn_predictions.csv

Each prediction file must contain the following columns:

Commands

create-analyzer-table

Merges predictions from multiple models into a single table with rank-normalized scores, data leakage annotations, and seen-peptide flags.

tcr-pmhc-analyzer create-analyzer-table [OPTIONS]

Example:

tcr-pmhc-analyzer create-analyzer-table \
  -i input.tsv \
  -o analyzer_table.csv \
  --ergo-version vdjdb

Output columns added:

• binding_score_{model} — raw binding score per model
• rank_score_{model} — rank-normalized score in [0, 1] (1 = highest)
• sample_in_train_{model} — True if the sample appears in the model's training data (data leakage)
• seen_in_{model} — True if the peptide was seen in the model's training data

benchmark

Generates ROC curve plots comparing model performance, split by seen vs unseen peptides. Data leakage samples are automatically removed before analysis.

tcr-pmhc-analyzer benchmark [OPTIONS]

* Either --input or --table must be provided.

Examples:

# Benchmark from raw predictions
tcr-pmhc-analyzer benchmark -i input.tsv -o results/

# Benchmark from a pre-created analyzer table
tcr-pmhc-analyzer benchmark --table analyzer_table.csv -o results/

# Benchmark specific models only
tcr-pmhc-analyzer benchmark -i input.tsv -o results/ -m "ergo2 mixtcrpred tabr-bert"

Output files:

• roc_curve_unseen.png — ROC curves for peptides unseen by all selected models
• roc_curve_seen.png — ROC curves for peptides seen by all selected models

Supported models

Note on score direction. Most models output binding_score such that higher = more likely binder. tcren is the exception: lower (more negative) score = more likely binder. The analyzer is aware of this and inverts direction internally for rank_score_tcren and for ROC/PR benchmarking, so rank_score_{model} (1.0 = strongest binder) and AUC/AUPRC are directly comparable across all models.

Note on the stag-llm training set. STAG-LLM trains in two stages: a self-supervised ESM-2 masked-language-model pretraining on a large sequence-only corpus (full_seq_df_new.csv, binders only, no labels), then the supervised binding classifier — the deployed checkpoint — trained with labels only on the structurally-modeled set (final_dataset_modeled.csv, 46,201 pairs). The bundled stag-llm_train.csv is that supervised structural set, since binding-benchmark leakage means the labeled pairs the model actually learned; the MLM pretraining corpus is treated like ESM-2's own pretraining and is not counted as label leakage. To experiment with the full sequence corpus later, point stag-llm's training_data column in the input samplesheet at an alternate file — no code change needed.

ERGO-II "both" variant mode

The upstream tcrpmhcbinding Nextflow pipeline can run ERGO-II against both training-database variants (vdjdb and mcpas) in a single run by passing --ergo2_variant both. In that mode the pipeline emits two prediction columns labelled ergo2-vdjdb and ergo2-mcpas instead of the single ergo2 column.

When you pass that predictions TSV to this tool, use --ergo-version both:

tcr-pmhc-analyzer create-analyzer-table \
  -i input.tsv \
  -o analyzer_table.csv \
  --ergo-version both

Each ERGO column is then checked against its respective training set:

• sample_in_train_ergo2-vdjdb and seen_in_ergo2-vdjdb are computed against the vdjdb training data.
• sample_in_train_ergo2-mcpas and seen_in_ergo2-mcpas are computed against the mcpas training data.

--ergo-version both requires the input config to contain both ergo2-vdjdb and ergo2-mcpas model rows. Single-variant inputs continue to use --ergo-version vdjdb or --ergo-version mcpas with the legacy ergo2 model name; that path is unchanged.

How it works

1. Merge: Prediction outputs from multiple models are merged on the identifier column into a single DataFrame.
2. Rank normalization: Each model's binding_score is rank-normalized to [0, 1] using descending order with average tie-breaking. NaN values are preserved.
3. Data leakage detection: Each sample is checked against bundled training data to flag samples that appear in a model's training set. The match rule is per-model because each training file exposes the TCR-pMHC tuple at a different granularity — see Data leakage detection below for the per-model rules.
4. Seen peptide detection: Each peptide is checked against training data to identify whether it was seen during model training.
5. Benchmarking: ROC curves are generated after removing leaked samples, separately for seen and unseen peptides.

Data leakage detection

sample_in_train_{model} is computed by a per-model matcher in Analyzer._find_leakage_<model>. Each function does its own normalization (V/J *NN allele stripping, MHC field truncation, etc.) and its own pd.merge. The rules are derived from what each bundled training file actually ships (column presence, NaN rates, MHC notation), not from the underlying model paper, so the leakage check reflects the granularity the training file was actually curated at.

Several of these rules mirror the reference implementations in benchmarker.find_leakage_* (from the t2pmhc revisions benchmark pipeline at t2pmhc_revisions/benchmark_m3/benchmarker.py); the others are derived directly from the training-file profile.

Regeneration note. Per-model rules changed in [dev] — see CHANGELOG.md. Analyzer tables generated before that change have stale sample_in_train_* columns for mixtcrpred, ergo2_vdjdb, ergo2_mcpas, tulip-tcr, tcren, nettcr_peptide, and nettcr_pan and should be regenerated.

Citations

If you use tcr-pmhc-analyzer in your research, please cite the underlying prediction models:

ATM-TCR

Cai, M. et al. (2022). ATM-TCR: TCR-Epitope Binding Affinity Prediction Using a Multi-Head Self-Attention Model. Frontiers in Immunology, 13, 893247. https://doi.org/10.3389/fimmu.2022.893247

ERGO-II

Springer, I. et al. (2021). Contribution of T Cell Receptor Alpha and Beta CDR3, MHC Typing, V and J Genes to Peptide Binding Prediction. Frontiers in Immunology, 12, 664514. https://doi.org/10.3389/fimmu.2021.664514

MIXTCRpred

Croce, G. et al. (2024). Deep learning predictions of TCR-epitope interactions reveal epitope-specific chains in dual alpha T cells. Nature Communications, 15, 3211. https://doi.org/10.1038/s41467-024-47461-8

t2pmhc

Polster, M. et al. (2026). t2pmhc: A Structure-Informed Graph Neural Network to Predict TCR-pMHC Binding. bioRxiv. https://doi.org/10.64898/2026.02.27.708137

TABR-BERT

Zhang, J. et al. (2024). Accurate TCR-pMHC interaction prediction using a BERT-based transfer learning method. Briefings in Bioinformatics, 25(1), bbad436. https://doi.org/10.1093/bib/bbad436

TULIP

Meynard-Piganeau, B. et al. (2024). TULIP — a Transformer-based Unsupervised Language model for Interacting Peptides and T-cell receptors that generalizes to unseen epitopes. Proceedings of the National Academy of Sciences, 121(13). https://doi.org/10.1073/pnas.2316401121

NetTCR_2_2

Jensen, M. F., & Nielsen, M. (2023). NetTCR 2.2-Improved TCR specificity predictions by combining pan-and peptide-specific training strategies, loss-scaling and integration of sequence similarity. bioRxiv, 2023-10. https://www.biorxiv.org/content/10.1101/2023.10.12.562001v2.abstract

TCRen

Karnaukhov, V. K. et al. (2024). Structure-based prediction of T cell receptor recognition of unseen epitopes using TCRen. Nature Computational Science, 4(7), 510–521. https://doi.org/10.1038/s43588-024-00653-0

License

MIT