hitlist

A curated, harmonized, ML-training-ready MHC ligand mass-spectrometry dataset.

hitlist ingests immunopeptidome data from IEDB, CEDAR, and paper supplementary tables (PRIDE/jPOSTrepo); partitions MS-eluted observations from in-vitro binding-assay measurements into two separate parquet files (so downstream consumers never silently conflate them); joins every MS observation to expert-curated sample metadata (HLA genotype, tissue, disease, perturbation, instrument); and ships both indexes as parquet + a pandas-friendly Python API.

What's in the two indexes

After hitlist data build (snapshot of the shipping 1.10.x default build):

observations.parquet — MS-eluted immunopeptidome

binding.parquet — in-vitro binding-assay measurements

The two indexes share the schema (including gene annotations from the peptide-mappings sidecar), but supplementary curation is MS-only — binding is pure IEDB/CEDAR.

Human MHC-I breakdown

What's curated

Those numbers rest on per-study human curation. Rather than trust the raw IEDB/CEDAR cell_name, disease, and tissue fields — which carry mislabels, free-text placeholders, and Other/Unknown sentinels — hitlist annotates each study (keyed by PMID) in pmid_overrides.yaml. Known mislabels are corrected, and every MS sample is tagged with the metadata a training pipeline actually needs (HLA genotype, tissue, disease, perturbation, instrument) plus a reference proteome for flanking and source-protein attribution. A few papers whose peptides never reached IEDB are ingested directly from their PRIDE / jPOSTrepo supplementary tables.

Supplementary papers ingested (data not in IEDB, read straight from PRIDE/jPOSTrepo — 9 CSVs total):

• Abelin 2019 — MAPTAC mono-allelic class I, class II, and DR-tissue (3)
• Gómez-Zepeda 2024 — JY, HeLa, Raji, SK-MEL-37, and plasma (5)
• Stražar 2023 — HLA-II (1)

Install

pip install hitlist

Quick start for ML training

# One-time: register IEDB + CEDAR downloads and build
hitlist data register iedb /path/to/mhc_ligand_full.csv
hitlist data register cedar /path/to/cedar-mhc-ligand-full.csv
hitlist data build                           # a few minutes end-to-end;
                                             # writes observations.parquet +
                                             # binding.parquet + peptide_mappings.parquet

# Export training-ready CSVs
hitlist export training --include-evidence ms --class I --species "Homo sapiens" --mono-allelic \
    --min-allele-resolution four_digit -o mono_allelic_classI.csv

hitlist export training --include-evidence ms --class II --species "Homo sapiens" \
    -o classII_training.csv

# Presto-style flank-aware export: one row per (evidence row, peptide mapping)
hitlist export training --include-evidence both --class I --species "Homo sapiens" \
    --explode-mappings -o presto_training.parquet

hitlist export training does not create a new canonical store. It composes the existing observations.parquet, binding.parquet, and peptide_mappings.parquet indexes into one training-facing export surface. The low-level indexes keep their semantic boundaries; the training export gives downstream consumers one obvious API/CLI path when they want model-ready tables.

Python API

Training-data export

from hitlist.export import generate_ms_observations_table
from hitlist.export import generate_training_table

# Mono-allelic human class I MS observations with ground-truth allele
mono_ms = generate_ms_observations_table(
    mhc_class="I",
    species="Homo sapiens",
    is_mono_allelic=True,
    min_allele_resolution="four_digit",
)

# Presto-style mapping-aware export: one row per (evidence row, peptide mapping)
presto = generate_training_table(
    include_evidence="both",
    mhc_class="I",
    species="Homo sapiens",
    explode_mappings=True,
)
# columns now include: evidence_kind, evidence_row_id, protein_id, position,
# n_flank, c_flank, proteome, proteome_source

generate_observations_table() remains available as a backward-compatible alias.

Species filters accept any variant — "Homo sapiens", "human", "homo_sapiens", "Homo sapiens (human)" all work.

Raw observations loading

from hitlist.observations import (
    load_ms_observations,     # MS-eluted immunopeptidome
    load_binding,             # in-vitro binding-assay measurements
    load_all_evidence,        # union, tagged with an evidence_kind column
    is_built, is_binding_built,
    observations_path, binding_path,
)

# MS-elution (the default training-data path)
df = load_ms_observations()                       # everything (MS-eluted only)
df = load_ms_observations(mhc_class="I")          # class I only
df = load_ms_observations(species="Homo sapiens") # human only
df = load_ms_observations(source="iedb")          # filter by source
df = load_ms_observations(columns=["peptide", "mhc_restriction", "src_cancer"])

# Binding assays — same filter API, reads binding.parquet
bd = load_binding(mhc_class="I", mhc_restriction="HLA-A*02:01")

# Union — for affinity-predictor training, or UI flags that want both.
# Rows are tagged with evidence_kind ∈ {"ms", "binding"}.
both = load_all_evidence(gene_name="PRAME", mhc_class="I")
both["evidence_kind"].value_counts()

load_observations() remains available as a backward-compatible alias.

Building / curation

from hitlist.builder import build_observations
from hitlist.curation import (
    classify_ms_row,
    normalize_species,
    normalize_allele,
    load_pmid_overrides,
)
from hitlist.supplement import scan_supplementary, load_supplementary_manifest

build_observations(with_flanking=True, use_uniprot_search=True, force=False)
normalize_species("human")           # → "Homo sapiens"
normalize_allele("H-2Kb")            # → "H2-K*b"
scan_supplementary()                 # DataFrame of curated paper-supplement peptides

Peptide → protein attribution and flanking context

hitlist data build always produces three parquet files (use --no-mappings to skip peptide_mappings.parquet):

• ~/.hitlist/observations.parquet — one row per assay observation
• ~/.hitlist/binding.parquet — one row per binding-assay observation
• ~/.hitlist/peptide_mappings.parquet — one row per (peptide, protein, position)

The mappings sidecar preserves multi-mapping so a peptide shared by MAGEA1/A4/A10/A12 keeps every paralog. Observations additionally carry semicolon-joined identity columns:

from hitlist.observations import load_ms_observations
from hitlist.mappings import load_peptide_mappings

# Central columns — fast for everyday filters (uses mappings sidecar for pushdown)
df = load_ms_observations(gene_name="PRAME")

# Long form for paralog / position / flank analysis
mappings = load_peptide_mappings(gene_name="MAGEA4")
# columns: peptide, protein_id, gene_name, gene_id, position, n_flank, c_flank, proteome

For ad-hoc queries without building the full table:

from hitlist.proteome import ProteomeIndex

idx = ProteomeIndex.from_ensembl(release=112, species="human")
flanking = idx.map_peptides(["SLLMWITQC", "GILGFVFTL"], flank=10)

Proteome registry / UniProt resolution

from hitlist.downloads import (
    lookup_proteome,           # org string → registry entry (dict)
    fetch_species_proteome,    # download FASTA and cache to ~/.hitlist/proteomes/
    resolve_proteome_via_uniprot,  # direct UniProt REST lookup
    list_proteomes,            # manifest section
)

lookup_proteome("Mycobacterium tuberculosis")
# → {'kind': 'uniprot', 'proteome_id': 'UP000001584', ...}  # H37Rv reference

Output schema — generate_ms_observations_table()

sample_match_type — join provenance

Biological source classification

Every observation is classified by mutually-exclusive biological source category:

Cancer-specific = src_cancer AND NOT src_healthy_tissue. Thymus, reproductive tissue, adjacent tissue, EBV-LCLs, and activated APCs do NOT disqualify a peptide from being cancer-specific.

CLI reference

Data management

hitlist data register <name> <path> [-d DESCRIPTION]    # register a local file
hitlist data fetch <name> [--force]                     # download a known dataset (IEDB/CEDAR/viral FASTAs)
hitlist data refresh <name>                             # re-download
hitlist data info <name>                                # detailed metadata (JSON)
hitlist data path <name>                                # print the registered path
hitlist data remove <name> [--delete]                   # unregister (optionally delete file)
hitlist data list                                       # show registered datasets
hitlist data available                                  # show all known datasets

Build the observations table

hitlist data build [--force]                            # ~90s full scan with tqdm progress
hitlist data build                                      # always builds peptide_mappings.parquet
hitlist data build --use-uniprot                        # broader proteome coverage via UniProt REST
hitlist data build --no-mappings                        # skip mapping step (faster, no gene attribution)
hitlist data build --no-fetch-proteomes                 # don't auto-download missing proteomes
hitlist data build --proteome-release 112               # Ensembl release for human/mouse/rat

Proteome management

hitlist data fetch-proteomes [--min-observations N] [--use-uniprot] [--force]
hitlist data list-proteomes

Index (for raw peptide counts)

hitlist data index [--source iedb|cedar|merged|all] [--force]

Export

hitlist export ms [filters...] -o train.csv             # MS immunopeptidome + sample metadata
hitlist export ms -o train.parquet                      # parquet output supported
hitlist export peptide-summary --gene PRAME --serotype A24   # per-peptide support for one allele/serotype
hitlist export binding [filters...] -o binding.csv      # binding-assay index (separate from MS)
hitlist export training [filters...] -o training.csv    # unified training export from canonical indexes
hitlist export samples [--class I|II]                   # per-sample conditions (YAML curation only)
hitlist export summary                                  # species x class summary
hitlist export counts [--source iedb|cedar|merged|all]  # peptide counts per PMID
hitlist export alleles                                  # validate YAML alleles with mhcgnomes
hitlist export data-alleles                             # validate all IEDB/CEDAR alleles

hitlist export ms is the canonical name for the MS observations export; hitlist export observations remains as a backward-compatible alias.

Canonical indexes and the training export

Each hitlist data build writes three parquet files to ~/.hitlist/:

• observations.parquet — MS-eluted immunopeptidome (IEDB + CEDAR + curated supplementary).
• binding.parquet — binding-assay rows (peptide microarray, refolding, MEDi, and quantitative-tier measurements like Positive-High/Intermediate/Low).
• peptide_mappings.parquet — long-form peptide → protein/position/flank mappings.

The canonical indexes are never silently mixed. Supplementary data is MS-only. Use hitlist export observations and hitlist export binding when you want the raw evidence families separately. Use hitlist export training or generate_training_table(...) when you want a composed model-facing export with evidence_kind tagging and optional mapping explosion for flank-aware training pipelines.

Filters on hitlist export observations

All filters are pushed down to the parquet reader (pyarrow), so --gene PRAME reads only the matching row groups — typically milliseconds rather than a full table scan. Examples:

• hitlist export ms --gene PRAME --class I -o prame_classI.csv
• hitlist export ms --gene "MART-1" (HGNC resolves to MLANA)
• hitlist export ms --mhc-allele HLA-A*02:01 --mono-allelic
• hitlist export ms --serotype A24 (locus-specific)
• hitlist export ms --serotype Bw4 (public epitope — A23/24/25/32, B13/27/44/51/52/53/57/58)

hitlist export peptide-summary

Collapses the MS observations to one row per peptide, scoring how strongly each peptide is supported on a single target allele or serotype. Answers questions like "which PRAME peptides might be presented on A24 in cancers?"

hitlist export peptide-summary --gene PRAME --serotype A24 -o prame_a24.csv
hitlist export peptide-summary --gene PRAME --mhc-allele HLA-A*24:02

Requires exactly one of --mhc-allele / --serotype (a single value) plus a gene/peptide scope filter. Each peptide's support is split into ranked tiers, strongest first:

best_support names the strongest tier with any rows. A cancer / healthy / adjacent / other source breakdown (n_cancer_rows, ...) comes along for free, so you can prioritize peptides seen in tumors over healthy tissue.

Filters on hitlist export binding

Same shape as the observations filters minus the MS-specific ones (--mono-allelic, --instrument-type, --acquisition-mode). --source accepts only iedb or cedar — supplementary data is MS-only and never appears in the binding index.

hitlist export binding --gene PRAME --class I -o prame_binding.csv
hitlist export binding --mhc-allele HLA-A*02:01 --serotype Bw4

Filters on hitlist export training

hitlist export training exposes the shared pMHC filters plus two export-shape controls:

• --include-evidence ms|binding|both chooses which canonical evidence families to compose.
• --explode-mappings expands the output to one row per (evidence row, peptide mapping) with protein_id, position, n_flank, c_flank, proteome, and proteome_source.

MS-specific filters (--mono-allelic, --instrument-type, --acquisition-mode) apply only to the MS slice. Binding rows never gain fake sample context; they remain tagged as evidence_kind="binding" with sample_match_type="not_applicable".

hitlist export training --include-evidence both --gene PRAME --class I -o prame_training.csv
hitlist export training --include-evidence ms --mono-allelic --class I -o mono_ms.csv
hitlist export training --include-evidence both --explode-mappings -o presto_training.parquet

Sample-level expression anchors (issue #140)

For line-like ms_samples (C1R, 721.221, JY and sibling EBV-LCLs, HAP1, HeLa, HEK293, THP-1, SaOS-2, A375, K562, GM12878, and common engineered derivatives) hitlist.line_expression resolves every sample to a stable expression backend + key via a 6-tier fallback hierarchy:

1. exact-line RNA / transcript quant (registry hit with shipped data)
2. parent-line / engineered-derivative RNA (e.g. HeLa.ABC-KO → HeLa)
3. line-family class anchor (EBV-LCL → GM12878; mono-allelic host → K562)
4. cancer-type surrogate (caller-supplied pirlygenes backend)
5. broad tissue / lineage surrogate (HPA)
6. no_expression_anchor

Four provenance columns — expression_backend, expression_key, expression_match_tier, expression_parent_key — are written on every row so downstream tooling can distinguish "exact JY RNA" from "generic EBV-LCL stand-in" from "melanoma cohort surrogate" instead of treating them as equally trustworthy.

CLI:

hitlist export samples --with-expression-anchors -o sample_anchors.csv

hitlist export training \
    --include-evidence ms --class I --mono-allelic \
    --with-peptide-origin \
    -o training_with_origin.parquet

hitlist export line-expression --line-key GM12878 --gene-name TP53

--with-peptide-origin attaches, per row, the argmax-TPM candidate gene (peptide_origin_gene, peptide_origin_tpm) for the peptide in the sample's resolved line. When transcript-level TPM is available the score is the sum across only those transcripts whose translation actually contains the peptide — isoforms that splice out the peptide contribute zero, and a transcript is counted once regardless of how many times the peptide appears in its protein.

Register DepMap matrices to broaden exact-line coverage:

hitlist data register depmap_rna /path/to/OmicsExpressionProteinCodingGenesTPMLogp1.csv
hitlist data register depmap_rna_transcript /path/to/OmicsExpressionTranscriptsTPMLogp1.csv
hitlist data build

A note on mono-allelic curation

Mono-allelic is a PMID-level flag, not a per-sample property. Curation lives in pmid_overrides.yaml (see mono_allelic_host and ms_samples). Rows can legitimately have is_monoallelic=True with an empty mhc_restriction — for example, supplementary contaminant peptides under a mono-allelic PMID override carry the flag but not a per-row allele. If your downstream pipeline needs a strict "mono-allelic AND has allele" subset, post-filter on is_monoallelic & mhc_restriction.str.startswith("HLA-").

Reports

hitlist report [--class I|II] [--output report.txt]

QC diagnostics

hitlist qc                                 # run all checks, print summary
hitlist qc resolution                      # allele-resolution histogram
hitlist qc normalization                   # YAML alleles whose normalize_allele output drifts
hitlist qc cross-reference                 # alleles in YAML but not data, and reverse
hitlist qc discrepancies [--by sample]     # per-PMID curation drift signals
hitlist qc plan [--top 10]                 # ranked next-PMID-to-curate roadmap
hitlist qc proteome-coverage               # per-source-organism proteome registry coverage
hitlist qc proteome-coverage --missing-only --min-rows 100

Development

./develop.sh    # install in dev mode
./format.sh     # ruff format
./lint.sh       # ruff check + format check
./test.sh       # pytest with coverage (~3 min)
./deploy.sh     # lint + test + build + upload to PyPI

See docs/pmid-curation.md for the curation YAML format and per-study overrides.