geniekit

Python tools for the AACR Project GENIE Biopharma Collaborative (BPC) clinicogenomic cohorts.

• Harmonise the six public BPC cohorts (NSCLC, CRC, BrCa, PANC, Prostate, BLADDER) to one schema so that cross-cohort analyses do not have to deal with per-cohort variable names, types and code lists.
• Lines of therapy: derive first and subsequent lines from the regimen data relative to an index event that the user chooses (diagnosis, metastatic diagnosis, first progression, first systemic therapy, or any custom event), with explicit, configurable rules.
• Aggregate-only analytic layer: produce cohort-level summaries with disclosure control (minimum cell sizes, suppression, no row-level output) for users who may not access patient-level records.

geniekit ships metadata only. The data themselves come from Synapse under the GENIE access requirement (see Data access).

Status

Version 0.1.0 covers the three challenge capabilities (harmonisation, lines of therapy, aggregate-only analytic layer) on top of the foundation (release registry, data dictionary, download and loading, synthetic cohorts). Every layer has been run against the six public releases; the test suite runs on synthetic data and a simulated fixture, and repeats the real-data checks when GENIEKIT_DATA_DIR points at downloaded releases.

Module	Purpose	State
geniekit.releases	Registry of the six public BPC releases with the Synapse id of every clinical, genomic, cBioPortal and documentation file	done
geniekit.dictionary	The six Variable Synopsis workbooks compiled into one data dictionary, with cross-cohort comparison helpers	done
geniekit.io	Download from Synapse (personal access token, access-status check, MD5 verification, retries) and load release folders from disk into one CohortData object, checked against the dictionary	done
geniekit.synthetic	Dictionary-driven synthetic cohorts: every declared column, cohort-specific naming, coherent simulated histories	done
geniekit.harmonize	Crosswalk-driven normalisation of the six cohorts (renames, code-list collapses, spelling, types, derived variables, genomic format unification) with a provenance report; stacking across cohorts	done; metastasis-site rederivation follows
geniekit.lot	Lines of therapy from the regimen tables, counted from an index event the user chooses, with explicit rules, a drug vocabulary covering both schema generations, and a reason recorded for every assignment	done
geniekit.aggregate	Aggregate-only analytic layer: disclosure control (minimum cell size, complementary suppression or omission of small categories, suppression of derived statistics), summary tables, Kaplan-Meier and early-failure endpoints, genomic flags, and the CRC liver-metastasis report of Challenge 10	done

Installation

pip install geniekit

Optional extras: geniekit[synapse] (official Synapse client), geniekit[survival] (Kaplan-Meier summaries), geniekit[notebook] (Jupyter and plotting), geniekit[dev] (tests and linting).

From a clone:

pip install -e ".[dev]"
pytest tests/ -v --cov=geniekit --cov-report=term

Quick start

from geniekit import releases, dictionary

# What is available, and where on Synapse
for r in releases.list_releases():
    print(r.label, r.n_patients, r.release_folder, sorted(r.clinical))

# How the cohorts differ before any harmonisation
dd = dictionary.load()
print(dd.compare("ca_cigarette", table="cancer").summary())
print(dd.differences(table="patient").head())

Loading data (after approval, see Data access):

from geniekit import io

# download one release with a personal access token in SYNAPSE_AUTH_TOKEN
folder = io.download_release("CRC", dest="data/genie_bpc")       # data/genie_bpc/CRC_v2.0-public
crc = io.load_release(folder)                                     # CohortData with 12 tables
crc["regimen"].head()

# or point at a folder you downloaded through the Synapse website
nsclc = io.load_release("downloads/NSCLC_2.0-public")

Working without the data:

from geniekit import synthetic

fake = synthetic.generate("BLADDER", n_patients=300, seed=0)     # same tables and columns as the real release
fake["cancer_index"][["record_id", "stage_dx", "dist_mets_liver", "tt_os_dx_days"]].head()

Synthetic cohorts carry the exact column set of the corresponding release (checked against the dictionary in the test suite) and mutually consistent identifiers, intervals and outcomes, but their values are invented. They exist so that code can be written and tested before access is granted; never report numbers computed from them.

Harmonising and stacking:

from geniekit import io, harmonize

nsclc = harmonize.harmonize(io.load_release("data/genie_bpc/NSCLC_v2.0-public"))
nsclc.report.to_frame()            # every rename, recode, collapse, coercion and derivation, with counts
nsclc.report.unmapped()            # values the crosswalk did not recognise (should be empty)

stacked = harmonize.stack([nsclc, bladder, crc])   # one frame per table, cohort and schema_generation columns
stacked["cancer_index"]["ca_cigarette"]           # generation-1 ca_lung_cigarette and generation-2 ca_cigarette, unified
stacked.availability("cancer_index")               # which cohorts contributed each column

What the crosswalk does (geniekit crosswalk --basis prints the full list with the basis for each rule in the data guides):

• renames generation-1 variables to the generation-2 name where the definitions coincide (ca_lung_cigarette, dmets_stage_i_iii, and the liver, breast and brain/CNS metastasis flags with their time intervals);
• collapses the detailed generation-1 NAACCR ethnicity, race and sex code lists onto the generation-2 categories, keeping the detailed value in a _detail column;
• reconciles cohort-specific level sets (ca_heme_type, pfs_cohort, bca_subtype, tumor-deposit text) and spelling differences (dashes, capitalisation, <1 versus < 1);
• converts cpt_seq_date to an integer year and naaccr_seer_sum_stage to the NAACCR #759 code plus label in every cohort;
• derives race_ethnicity for every cohort with the rule behind the released generation-1 variable, recovered from the NSCLC and CRC releases where it reproduces every released value (the released value is kept in race_ethnicity_released; a conservative alternative is selectable in the crosswalk), and pfs_time_origin, which states whether the PFS endpoints run from diagnosis or from the first distant metastasis;
• unifies data_fusions.txt and data_sv.txt into one structural-variant schema and melts the copy-number matrix into a long table.

What it deliberately does not do: the remaining generation-1 metastasis regions (dmets_abdomen, dmets_bone, dmets_extremity, dmets_head_neck, dmets_pelvis, dmets_thorax) keep their names, because a code-by-code comparison of the classification appendices (data-raw/build_metastasis_sites.py) shows they do not coincide with any generation-2 site: generation-1 Abdomen includes the peritoneum, adrenals and intra-abdominal nodes; Thorax includes lung, pleura, pericardium, ribs and intrathoracic nodes; Pelvis includes pelvic nodes; Extremity is split across Skin, Other and Lymph nodes. Only liver, breast and brain/CNS (the latter up to one NOS code) have the same code set in both generations. The harmonised 17-site set is rederived from the imaging and pathology site codes in a later step rather than approximated by renaming.

Lines of therapy:

from geniekit import io, lot

crc = io.load_release("data/genie_bpc/CRC_v2.0-public")
result = lot.derive(crc, index="advanced")   # lines counted from stage IV diagnosis or first distant metastasis
result.lines                                  # one row per line: start, end, agents, backbone agents, regimens, why it ended
result.regimens                               # every regimen with its status, line number, role and the reason
result.cancers                                # one row per cancer: index event, number of lines, first-line agents
result.summary()

# the index event is the user's choice
lot.derive(crc, index="first_progression")    # first radiologist or oncologist assessment of progression
lot.derive(crc, index="first_systemic")       # start of the first line-defining regimen
lot.derive(crc, index=my_table)               # record_id, ca_seq, index_days (for example a trial-entry date)

# and so are the rules
rules = lot.LineRules(restart_gap_days=90, progression_aware=True, pre_index="line_zero")
lot.derive(crc, index="advanced", rules=rules)
lot.derive(prostate, index="advanced", rules="prostate")   # ADT and first-generation antiandrogens as a backbone

How a line is formed (every setting has a default in lot.LineRules and is written into the output):

• regimens that start before the index are excluded (or kept as line 0);
• the first regimen with a line-defining agent on or after the index starts line 1; a regimen starting within combination_window_days (28) of the line start joins it as part of the initial combination;
• a later regimen whose agents are all already in the line continues it (maintenance, de-escalation or resumption after a holiday), unless the gap since the last known administration exceeds restart_gap_days;
• a regimen with a new agent starts the next line; with progression_aware=True it does so only when a progression assessment (imaging or oncologist note) falls between the current line's start and the new regimen, and a switch without one continues the line;
• LHRH agonists and antagonists (backbone role) and leucovorin (modulator) never start or end a line and are recorded as components; locally administered agents (intravesical BCG, valrubicin, and any regimen whose released route is intravesical, intraperitoneal, intrapleural, intralesional or intrathecal) are not lines of systemic therapy;
• an investigational drug counts as a line-defining agent whose identity and end date are masked, so two investigational regimens are two lines.

Drug identities come from resources/drug_vocabulary.csv: every one of the 205 drug labels released across the six cohorts is mapped to a canonical agent, a class and a role (lot.load_vocabulary()). The two schema generations spell eleven agents differently ("Gemcitabine Hydrochloride" versus "Gemcitabine HCL", "Leucovorin Calcium" versus "Leucovorin", "Tamoxifen Citrate" versus "Tamoxifen", and so on) and generation 2 misspells goserelin; the vocabulary resolves all of them. Classes and roles were assigned by the package author and are kept in one reviewable table (data-raw/build_drug_vocabulary.py).

Aggregate-only analysis:

from geniekit import io, aggregate

crc = io.load_release("data/genie_bpc/CRC_v2.0-public")
report = aggregate.liver_metastasis_report(crc, control=aggregate.DisclosureControl(min_count=5))
report.summary()                                 # sixteen tables, with the number of suppressed cells in each
report["early_failure_by_liver_mets"].frame      # early treatment failure at 6 and 9 months, liver versus no liver metastasis
report["alterations_by_liver_mets"].frame        # APC, TP53, KRAS, NRAS, RAS, BRAF, BRAF V600E, SMAD4 by liver metastasis
report.to_dir("outputs/crc_liver")               # one CSV per table plus report.json: settings, notes, every suppression

# the same building blocks for other questions
table = aggregate.analytic_table(crc, genes=["KRAS", "BRAF"], msi=True)    # patient-level; stays with you
aggregate.proportion_table(table, "first_line", by="stage_dx").frame     # aggregate; can be shared
aggregate.km_table(table, "os_from_index_days", "died", by="liver_mets", at_months=[12, 24]).frame

What the layer guarantees: every function returns a SafeTable whose cells are counts, percentages, intervals, medians and quartiles (never a minimum or maximum), estimates or p-values. A count below min_count (default 5) is replaced by a marker, and every statistic derived from it is replaced too. Where a suppressed count could be recovered from a table margin, the layer compares the records hidden by suppressing a second cell with the records hidden by omitting the small row or column, takes the cheaper option, and writes what it did into the table's notes. Groups too small to summarise are omitted with a note instead of appearing as columns of markers. The report's report.json records the index event, the line rules, the disclosure settings and each suppressed cell, so a reader of the aggregate output can see how it was produced.

The notebook examples/crc_liver_metastasis.ipynb walks through the report; it uses the real CRC release when GENIEKIT_DATA_DIR is set and a synthetic cohort otherwise.

Command line:

geniekit releases --files
geniekit dictionary lookup naaccr_seer_sum_stage --table cancer --values
geniekit dictionary diff --table cancer --kind dtype --kind value_set
geniekit dictionary presence --table patient --only-partial
geniekit access                                   # token and GENIE approval status
geniekit download CRC --dest data/genie_bpc       # after approval
geniekit inspect data/genie_bpc/CRC_v2.0-public   # table summary and column check
geniekit crosswalk --validate                     # the harmonisation rules, checked against the dictionary
geniekit harmonize data\genie_bpc\NSCLC_v2.0-public data\genie_bpc\BLADDER_v1.0-public --out data\harmonized
geniekit lot data\genie_bpc\CRC_v2.0-public --index advanced --out data\lot_crc
geniekit lot data\genie_bpc\Prostate_v1.0-public --rules prostate --progression-aware --out data\lot_prostate
geniekit aggregate data\genie_bpc\CRC_v2.0-public --min-count 5 --out outputs\crc_liver

The six public releases

Cohort	Version	Patients	Institutions	Synapse	Schema generation
NSCLC	v2.0-public	1846	DFCI, MSK, UHN, VICC	syn27056697	1
CRC	v2.0-public	1485	DFCI, MSK, VICC	syn30991637	1
BrCa	v1.0-public	1130	DFCI, MSK, VICC	syn71825332	2
PANC	v1.0-public	1109	DFCI, MSK, UHN, VICC	syn72666404	2
Prostate	v1.0-public	1116	DFCI, MSK, UHN, VICC	syn73719283	2
BLADDER	v1.0-public	716	DFCI, MSK, UHN, VICC	syn75201334	2

"Schema generation" is a geniekit label, not a GENIE term. The two 2022 releases (NSCLC, CRC) and the four 2025 to 2026 releases share most of their variables, but the later releases renamed or recoded a measurable subset. Of the 1105 distinct (table, variable) pairs declared across the six Variable Synopses, 505 are declared in all six cohorts. Examples of what differs:

• smoking history is ca_lung_cigarette in generation 1 and ca_cigarette in generation 2;
• distant metastasis sites are dmets_* / dx_to_dmets_* (9 sites) in generation 1 and dist_mets_* / dx_to_dist_mets_* (17 sites) in generation 2;
• naaccr_seer_sum_stage is declared as character in the generation-1 synopses and numeric in generation 2 (the released files hold the numeric NAACCR code in all six, so this one is a documentation difference);
• cpt_seq_date is character in generation 1 and numeric in generation 2;
• ca_grade, pfs_cohort, ca_heme_type, ca_clin_t_stage and others have different permitted value sets;
• tumor-marker tables exist only for CRC, BrCa, PANC and Prostate; radiation-therapy tables only for PANC, Prostate and BLADDER;
• structural variants are data_fusions.txt in generation 1 and data_sv.txt in generation 2;
• UHN contributes to NSCLC, PANC, Prostate and BLADDER but not to CRC or BrCa.

geniekit dictionary diff lists every such difference. The dictionary also records the 17 variables the synopses declare that the released files do not contain (dictionary.load().declared_not_released()); every released clinical column is declared, so the loader's column check is exact.

Data access

1. Create a Synapse account and request access to the GENIE BPC data: open https://www.synapse.org/Synapse:syn27056700 and click "Request Access". One access requirement (id 9606645) covers both the BPC cohorts and the main GENIE registry.
2. Complete the Sage Bionetworks Access and Compliance Team (ACT) form linked from that dialog, using your Synapse username. Both steps are required.
3. After approval, create a personal access token (Synapse profile, Settings, Personal Access Tokens) with "view" and "download" scope and expose it as SYNAPSE_AUTH_TOKEN (see .env.example). Never commit the token. geniekit access reports whether the token works and whether the access request is approved, pending or missing.

The per-cohort Analytic Data Guides and Variable Synopses are public and are linked from https://aacrprojectgenie.org/data/.

Relationship to the R package genieBPC

The R package genieBPC by the MSK Biostatistics team downloads BPC releases and builds analytic cohorts one cohort at a time. geniekit uses the same logical table names where they exist (releases.GENIEBPC_TABLE_NAMES maps between the two) so that workflows can move between the languages, and adds the cross-cohort harmonisation, line-of- therapy and aggregate-only layers.

Development

git clone https://github.com/jemsbhai/geniekit.git
cd geniekit
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -e ".[dev]"
pytest tests/ -v --cov=geniekit --cov-report=term
ruff check src tests data-raw
python scripts/check_vocabulary.py

Regenerating the packaged metadata (requires network access, no Synapse login):

python data-raw/build_releases.py
python data-raw/build_dictionary.py
python data-raw/build_drug_vocabulary.py   # with GENIEKIT_DATA_DIR set, checks coverage of the released labels

License

MIT. See LICENSE. See NOTICE.md for the provenance of the packaged metadata and for how to cite the data.