MC-MED MEDS Extraction ETL

An ETL to extract the MC-MED (Multimodal Clinical Monitoring in the Emergency Department) dataset into the MEDS format. Built on MEDS-Extract.

Usage

pip install MC-MED-MEDS
suMEDS-extract spec=MC-MED output_dir=/path/to/output

suMEDS' suMEDS-extract (installed with this package) runs the ETL with meds-extract-run spec=MC-MED, then fills the official ICD titles and parents into metadata/codes.parquet (see Enriching code metadata with suMEDS). Arguments before a -- go to meds-extract-run, and arguments after it go to suMEDS-enrich. For the plain ETL, run meds-extract-run spec=MC-MED output_dir=... with the same arguments.

Raw data is downloaded automatically per the sources: block of the packaged configuration; set any required credentials as environment variables (see below). To run against pre-staged raw data instead, pass do_download=false input_dir=/path/to/raw/data.

The output directory will contain the extracted MEDS dataset (data/ and metadata/, per the MEDS standard), with dataset name and version provenance stamped in metadata/dataset.json automatically.

Dataset access

MC-MED v1.0.1 (DOI 10.13026/wvyw-g663) is a credentialed-access PhysioNet dataset: you need a credentialed PhysioNet account, the required CITI training, and a signed DUA for the project. Export your PhysioNet credentials before running the ETL:

export DATASET_DOWNLOAD_USERNAME=... # your PhysioNet username
export DATASET_DOWNLOAD_PASSWORD=... # and password
suMEDS-extract spec=MC-MED output_dir=/path/to/output download_dest_dir=/path/to/raw

Only the tabular CSVs the ETL reads are downloaded (visits, orders, labs, rads, numerics, meds, pmh); the WFDB waveform tree is skipped. If you already have the files, use do_download=false input_dir=/path/to/mc-med/1.0.1.

What gets extracted

subject_id is MC-MED's MRN. Tables keyed only by visit (CSN) are joined to visits.csv to obtain it, and every visit-scoped event carries its visit's CSN in an extra csn column (useful for linking back to the raw tables and the waveforms).

Code descriptions (order names, medication names, PMH descriptions) are written to metadata/codes.parquet; official ICD titles and parents come from suMEDS (see below). A missing value in a categorical code component (race, ethnicity, arrival mode, payor, acuity, order type, unit, admit service, dispositions) becomes UNK rather than dropping the row. The CSVs write these as quoted empty strings (""), which the config treats the same as null.

Modelling choices and caveats

• Timestamps are shifted per patient into 2150–2350 by the dataset authors. They are parsed leniently against several ISO-8601 layouts. Rows matching none are dropped, and MEDS-Extract logs a WARNING with the count, so check the logs of your first full run. These per-event warnings are in $OUTPUT_DIR/convert_to_MEDS_events/.logs/, not in the runner's own output. In v1.0.1 every timestamp matches the first layout, so every dropped row is one whose time is missing.
• Triage has no timestamp of its own and is placed at ED arrival.
• Age is perturbed (±2 years, clipped to 18–90) and there is no birth date, so it is an AGE measurement at each arrival, and there is no MEDS_BIRTH.
• Deaths are in-hospital only. MC-MED records a death as a hospital discharge disposition starting with Expired, and no death registry is linked, so deaths after discharge are not captured and a subject without MEDS_DEATH is not known to be alive. MEDS_DEATH is placed at the reconstructed hospital discharge time (see below), or at ED departure for the few death visits without a Hosp_LOS. MEDS_DEATH is not always a subject's last event: like the discharge time, it is only accurate to about a day, so some in-stay vitals, labs or orders come after it (typically a few hours), and the source data has a few patients with a later ED visit or a past-medical-history or home-medication entry dated after their death.
• Several values in one field. Dx_ICD10 and Dx_ICD9 can hold up to 4 codes (joined by ", "), and CC up to 8 chief complaints (joined by "," or ", "). Each value becomes its own event. MEDS-Extract cannot turn one row into a variable number of events, so the config has fixed slots (ed_diagnosis_icd10_1 … _4, triage_chief_complaint_1 … _8). Empty slots produce a null code and are dropped; the per-event WARNINGs count those as "with null code", which is expected. A field with more values than slots would lose the extra ones; MC-MED v1.0.1 has none.
• ED diagnosis names. Dx_name is the EHR's own name for the visit's diagnosis and varies for one ICD-10 code, so it is the text_value of the visit's first ICD-10 event rather than a code description.
• Past medical history codes are written without the dot in pmh.csv (CodeType Dx10 / Dx9). The ETL restores the dot (after the 3rd character, or the 4th for ICD-9 E codes), so past and ED diagnoses share one layout, e.g. E119 → PMH//ICD10CM//E11.9.
• Hospital discharge has no timestamp in MC-MED. It is reconstructed as Arrival_time + Hosp_LOS days, assuming Hosp_LOS covers the whole encounter, ED stay included. Hosp_LOS is a whole number of days (0 for about 4% of admissions), so the time is clamped to no earlier than ED departure. It is still only accurate to about a day: in-stay vitals, labs or orders can occur after it.
• Home medications are placed at Entry_date (when they were recorded), not Start_date, so they never appear before the information was available.
• Not extracted: waveforms and waveform_summary.csv, the next-visit outcome columns (Hours_to_next_visit, Dispo_class_next_visit; these follow from the next visit's events), and the lab panel-level fields (Display_name, Abnormal, normal ranges).
• Splits: MEDS-Extract draws its own random subject-level splits. To use MC-MED's provided split_random_* / split_chrono_* files instead, convert them to a {split: [subject_ids]} JSON and pass it via external_splits_json_fp.

Enriching code metadata with suMEDS

ICD vocabulary work lives in suMEDS rather than in this ETL. The package ships the rules for it, src/MC_MED_MEDS/sumeds_rules.yaml, registered in suMEDS's sumeds.rules entry-point group under the same name as the MEDS-Extract spec, MC-MED. That is how suMEDS-extract spec=MC-MED finds them, and --rules MC-MED works for any suMEDS command. suMEDS-extract replaces metadata/codes.parquet only once the enriched table is complete, and it stays valid MEDS code metadata. By default the ICD data comes from suMEDS's ontology packs, fetched from GitHub on first use and cached; no Athena download is needed. suMEDS options go after a --:

# Also look codes up in your own OHDSI Athena download:
suMEDS-extract spec=MC-MED output_dir=/path/to/output -- --athena-csv /path/to/athena
# Use only already cached packs (no network):
suMEDS-extract spec=MC-MED output_dir=/path/to/output -- --offline

Running it again on an existing output skips the pipeline stages that are already done and enriches again. To enrich the codes.parquet of an existing build instead:

suMEDS-enrich "$OUTPUT_DIR/metadata/codes.parquet" -o "$OUTPUT_DIR/metadata/codes.parquet" --rules MC-MED

The rules validate the ED_DIAGNOSIS//ICD10CM//…, ED_DIAGNOSIS//ICD9CM//…, PMH//ICD10CM//… and PMH//ICD9CM//… codes against suMEDS's versioned ICD-10-CM and ICD-9-CM data (built from the CDC/NCHS and CMS files). They add the official title where a code has none, the vocabulary ancestors as parent_codes, and the nearest ancestor that is itself a code of the same family (ED diagnoses and past medical history are linked separately). The same rules work with suMEDS … --rules … to build a privacy-masked code catalog of the output.

Summary statistics

tests/regression/full_summary_stats.json is a meds_summary_stats fingerprint of the MEDS dataset this ETL builds from the full MC-MED 1.0.1 release with suMEDS-extract spec=MC-MED, so its code-metadata coverage includes the suMEDS enrichment. It covers subject, event and measurement counts, digests of the code vocabulary, per-subject, value and time distributions, code-metadata coverage, the splits and the schema. It names no code and reports no cell of fewer than 20 subjects and no minimum or maximum, which is what makes it safe to publish. Because it is committed next to the ETL, its diff shows what an ETL change did to the output, and two ETL versions can be compared without access to the data.

Regenerate it in every pull request that changes the output, from a build of that branch:

# Build the release with this checkout, then summarize it
scripts/summary_stats.sh full --input-dir $RAW_INPUT_DIR
# Or summarize a build of this checkout you already have
scripts/summary_stats.sh full --meds-dir $MEDS_OUTPUT_DIR

Both print how the statistics changed against the committed baseline before overwriting it. Add --compare to check a build without writing anything; it exits 1 when tests/regression/policy.yaml flags an error. Rebuilding the same commit reproduces the baseline exactly, so any finding is a change in the output. CI cannot rebuild the credentialed release, so the Summary stats workflow only warns when a pull request changes src/ (the config or the suMEDS rules) without regenerating the baseline. A MEDS-Extract, suMEDS or polars upgrade can also move the output, so check it with --compare.

How this package works

This is a config-only ETL package: its entire substance is src/MC_MED_MEDS/configs/event_configs.yaml, a MESSY (MEDS-Extract Specification Syntax YAML) file declaring where the raw data lives (sources:) and how its rows map to MEDS events. The MEDS_extract.pipelines entry point in pyproject.toml registers that file under the name MC-MED, and MEDS-Extract's generic runner (meds-extract-run) does everything else: staged download, the canonical extraction pipeline, and provenance stamping. See the MEDS-Extract documentation for the MESSY format and runner details. A second entry point registers src/MC_MED_MEDS/sumeds_rules.yaml with suMEDS under the same name, which is all suMEDS-extract spec=MC-MED needs to enrich the output.

Development

uv sync --all-groups
uv run pre-commit install
uv run pytest tests/ -v

See CONTRIBUTORS.md for the contribution guidelines, the checks, regenerating the summary-stats baseline, and releasing.

The test suite validates offline that the packaged MESSY file is registered and parses, then runs the full pipeline end to end on a tiny, entirely synthetic dataset with MC-MED's documented column layout and the quirks of the real files (tests/conftest.py, tests/test_e2e_synthetic.py), and checks that the packaged suMEDS rules match the codes it writes (tests/test_sumeds_rules.py) and that suMEDS-extract spec=MC-MED enriches them, offline against a tiny synthetic ontology pack (tests/test_run_with_enrichment.py). No real MC-MED data is used in CI.

To release, bump project.version in pyproject.toml with uv version --bump minor|patch|major in a pull request and merge it into main. The publish workflow then uploads the new version to PyPI (through a PyPI trusted publisher) and creates its tag and GitHub Release; see CONTRIBUTORS.md. Optionally archive releases on Zenodo for a citable DOI, and update CITATION.cff accordingly.

This repository was generated from the ETL_MEDS_Template copier template; run uvx copier update (with a clean git tree) to pull in template improvements.