Zigong MEDS Extraction ETL

This pipeline extracts the Zigong Fourth People's Hospital ICU infection database v1.1 from PhysioNet into the MEDS format. The release holds 2,790 ICU patients with infection, one hospital admission each, from Zigong Fourth People's Hospital in Sichuan, China. It was built from the ETL_MEDS_Template and uses the config-only layout introduced by MEDS-Extract 0.7.

Usage

Zigong is a credentialed dataset. You need a PhysioNet account that has signed the project's data use agreement.

pip install git+https://github.com/rvandewater/ZIGONG_MEDS.git
export DATASET_DOWNLOAD_USERNAME=$PHYSIONET_USERNAME
export DATASET_DOWNLOAD_PASSWORD=$PHYSIONET_PASSWORD
meds-extract-run spec=Zigong output_dir=$MEDS_OUTPUT_DIR

When you run this, the program will:

1. Download the release (seven dt*.csv tables, about 840 MB) from PhysioNet into $MEDS_OUTPUT_DIR/.meds_extract_run/raw_input. Pass download_dest_dir=$RAW_INPUT_DIR to keep the raw data somewhere durable and reuse it across runs.
2. Build the MEDS cohort from those files and write it to $MEDS_OUTPUT_DIR (data/ and metadata/). All transformations and joins are declared in src/Zigong_MEDS/event_configs.yaml.

If you already have the PhysioNet release on disk, skip the download and point input_dir at the directory that contains dtBaseline.csv, dtDrugs.csv, dtICD.csv, dtLab.csv, dtNursingChart.csv, dtOutCome.csv and dtTransfer.csv:

meds-extract-run spec=Zigong output_dir=$MEDS_OUTPUT_DIR \
	do_download=false input_dir=/path/to/icu-infection-zigong-fourth-1.1

To parallelize the stages, pass a runner file:

printf 'parallelize:\n  n_workers: 8\n  launcher: joblib\n' >runner.yaml
meds-extract-run spec=Zigong output_dir=$MEDS_OUTPUT_DIR stage_runner_fp=runner.yaml

Run meds-extract-run --help for every option.

How this ETL is defined

The package contains no Python. The whole ETL is one MESSY (MEDS-Extract Specification Syntax YAML) file, src/Zigong_MEDS/event_configs.yaml. It declares:

• sources:: where the raw data lives (the PhysioNet release).
• etl:: the dataset name and sharding options.
• One block per raw table: how each row becomes one or more MEDS events.

pyproject.toml registers the file with MEDS-Extract's MEDS_extract.pipelines entry-point group under the name Zigong. That registration is what makes spec=Zigong work. No pre-MEDS step is needed: tables without PATIENT_ID (dtLab, dtNursingChart) are joined to dtBaseline on INP_NO inside the config.

Times. Every source time is an offset in hours from hospital admission. The ETL places them on a shared pseudotime anchor, 1970-01-01 00:00 = admission, which preserves order and intervals without claiming a real calendar date. Each patient has exactly one admission, so offsets never collide.

Events

Every event linked to a stay carries inpatient_stay_id (INP_NO). The R-written CSVs encode missing values as the string NA or a quoted empty string; both are treated as missing, as are text entries with no letters or digits.

Data notes

• Coarse nursing chart timestamps. dtNursingChart has about 1 million rows with a usable ChartTime but only about 23,000 distinct (patient, ChartTime) pairs. A typical timestamp covers 26 source rows, and the largest covers 266. This comes from the source release, not the ETL. In the MEDS output, one timestamp therefore holds many readings of the same measurement, on average about 15 NURSING//HEART_RATE values, and similarly for blood pressure, SpO2 and other vitals. The order of events within a timestamp carries no clinical meaning. When defining tasks or features:
  • Do not rely on order within a timestamp, such as "last value before X", minute-level features, or sequence models that treat event order as real time order.
  • Choose prediction times that do not fall inside a timestamp's group of events, and aggregate values within a timestamp (for example mean, min, or max) rather than picking one.
• Dropped nursing rows. 313,974 nursing rows (23%) belong to 723 stays that are not in the cohort (dtBaseline), and all of them have ChartTime = NA; they are dropped. Rows charted more than 24 h before admission (9,011 rows, including some corrupt offsets) or after discharge (141) are dropped too.
• Dropped labs. Results timed after hospital discharge (4,415 rows, 0.3%) and placeholder values made only of dots or asterisks are dropped.
• Coarse death dates. Death_Date has about day precision. In-hospital deaths typically fall about 12 h before the recorded discharge, so a stay's last events can follow MEDS_DEATH. See #4.
• Line and tube columns (CVC, PICC, Gastric_tube, …) hold the length in cm, used as the numeric_value, or a status such as "removed" kept as text. The numeric values of Endotracheal_intubation duplicate Endotracheal_intubation_Depth, so only its statuses are emitted.
• Bounded scores. RASS, NRS and CPOT values outside their scales (-5..+4, 0..10, 0..8) are dropped. Other vitals are not range-filtered; see #7.
• Unmapped columns. Five nursing columns are NA in every row. MAP, PSIPAP, R and FIO2 hold only unit strings. Invasive_SBP._1 is a documented technical duplicate, and F is documented as having no clinical relevance. See #6.
• Local vocabularies. Diagnoses use the release's Chinese national extension of ICD-10, and labs and drugs use local names. No standard-terminology parents are attached yet; see #1 and #8. SF-36 answers are inconsistently coded in the source; see #5.
• No ICU admission event. The release records ICU discharge but not ICU entry; see #2.

Testing

Zigong has no open demo release, so there are two synthetic checks:

• tests/e2e_synthetic_test.py writes a few fully synthetic rows with the release's exact headers and R write.csv encoding, runs meds-extract-run spec=Zigong over them, and checks the events, times, values and missing-value handling it produces.
• The Synthetic ETL workflow generates raw files straight from the config with MESSY-synth and runs the ETL over them. It fails if any event block produces no events.

uv sync --group dev
uv run pytest tests/ -v
uv run pre-commit run --all-files

# MESSY-synth needs Python >= 3.12.
uv run --with MESSY-synth==0.0.1 MESSY-synth ./src/Zigong_MEDS/event_configs.yaml -o raw_synthetic/
uv run meds-extract-run spec=Zigong output_dir=zigong_meds_synth/ \
	do_download=false input_dir=raw_synthetic/

For a reusable checklist on mapping a new dataset, see docs/MEDS_ETL_MAPPING.md.

Citation

If you use this ETL, please cite the Zigong database and its descriptor paper:

• Xu, P., Chen, L., & Zhang, Z. (2022). Critical care database comprising patients with infection at Zigong Fourth People's Hospital (version 1.1). PhysioNet. https://doi.org/10.13026/xpt9-z726
• Xu, P., Chen, L., Zhu, Y. et al. Critical Care Database Comprising Patients With Infection. Front. Public Health 10 (2022). https://doi.org/10.3389/fpubh.2022.852410

See also CITATION.cff.