MEDS-extract 0.6.2

MEDS ETL building support leveraging MEDS-Transforms.

MEDS-Extract

MEDS Extract is a Python package that leverages the MEDS-Transforms framework to build efficient, reproducible ETL (Extract, Transform, Load) pipelines for converting raw electronic health record (EHR) data into the standardized MEDS format. If your dataset consists of files containing patient observations with timestamps, codes, and values, MEDS Extract can automatically convert your raw data into a compliant MEDS dataset in an efficient, scalable, and communicable way.

🚀 Quick Start

1. Install via pip:

pip install MEDS-extract

[!NOTE] The current release line is MEDS-Extract 0.6.x, which pins meds ~=0.4.0 and MEDS-transforms ~=0.6.0. Hotfixes are released within that namespace as required; older releases remain available on PyPI. Each code/time/property value in the MESSY file is a dftly expression (see Event Configuration Deep Dive).

[!WARNING] Breaking change in v0.6.0: The MESSY event configuration syntax has changed significantly. Event field expressions (e.g., code and time) are now parsed by dftly, a lightweight declarative expression language. The old col() function syntax and list-based code construction are no longer supported. The time_format key has been replaced by inline type casting with the as operator (e.g., $timestamp as "%Y-%m-%d"). See the Event Configuration Deep Dive for the updated syntax.

2. Prepare your raw data

Ensure your data meets these requirements:

• File-based: Data stored in .csv, .csv.gz, or .parquet files. These may be stored locally or in the cloud, though intermediate processing currently must be done locally.
• Comprehensive Rows: Each file contains a dataframe structure where each row contains all required information to produce one or more MEDS events at full temporal granularity, without additional joining or merging.
• Integer subject IDs: The subject_id column must contain integer values (int64). You can use subject_id_expr: "hash($string_col)" in your MESSY file to automatically convert string IDs to integers.

If these requirements are not met, you may need to perform some pre-processing steps to convert your raw data into an accepted format, though typically these are very minor (e.g., joining across a join key, converting time deltas into timestamps, etc.).

3. Create a MESSY file for your messy data!

The secret sauce of MEDS-Extract is how you configure it to identify events within your raw data files. This is done by virtue of the "MEDS-Extract Specification Syntax YAML" (MESSY) file. Event field values like code and time are written as dftly expressions -- a small declarative language for column references, string interpolation, type casting, and arithmetic. See the dftly documentation for the full expression syntax.

Let's see an example of this event configuration file in action:

# Global subject ID column (can be overridden per file)
subject_id_col: patient_id

# File-level configurations
patients:
  subject_id_col: MRN # This file has a different subject ID column
  demographics: # One kind of event in this file.
    code: f"DEMOGRAPHIC//{$gender}"
    time:       # Static event
    race: $race           # Column references are `$`-prefixed
    ethnicity: $ethnicity

admissions:
  admission: # One kind of event in this file.
    code: f"HOSPITAL_ADMISSION//{$admission_type}"
    time: $admit_datetime as "%Y-%m-%d %H:%M:%S"
    department: $department # Extra columns get tracked
    insurance: $insurance

  discharge: # A different kind of event in this file.
    code: f"HOSPITAL_DISCHARGE//{$discharge_location}"
    time: $discharge_datetime as "%Y-%m-%d %H:%M:%S"

lab_results:
  lab:
    code: f"LAB//{$test_name}//{$units}"
    time: $result_datetime as "%Y-%m-%d %H:%M:%S"
    numeric_value: $result_value # This will get converted to a numeric
    text_value: $result_text # This will get converted to a string

This file is also called the "Event conversion configuration file" and is the heart of the MEDS Extract system.

[!IMPORTANT] Every code, time, and property value is a dftly expression, so the syntax matters:

• A column reference must be $-prefixed: $gender. A bare, unquoted word (e.g. gender) is a string literal, not a column — a common gotcha.
• String interpolation uses an f-string: f"DEMOGRAPHIC//{$gender}".
• Type casts use as (or the equivalent ::): $admit_datetime as "%Y-%m-%d %H:%M:%S".

See the Event Configuration Deep Dive for the full syntax (including when YAML quoting is required).

4. Assemble your pipeline configuration

Beyond your extraction event configuration file, you also need to specify what pipeline stages you want to run. You do this through a typical MEDS-Transforms pipeline configuration file. Here is a typical pipeline configuration file example. Values like $RAW_INPUT_DIR are placeholders for your own paths or environment variables and should be replaced with real values:

input_dir: $RAW_INPUT_DIR
output_dir: $PIPELINE_OUTPUT

description: This pipeline extracts a dataset to MEDS format.

etl_metadata:
  dataset_name: $DATASET_NAME
  dataset_version: $DATASET_VERSION

# Points to the event conversion (MESSY) yaml file defined above. Replace with a real path.
event_conversion_config_fp: $EVENT_CONVERSION_CONFIG
# The shards mapping is stored in the root of the final output directory.
shards_map_fp: ${output_dir}/metadata/.shards.json

# Used if you need to load input files from cloud storage.
cloud_io_storage_options: {}

stages:
  - shard_events:
      data_input_dir: ${input_dir}
  - split_and_shard_subjects
  - convert_to_subject_sharded
  - convert_to_MEDS_events
  - merge_to_MEDS_cohort
  - extract_code_metadata
  - finalize_MEDS_metadata
  - finalize_MEDS_data

Save it on disk to $PIPELINE_YAML (e.g., pipeline_config.yaml).

[!NOTE] A pipeline with these defaults is provided at MEDS_extract.configs._extract.yaml. Instead of writing your own pipeline file you can reference the packaged one directly with the pkg:// prefix (note the .yaml suffix is required) and supply the per-run values as overrides. The packaged config ships no dataset name/version, so pass those too:

MEDS_transform-pipeline pkg://MEDS_extract.configs._extract.yaml \
	--overrides \
	input_dir="$RAW_INPUT_DIR" \
	output_dir="$PIPELINE_OUTPUT" \
	event_conversion_config_fp="$EVENT_CONVERSION_CONFIG" \
	dataset.name="$DATASET_NAME" \
	dataset.version="$DATASET_VERSION"

5. Run the extraction pipeline

MEDS-Extract does not have a stand-alone CLI runner; instead, you run it via the default MEDS-Transforms pipeline runner, passing your pipeline configuration file as the first positional argument (there is no pipeline_config_fp= flag):

MEDS_transform-pipeline "$PIPELINE_YAML"

Any field in the pipeline file can be overridden on the command line after --overrides, e.g. MEDS_transform-pipeline "$PIPELINE_YAML" --overrides event_conversion_config_fp=/path/to/messy.yaml.

The result of this will be an extracted MEDS dataset in the specified output directory!

📊 End-to-End Example

MEDS Extract ships with a small synthetic dataset in the example/ directory. Here we run the full pipeline and inspect the output. This section also serves as an automated test — it is executed by pytest via --doctest-glob.

>>> import subprocess, tempfile, shutil, json
>>> from pathlib import Path
>>> import polars as pl
>>> from pretty_print_directory import print_directory, PrintConfig

First, copy the example data into a temporary directory and run the pipeline:

>>> tmpdir = tempfile.mkdtemp()
>>> _ = shutil.copytree("example/raw_data", f"{tmpdir}/raw_data")
>>> _ = shutil.copy("example/event_cfg.yaml", tmpdir)
>>> result = subprocess.run(
...     f"MEDS_transform-pipeline "
...     f"pkg://MEDS_extract.configs._extract.yaml "
...     f"--overrides "
...     f"input_dir={tmpdir}/raw_data "
...     f"output_dir={tmpdir}/output "
...     f"event_conversion_config_fp={tmpdir}/event_cfg.yaml "
...     f"dataset.name=EXAMPLE "
...     f"dataset.version=1.0",
...     shell=True, capture_output=True,
... )
>>> assert result.returncode == 0, result.stderr.decode()[-500:]

The pipeline produces MEDS-format parquet shards split into train/tuning/held_out:

>>> output = Path(f"{tmpdir}/output")
>>> print_directory(output / "data", PrintConfig(ignore_regex=r"\.logs"))
├── held_out
│   └── 0.parquet
├── train
│   └── 0.parquet
└── tuning
    └── 0.parquet

Each shard contains the standard MEDS columns:

>>> df = pl.read_parquet(output / "data" / "train" / "0.parquet")
>>> sorted(df.columns)
['code', 'code_components', 'numeric_value', 'source_block', 'subject_id', 'time']
>>> df.schema["subject_id"]
Int64
>>> df.schema["code"]
String

MEDS-Extract also adds provenance and structure columns to help trace and query events. The source_block column tracks which MESSY config block produced each event:

>>> df.group_by("source_block").len().sort("source_block")
shape: (7, 2)
┌─────────────────────┬─────┐
│ source_block        ┆ len │
│ ---                 ┆ --- │
│ str                 ┆ u32 │
╞═════════════════════╪═════╡
│ diagnoses/dx        ┆ 10  │
│ labs_vitals/lab     ┆ 70  │
│ medications/med     ┆ 10  │
│ patients/dob        ┆ 8   │
│ patients/dod        ┆ 1   │
│ patients/eye_color  ┆ 8   │
│ patients/hair_color ┆ 8   │
└─────────────────────┴─────┘

The code_components struct column preserves the individual column values that were combined to form the code. This enables queries on code components without parsing the code string — for example, finding all Glucose readings regardless of units:

>>> glucose = df.filter(
...     pl.col("code_components").struct.field("test_name") == "Glucose (mg/dL)"
... )
>>> glucose.select("subject_id", "time", "numeric_value").sort("subject_id", "time").head(3)
shape: (3, 3)
┌────────────┬─────────────────────┬───────────────┐
│ subject_id ┆ time                ┆ numeric_value │
│ ---        ┆ ---                 ┆ ---           │
│ i64        ┆ datetime[μs]        ┆ f32           │
╞════════════╪═════════════════════╪═══════════════╡
│ 1          ┆ 2025-03-09 15:18:00 ┆ 122.290001    │
│ 1          ┆ 2025-06-05 17:02:00 ┆ 185.919998    │
│ 2          ┆ 2024-08-12 20:57:00 ┆ 157.539993    │
└────────────┴─────────────────────┴───────────────┘

The metadata directory contains a dataset descriptor, code metadata, and subject splits:

>>> print_directory(output / "metadata", PrintConfig(ignore_regex=r"\.shards|\.logs"))
├── codes.parquet
├── dataset.json
└── subject_splits.parquet
>>> meta = json.loads((output / "metadata" / "dataset.json").read_text())
>>> meta["dataset_name"]
'EXAMPLE'
>>> splits = pl.read_parquet(output / "metadata" / "subject_splits.parquet")
>>> sorted(splits["split"].unique().to_list())
['held_out', 'train', 'tuning']
>>> len(splits)
10

The event config includes _metadata blocks that link events to description files. Lab descriptions use full matching (the metadata table has the same test_name column as the code). Medication descriptions use partial matching via _match_on — the code is f"{$medication_name}//{$dose}" but the metadata only has medication_name:

>>> codes = pl.read_parquet(output / "metadata" / "codes.parquet")
>>> codes.filter(pl.col("code").str.starts_with("Metformin") | (pl.col("code") == "Glucose (mg/dL)")).sort("code")
shape: (2, 3)
┌───────────────────┬─────────────────────┬────────────────────────────────┐
│ code              ┆ description         ┆ code_template                  │
│ ---               ┆ ---                 ┆ ---                            │
│ str               ┆ str                 ┆ str                            │
╞═══════════════════╪═════════════════════╪════════════════════════════════╡
│ Glucose (mg/dL)   ┆ Blood glucose level ┆ $test_name                     │
│ Metformin//500 mg ┆ Antidiabetic        ┆ f"{$medication_name}//{$dose}" │
└───────────────────┴─────────────────────┴────────────────────────────────┘
>>> _ = shutil.rmtree(tmpdir)

Real-World Datasets

MEDS Extract has been successfully used to convert several major EHR datasets, including MIMIC-IV.

📖 Event Configuration Deep Dive

The event configuration file is the heart of MEDS Extract. Here's how it works:

Basic Structure

relative_table_file_stem:
  event_name:
    code: [required] How to construct the event code (dftly expression)
    time: [required] Timestamp expression (set to null for static events)
    property_name: $column_name  # Additional properties (also dftly expressions)
    _metadata:                  # Optional: link to external metadata tables
      metadata_file_prefix:
        output_column: source_column

All code and time values are parsed as dftly expressions. dftly is a lightweight declarative expression language for data transformations. The key syntax elements are:

• Column references: $-prefixed names (e.g., $test_name). A bare, unquoted word (e.g. test_name) is a string literal, not a column reference.
• String literals: a quoted value (e.g., "ADMISSION") or a single bare token (e.g., MEDS_BIRTH)
• String interpolation: an f-string with $-prefixed columns in braces (e.g., f"LAB//{$test_name}//{$units}")
• Type casting: the as operator, or the equivalent :: (e.g., $timestamp as "%Y-%m-%d" / $timestamp::"%Y-%m-%d", to parse a datetime)
• Arithmetic: $a + $b, $val * 2
• Hashing: hash($mrn) for converting string IDs to integers

[!NOTE] Quoting these expressions in YAML is optional for the forms shown here, but YAML requires it when a value would otherwise be misread (e.g. one beginning with {, [, or *). The shipped example/ configs single-quote exactly the expressions YAML would otherwise mangle — the f-strings and the ::/as casts (e.g. code: 'f"EYE_COLOR//{$eye_color}"', time: '$dob::"%Y-%m-%dT%H:%M:%S"') — while leaving bare literals (MEDS_BIRTH) and plain $column references unquoted.

Code Construction

Event codes can be built in several ways:

# Simple string literal
vitals:
  heart_rate:
    code: "HEART_RATE"

# Column reference (the value of the `measurement_type` column)
vitals:
  heart_rate:
    code: $measurement_type

# Composite codes with string interpolation (joined with "//")
vitals:
  heart_rate:
    code: f"VITAL_SIGN//{$measurement_type}//{$units}"

These rules are enforced by an executable doctest (run in CI), so the examples above cannot silently drift from the parser:

>>> import polars as pl
>>> from MEDS_extract.convert_to_MEDS_events.convert_to_MEDS_events import extract_event
>>> raw_demo = pl.DataFrame({
...     "subject_id": [1],
...     "measurement_type": ["HR"],
...     "units": ["bpm"],
...     "charttime": ["03/09/2025 15:18"],
... })
>>> def code_of(expr):  # the ``code`` produced by a dftly expression
...     return extract_event(raw_demo, {"code": expr, "time": None})["code"].to_list()
>>> code_of('"HEART_RATE"')                                  # quoted string literal
['HEART_RATE']
>>> code_of("$measurement_type")                             # column reference ($-prefixed)
['HR']
>>> code_of("measurement_type")                              # bare word -> literal, NOT the column
['measurement_type']
>>> code_of('f"VITAL_SIGN//{$measurement_type}//{$units}"')  # f-string interpolation
['VITAL_SIGN//HR//bpm']
>>> extract_event(  # type cast with `as` (or the equivalent `::`)
...     raw_demo, {"code": "VITALS", "time": '$charttime as "%m/%d/%Y %H:%M"'}
... )["time"].to_list()
[datetime.datetime(2025, 3, 9, 15, 18)]

Time Handling

# Source column that is already datetime-typed (no cast needed)
lab_results:
  lab:
    time: $result_time

# A string column: parse it with an explicit format via a type cast
lab_results:
  lab:
    time: $result_time as "%m/%d/%Y %H:%M"

# Static events (no time)
demographics:
  gender:
    time: null

Subject ID Configuration

# Global default
subject_id_col: patient_id

# File-specific override
admissions:
  subject_id_col: hadm_id
  admission:
    code: ADMISSION
    # ...

# Hash a string column into an integer subject ID (uses dftly expression)
patients:
  subject_id_expr: hash($MRN)
  demographics:
    code: DEMOGRAPHIC
    time:

Joining Tables

Sometimes subject identifiers are stored in a separate table from the events you wish to extract. You can specify a join within the event configuration so that the necessary columns are merged before extraction.

vitals:
  join:
    input_prefix: stays
    left_on: stay_id
    right_on: stay_id
    columns_from_right:
      - subject_id
  subject_id_col: subject_id
  HR:
    code: HR
    time: $charttime as "%m/%d/%Y %H:%M:%S"
    numeric_value: $HR

Metadata Linking

When your dataset has separate tables with code descriptions or other metadata, use _metadata blocks to link them. Each block names a metadata file prefix and maps output columns to source columns:

lab_results:
  lab:
    code: $test_name
    time: $timestamp
    numeric_value: $result
    _metadata:
      lab_descriptions:           # Matches lab_descriptions.csv in your input dir
        description: description  # Output "description" from source "description" column

The extract_code_metadata stage reads the metadata file, reconstructs the code using the same expression, and joins it to produce metadata/codes.parquet.

Partial matching with _match_on

When the code is composite (e.g., f"{$medication_name}//{$dose}") but your metadata table only has one of the components, use _match_on to join on that component alone. The metadata is broadcast to all codes sharing that component:

medications:
  med:
    code: f"{$medication_name}//{$dose}"
    time: $timestamp
    _metadata:
      medication_classes:
        _match_on: medication_name   # Join on just this code component
        description: drug_class      # "Metformin//500mg" gets "Antidiabetic"

Without _match_on, the metadata table would need both medication_name and dose columns to reconstruct the full code. With _match_on, only the specified column is needed. You can also specify multiple columns: _match_on: [col_a, col_b].

Output Columns

In addition to the standard MEDS columns (subject_id, time, code, numeric_value), MEDS-Extract adds these extension columns to the extracted data:

• code_components: A struct column with the individual source column values that were combined to form the code. For example, if code: f"{$test_name}//{$units}", each row has {test_name: "Glucose", units: "mg/dL"}. Only present when the code expression references source columns (not for literals like code: MEDS_BIRTH).

• source_block: A string column tracking which MESSY config block produced each event, formatted as "{file_prefix}/{event_name}" (e.g., "patients/eye_color", "labs_vitals/lab"). Useful for debugging and filtering events by origin.

The metadata/codes.parquet file also includes:

• code_template: The dftly expression string that produced each code (e.g., $test_name or f"{$medication_name}//{$dose}"). Enables downstream tools to understand code structure without access to the original MESSY config.

🛠️ Troubleshooting

Performance Optimization

• Manually pre-shard your input data if you have very large files. You can then configure your pipeline to skip the row-sharding stage and start directly with the convert_to_subject_sharded stage.
• Use parallel processing for faster extraction via the typical MEDs-Transforms parallelization options.

Future Roadmap

1. Incorporating more of the pre-MEDS and joining logic that is common into this repository.
2. Automatic support for running in "demo mode" for testing and validation.
3. Better examples and documentation for common use cases, including incorporating data cleaning stages after the core extraction.
4. Providing a default runner or multiple default pipeline files for user convenience.

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for more details.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

MEDS Extract builds on the MEDS-Transforms framework and the MEDS standard. Special thanks to:

• The MEDS community for developing the standard
• Contributors to MEDS-Transforms for the underlying infrastructure
• Healthcare institutions sharing their data for research

📖 Citation

If you use MEDS Extract in your research, please cite:

@software{meds_extract2024,
  title={MEDS Extract: ETL Pipelines for Converting EHR Data to MEDS Format},
  author={McDermott, Matthew and contributors},
  year={2024},
  url={https://github.com/mmcdermott/MEDS_extract}
}

---

Ready to standardize your EHR data? Start with our Quick Start guide or explore our example directory for a real, runnable configuration.