MEDS OMOP ETL

An ETL pipeline for transforming Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) datasets into MEDS using MEDS-Extract. We gratefully acknowledge the developers of the first OMOP MEDS ETL, from which we took inspiration, which can be found here: https://github.com/Medical-Event-Data-Standard/meds_etl.

We currently support OMOP 5.3 and 5.4 datasets. Earlier versions might work but are not tested and are perhaps (?) not used in practice anymore. Please open pull requests if you want to add support for earlier versions.

• More information about OMOP can be found here: https://ohdsi.github.io/CommonDataModel/
• More information about MEDS can be found here: https://medical-event-data-standard.github.io/

For reading in OMOP in the right schema, we use the omop-schema package (developed for this ETL but can be used elswehere), which can be found here: https://github.com/rvandewater/omop_schema. If your OMOP schema is non-standard (but still approximately OMOP), you should be able to use the omop-schema package to define your own schema and use it in this ETL.

Setup

Requires Python >=3.12,<3.14 (MEDS-extract 0.6.2 does not support 3.11).

First install the package:

pip install OMOP_MEDS

Then:

export DATASET_NAME="Your_OMOP_Dataset_Name" # e.g. MIMIC_IV_OMOP
export OMOP_VERSION="5.3" # or 5.4
export RAW_INPUT_DIR="path/to/your/input"
export ROOT_OUTPUT_DIR="/path/to/your/output"
OMOP_MEDS raw_input_dir=$RAW_INPUT_DIR root_output_dir=$ROOT_OUTPUT_DIR

To try with the MIMIC-IV OMOP demo dataset (this downloads a version to your local machine), you can run:

OMOP_MEDS raw_input_dir=path/to/your/input root_output_dir=/path/to/your/output do_download=True ++do_demo=True

Example config for an OMOP dataset:

dataset_name: MIMIC_IV_OMOP
raw_dataset_version: 1.0
omop_version: 5.3

urls:
  dataset:
    - https://physionet.org/content/mimic-iv-demo-omop/0.9/
    - url: EXAMPLE_CONTROLLED_URL
      username: ${oc.env:DATASET_DOWNLOAD_USERNAME}
      password: ${oc.env:DATASET_DOWNLOAD_PASSWORD}
  demo:
    - https://physionet.org/content/mimic-iv-demo-omop/0.9/
  common:
    - EXAMPLE_SHARED_URL # Often used for shared metadata files

Run this with:

OMOP_MEDS ++DATASET_CFG=your_config.yaml raw_input_dir=path/to/your/input root_output_dir=/path/to/your/output \
do_download=True

Differences with the original meds_etl_omop

This package is designed as a more flexible and configurable alternative to the original meds_etl_omop package. We make a few important choices that have impact on your downstream training and task definitions:

• We use the mapped concepts by default, which are more standardized across datasets and, for large, health systems can be more clean, especially if you are working with a limited tokenizer on a large dataset. You can still use the source concepts by setting ++prefer_source=True.
• We use more tables than in the original meds_etl_omop package, which can lead to more complete patient histories. Watch for potential information leakage. You can change your table configs in pre_MEDS.yaml and event_configs.yaml
• This package is more resource intensive, please adjust your n_shards and watch your memory usage.

Different package versions

We have different versions of the ETL pipeline, which are designed for different use cases:

pip install OMOP_MEDS[dev] # for development and testing
pip install OMOP_MEDS[tests] # for running tests
pip install OMOP_MEDS[local_parallelism] # for local parallelization with hydra-joblib-launcher
pip install OMOP_MEDS[slurm_parallelism] # for parallelization with slurm using hydra-submitit-launcher
pip install OMOP_MEDS[large_data] # for handling large datasets with polars (rt64), more than 3.2B rows

Run routine tests (without the downloaded e2e test makes it a lot faster, simply remove the -k flag to run all tests):

pytest -q -k 'not test_e2e'

Pre-MEDS settings

The following settings can be used to configure the pre-MEDS steps.

OMOP_MEDS \
	root_output_dir=/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/small_demo \
	raw_input_dir=/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/full_omop \
	do_download=False ++do_overwrite=True ++limit_subjects=50

• root_output_dir: Set the root output directory.
• raw_input_dir: Path to the raw input directory.
• do_download: Set to False to skip downloading the dataset.
• ++do_overwrite: Set to True to overwrite existing files.
• ++limit_subjects: Limit the number of subjects to process.
• ++prefer_source: Set to True to prefer source concepts over mapped concepts.
• ++join_on_visit: Set to True to join person table on having any associated visits.
• ++process_notes: Set to True to .

Pre-meds batching settings. This is relevant if (some of) your input tables are very large, and you want to process them in batches. This can be useful to reduce memory usage, but it also increases the runtime, so use with caution. The batching settings are as follows:

• ++pre_meds_chunked_tables: Tables eligible for batched pre-MEDS processing.
• ++pre_meds_batching_row_threshold: Row count threshold for batching.
• ++pre_meds_batch_mode: auto, per_shard, by_shards, or by_rows.
• ++pre_meds_batch_size_shards: Shards per batch in by_shards mode.
• ++pre_meds_batch_input_rows: Max rows per batch in by_rows mode.

Also check out the main.yaml config file for more default settings and details on how to configure the pre-MEDS steps, which can be found here: src/OMOP_MEDS/configs/main.yaml

MEDS-Extract settings

If you want to convert a large dataset, you can use parallelization with MEDS-transforms (the MEDS-transformation step that takes the longest).

Using local parallelization with the hydra-joblib-launcher package, you can set the number of workers:

pip install hydra-joblib-launcher --upgrade

Then, you can set the number of workers as environment variable:

export N_WORKERS=8

Moreover, you can set the number of subjects per shard to balance the parallelization overhead based on how many subjects you have in your dataset:

export N_SUBJECTS_PER_SHARD=100000

For large datasets, the pipeline keeps high shard_events limits by default:

row_chunksize: 20000000000
infer_schema_length: 999999999

Optional local parallelism is still available via OMOP_MEDS[local_parallelism] and N_WORKERS.

The MIMIC-IV OMOP Dataset

We use the demo dataset for MIMIC-IV in the OMOP format, which is a subset of the MIMIC-IV dataset. This dataset downloaded from Physionet does not include the standard dictionary linking definitions but should otherwise be functional. We also include a small sample of synthetic (LLM-generated) data in the test/demo_resources directory, which can be used for testing and development purposes.

Advanced datetime resolution

To handle datetime resolution in a more flexible way, you can configure the datetime_resolver settings in your config file. This allows you to specify which columns to use for determining the datetime of events, with support for both primary and override columns. The primary purpose of this is to prevent information leakage by using the last edit datetime (of notes) instead of the original note datetime, if available. The following block can be found in the pre_MEDS.yaml config file, and you can adjust the column names based on your dataset's schema:

datetime_resolver:
  primary_datetime_col: note_datetime     # preferred; full timestamp
  primary_date_col: note_date     # fallback; promoted to 23:59:59
  override_datetime_col: xtn_note_last_edit_datetime     # optional XTN column
  override_date_col: xtn_note_last_edit_date     # optional XTN column

The following is an example of how the data then looks like:

In this case it will prefer the xtn_note_last_edit_datetime column for the datetime of the event (if it is in fact later in time), which can help to prevent information leakage in some cases, but it will fall back to note_datetime/note_date if the override column is not available. See build_preferred_event_datetime function in the code for more details on how this works.

Particularities

• Care site is added to the visit as text
• Add support for care_site table (visit_detail)

Citation

If you use this ETL for your research, please use the citation link in Github, which points to the Zenodo DOI, and which is also included below:

@software{van_de_Water_OMOP_MEDS_ETL_2025,
author = {van de Water, Robin Philippus},
doi = {10.5281/zenodo.15132444},
license = {MIT},
month = feb,
title = {{OMOP\_MEDS ETL}},
url = {https://github.com/rvandewater/OMOP_MEDS},
year = {2025}
}