cfdb-ingest

Convert meteorological model output to cfdb with standardized CF conventions

Documentation: https://mullenkamp.github.io/cfdb-ingest/

Source Code: https://github.com/mullenkamp/cfdb-ingest

Overview

cfdb-ingest converts meteorological file formats (netCDF4/HDF5) from various model outputs into cfdb. It standardizes variable names and attributes to be consistent with CF conventions, making it straightforward to work with datasets from different sources through a single interface.

Supported sources:

• WRF -- wrfout NetCDF files (all variables in one file per time range)
• ERA5 -- NCAR ERA5 NetCDF files (one variable per file, surface + pressure level + invariant products)
• IFS -- ECMWF IFS open-data forecast cycles (GRIB2), into cfdb grid_forecast datasets (pip install 'cfdb-ingest[ifs]')

Key features:

• Automatic variable mapping -- source variable names are translated to CF-standard names with proper metadata via cfdb-vars, covering surface, soil, pressure/height-level, potential-temperature, vorticity, surface-flux, and moisture-transport fields
• Named height coordinates -- surface variables at specific heights (0m, 2m, 10m, 100m) get their own named coordinates (e.g. height_2m), allowing them to coexist with pressure-level variables without ambiguity
• Wind rotation (WRF) -- grid-relative wind components are rotated to earth-relative
• Moisture-transport variables -- vertically integrated moisture flux (VIMF) and integrated vapour transport (IVT) for WRF; VIMF for ERA5
• 3D level interpolation (WRF) -- eta-level variables are interpolated to user-specified height or pressure levels
• Native passthrough with computed fallback (WRF) -- derived fields (sea-level pressure, precipitable water, moisture flux) are read directly from newer WRF builds, or reconstructed from 3D fields on older wrfout files
• Auto pressure level detection (ERA5) -- pressure levels are read directly from source files
• Split or combined output (ERA5) -- create one cfdb per variable or combine into a single dataset
• Forecast datasets -- WRF runs and IFS cycles stored as grid_forecast (init × lead), appended one init at a time with every chunk written once
• Incremental inits -- a running forecast ingested one daily file at a time (leads=, mark_complete=False, forecast.missing_chunks), and the S3 archive protocol shared with ifs-download (pip install 'cfdb-ingest[archive]')
• WPS intermediate file export -- convert cfdb datasets (grid, or one init of a grid_forecast) to WPS intermediate format for metgrid.exe
• Spatial and temporal filtering -- subset by bounding box and/or date range
• Multi-file support -- seamlessly spans multiple input files

Performance

cfdb-ingest is designed for high-performance processing of large meteorological datasets:

• Vectorized rechunking -- utilizes rechunkit for optimized HDF5 reads, even when extracting small spatial subsets across many timesteps.
• Parallel initialization -- multi-threaded file scanning and metadata extraction for fast startup.
• HDF5 Chunk Caching -- intelligent management of the HDF5 chunk cache to prevent redundant I/O during per-timestep transformations.
• Synchronized multi-variable rechunking -- synchronized iteration for derived variables (like VIMF) to eliminate redundant reads of shared source variables.

Installation

Requires Python >= 3.10.

pip install cfdb-ingest

Quick Start

WRF

from cfdb_ingest import WrfIngest

wrf = WrfIngest('wrfout_d01_2023-02-12_00:00:00.nc')
wrf.convert(
    cfdb_path='output.cfdb',
    variables=['T2', 'WIND10'],
    start_date='2023-02-12T06:00',
    end_date='2023-02-12T18:00',
)

cfdb-ingest wrf wrfout_d01_*.nc output.cfdb -v T2,WIND10 -s 2023-02-12T06:00 -e 2023-02-12T18:00

For WPS export, use the --preset wps flag:

cfdb-ingest wrf /path/to/wrfout/ output.cfdb --preset wps -s 2023-02-10 -e 2023-02-10_06
cfdb-to-int output.cfdb -s 2023-02-10 -e 2023-02-10_06

IFS forecasts

cfdb-ingest ifs /data/ifs/2026091300/ nz_ifs.cfdb --preset wps --bbox 142,-54,192,-14 --max-lead-hours 144
cfdb-to-int nz_ifs.cfdb --init 2026-09-13T00 -h 3 -p IFS

from cfdb_ingest import IfsIngest
IfsIngest('/data/ifs/2026091300/').convert('nz_ifs.cfdb', bbox=(142, -54, 192, -14), max_lead_hours=144)

ERA5

from cfdb_ingest import Era5Ingest

era5 = Era5Ingest('/path/to/era5/*.nc')
era5.convert(
    cfdb_path='era5.cfdb',
    variables=['SP', 'VAR_2T', 'T', 'U', 'V'],
    start_date='2020-01-01',
    end_date='2020-01-31',
)

# Combined: multiple variables in one cfdb
cfdb-ingest era5 /path/to/era5/*.nc output.cfdb -v SP,VAR_2T,T,U,V -s 2020-01-01 -e 2020-01-31

# Split: one cfdb file per variable
cfdb-ingest era5 /path/to/era5/*.nc /output/dir/ --split -v SP,T

See the full documentation for details.

License

This project is licensed under the terms of the Apache Software License 2.0.