Skip to main content

cfdb-ingest

Convert meteorological model output to cfdb with standardized CF conventions

build codecov PyPI version


Documentation: https://mullenkamp.github.io/cfdb-ingest/

Source Code: https://github.com/mullenkamp/cfdb-ingest


Overview

cfdb-ingest converts meteorological file formats (netCDF4/HDF5) from various model outputs into cfdb. It standardizes variable names and attributes to be consistent with CF conventions, making it straightforward to work with datasets from different sources through a single interface.

Supported sources:

  • WRF -- wrfout NetCDF files (all variables in one file per time range)
  • ERA5 -- NCAR ERA5 NetCDF files (one variable per file, surface + pressure level + invariant products)
  • IFS -- ECMWF IFS open-data forecast cycles (GRIB2), into cfdb grid_forecast datasets (pip install 'cfdb-ingest[ifs]')

Key features:

  • Automatic variable mapping -- source variable names are translated to CF-standard names with proper metadata via cfdb-vars, covering surface, soil, pressure/height-level, potential-temperature, vorticity, surface-flux, and moisture-transport fields
  • Named height coordinates -- surface variables at specific heights (0m, 2m, 10m, 100m) get their own named coordinates (e.g. height_2m), allowing them to coexist with pressure-level variables without ambiguity
  • Wind rotation (WRF) -- grid-relative wind components are rotated to earth-relative
  • Moisture-transport variables -- vertically integrated moisture flux (VIMF) and integrated vapour transport (IVT) for WRF; VIMF for ERA5
  • 3D level interpolation (WRF) -- eta-level variables are interpolated to user-specified height or pressure levels
  • Native passthrough with computed fallback (WRF) -- derived fields (sea-level pressure, precipitable water, moisture flux) are read directly from newer WRF builds, or reconstructed from 3D fields on older wrfout files
  • Auto pressure level detection (ERA5) -- pressure levels are read directly from source files
  • Split or combined output (ERA5) -- create one cfdb per variable or combine into a single dataset
  • Forecast datasets -- WRF runs and IFS cycles stored as grid_forecast (init × lead), appended one init at a time with every chunk written once
  • Incremental inits -- a running forecast ingested one daily file at a time (leads=, mark_complete=False, forecast.missing_chunks), and the S3 archive protocol shared with ifs-download (pip install 'cfdb-ingest[archive]')
  • WPS intermediate file export -- convert cfdb datasets (grid, or one init of a grid_forecast) to WPS intermediate format for metgrid.exe
  • Spatial and temporal filtering -- subset by bounding box and/or date range
  • Multi-file support -- seamlessly spans multiple input files

Performance

cfdb-ingest is designed for high-performance processing of large meteorological datasets:

  • Vectorized rechunking -- utilizes rechunkit for optimized HDF5 reads, even when extracting small spatial subsets across many timesteps.
  • Parallel initialization -- multi-threaded file scanning and metadata extraction for fast startup.
  • HDF5 Chunk Caching -- intelligent management of the HDF5 chunk cache to prevent redundant I/O during per-timestep transformations.
  • Synchronized multi-variable rechunking -- synchronized iteration for derived variables (like VIMF) to eliminate redundant reads of shared source variables.

Installation

Requires Python >= 3.10.

pip install cfdb-ingest

Quick Start

WRF

from cfdb_ingest import WrfIngest

wrf = WrfIngest('wrfout_d01_2023-02-12_00:00:00.nc')
wrf.convert(
    cfdb_path='output.cfdb',
    variables=['T2', 'WIND10'],
    start_date='2023-02-12T06:00',
    end_date='2023-02-12T18:00',
)
cfdb-ingest wrf wrfout_d01_*.nc output.cfdb -v T2,WIND10 -s 2023-02-12T06:00 -e 2023-02-12T18:00

For WPS export, use the --preset wps flag:

cfdb-ingest wrf /path/to/wrfout/ output.cfdb --preset wps -s 2023-02-10 -e 2023-02-10_06
cfdb-to-int output.cfdb -s 2023-02-10 -e 2023-02-10_06

IFS forecasts

cfdb-ingest ifs /data/ifs/2026091300/ nz_ifs.cfdb --preset wps --bbox 142,-54,192,-14 --max-lead-hours 144
cfdb-to-int nz_ifs.cfdb --init 2026-09-13T00 -h 3 -p IFS
from cfdb_ingest import IfsIngest
IfsIngest('/data/ifs/2026091300/').convert('nz_ifs.cfdb', bbox=(142, -54, 192, -14), max_lead_hours=144)

ERA5

from cfdb_ingest import Era5Ingest

era5 = Era5Ingest('/path/to/era5/*.nc')
era5.convert(
    cfdb_path='era5.cfdb',
    variables=['SP', 'VAR_2T', 'T', 'U', 'V'],
    start_date='2020-01-01',
    end_date='2020-01-31',
)
# Combined: multiple variables in one cfdb
cfdb-ingest era5 /path/to/era5/*.nc output.cfdb -v SP,VAR_2T,T,U,V -s 2020-01-01 -e 2020-01-31

# Split: one cfdb file per variable
cfdb-ingest era5 /path/to/era5/*.nc /output/dir/ --split -v SP,T

See the full documentation for details.

License

This project is licensed under the terms of the Apache Software License 2.0.

Metadata

Release files for cfdb-ingest 0.10.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for cfdb-ingest 0.10.0
File Size Uploaded Provenance Actions
cfdb_ingest-0.10.0.tar.gz 141.8 kB Without provenance Details for cfdb_ingest-0.10.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for cfdb-ingest 0.10.0
File Interpreter ABI Platform Actions
cfdb_ingest-0.10.0-py3-none-any.whl Python 3 none any Details for cfdb_ingest-0.10.0-py3-none-any.whl

Total release size: 292.9 kB

Release files / cfdb_ingest-0.10.0.tar.gz

Download URL cfdb_ingest-0.10.0.tar.gz
Size 141.8 kB
Tags Source
SHA-256 checksum
How to use checksums
6370a4594b7770d934e74f85db980e192456086d447db72b6d05e78934b2097e
BLAKE2b-256 checksum
How to use checksums
57f4f021c53ff9491a5ad7900fc17de8a6fafb666ca4d72ff7cfe593faa6686a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.8.7

Release files / cfdb_ingest-0.10.0-py3-none-any.whl

Download URL cfdb_ingest-0.10.0-py3-none-any.whl
Size 151.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8c6dd4da3791cd845ee2e1583f6ddd3f6611a3f74dfb5c77b667afba1bb7884f
BLAKE2b-256 checksum
How to use checksums
84fc94418de3530bf12a94367fa5231ff476b44f47920d6481cd2ab36c8b7266
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.8.7

Security for cfdb-ingest 0.10.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.10.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page