cfdb-ingest
Convert meteorological model output to cfdb with standardized CF conventions
Documentation: https://mullenkamp.github.io/cfdb-ingest/
Source Code: https://github.com/mullenkamp/cfdb-ingest
Overview
cfdb-ingest converts meteorological file formats (netCDF4/HDF5) from various model outputs into cfdb. It standardizes variable names and attributes to be consistent with CF conventions, making it straightforward to work with datasets from different sources through a single interface.
Supported sources:
- WRF -- wrfout NetCDF files (all variables in one file per time range)
- ERA5 -- NCAR ERA5 NetCDF files (one variable per file, surface + pressure level + invariant products)
- IFS -- ECMWF IFS open-data forecast cycles (GRIB2), into cfdb
grid_forecastdatasets (pip install 'cfdb-ingest[ifs]')
Key features:
- Automatic variable mapping -- source variable names are translated to CF-standard names with proper metadata via cfdb-vars, covering surface, soil, pressure/height-level, potential-temperature, vorticity, surface-flux, and moisture-transport fields
- Named height coordinates -- surface variables at specific heights (0m, 2m, 10m, 100m) get their own named coordinates (e.g.
height_2m), allowing them to coexist with pressure-level variables without ambiguity - Wind rotation (WRF) -- grid-relative wind components are rotated to earth-relative
- Moisture-transport variables -- vertically integrated moisture flux (VIMF) and integrated vapour transport (IVT) for WRF; VIMF for ERA5
- 3D level interpolation (WRF) -- eta-level variables are interpolated to user-specified height or pressure levels
- Native passthrough with computed fallback (WRF) -- derived fields (sea-level pressure, precipitable water, moisture flux) are read directly from newer WRF builds, or reconstructed from 3D fields on older
wrfoutfiles - Auto pressure level detection (ERA5) -- pressure levels are read directly from source files
- Split or combined output (ERA5) -- create one cfdb per variable or combine into a single dataset
- Forecast datasets -- WRF runs and IFS cycles stored as
grid_forecast(init × lead), appended one init at a time with every chunk written once - Incremental inits -- a running forecast ingested one daily file at a time (
leads=,mark_complete=False,forecast.missing_chunks), and the S3 archive protocol shared withifs-download(pip install 'cfdb-ingest[archive]') - WPS intermediate file export -- convert cfdb datasets (
grid, or one init of agrid_forecast) to WPS intermediate format for metgrid.exe - Spatial and temporal filtering -- subset by bounding box and/or date range
- Multi-file support -- seamlessly spans multiple input files
Performance
cfdb-ingest is designed for high-performance processing of large meteorological datasets:
- Vectorized rechunking -- utilizes rechunkit for optimized HDF5 reads, even when extracting small spatial subsets across many timesteps.
- Parallel initialization -- multi-threaded file scanning and metadata extraction for fast startup.
- HDF5 Chunk Caching -- intelligent management of the HDF5 chunk cache to prevent redundant I/O during per-timestep transformations.
- Synchronized multi-variable rechunking -- synchronized iteration for derived variables (like VIMF) to eliminate redundant reads of shared source variables.
Installation
Requires Python >= 3.10.
pip install cfdb-ingest
Quick Start
WRF
from cfdb_ingest import WrfIngest
wrf = WrfIngest('wrfout_d01_2023-02-12_00:00:00.nc')
wrf.convert(
cfdb_path='output.cfdb',
variables=['T2', 'WIND10'],
start_date='2023-02-12T06:00',
end_date='2023-02-12T18:00',
)
cfdb-ingest wrf wrfout_d01_*.nc output.cfdb -v T2,WIND10 -s 2023-02-12T06:00 -e 2023-02-12T18:00
For WPS export, use the --preset wps flag:
cfdb-ingest wrf /path/to/wrfout/ output.cfdb --preset wps -s 2023-02-10 -e 2023-02-10_06
cfdb-to-int output.cfdb -s 2023-02-10 -e 2023-02-10_06
IFS forecasts
cfdb-ingest ifs /data/ifs/2026091300/ nz_ifs.cfdb --preset wps --bbox 142,-54,192,-14 --max-lead-hours 144
cfdb-to-int nz_ifs.cfdb --init 2026-09-13T00 -h 3 -p IFS
from cfdb_ingest import IfsIngest
IfsIngest('/data/ifs/2026091300/').convert('nz_ifs.cfdb', bbox=(142, -54, 192, -14), max_lead_hours=144)
ERA5
from cfdb_ingest import Era5Ingest
era5 = Era5Ingest('/path/to/era5/*.nc')
era5.convert(
cfdb_path='era5.cfdb',
variables=['SP', 'VAR_2T', 'T', 'U', 'V'],
start_date='2020-01-01',
end_date='2020-01-31',
)
# Combined: multiple variables in one cfdb
cfdb-ingest era5 /path/to/era5/*.nc output.cfdb -v SP,VAR_2T,T,U,V -s 2020-01-01 -e 2020-01-31
# Split: one cfdb file per variable
cfdb-ingest era5 /path/to/era5/*.nc /output/dir/ --split -v SP,T
See the full documentation for details.
License
This project is licensed under the terms of the Apache Software License 2.0.
Metadata
Release files for cfdb-ingest 0.8.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| cfdb_ingest-0.8.1.tar.gz | 118.5 kB | Without provenance | Details for cfdb_ingest-0.8.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| cfdb_ingest-0.8.1-py3-none-any.whl | Python 3 | none | any | Details for cfdb_ingest-0.8.1-py3-none-any.whl |
Total release size: 243.7 kB
Release files / cfdb_ingest-0.8.1.tar.gz
| Download URL | cfdb_ingest-0.8.1.tar.gz |
|---|---|
| Size | 118.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c5a1b67f2672e416f19a27dccbcd034697a933f57302098dea7c4dde6aec8cac
|
|
BLAKE2b-256 checksum How to use checksums |
479001ecc1009846787a3f445b0f55d439303ab19f4ad2c95b1bb61bac7a693b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.8.7
|
Release files / cfdb_ingest-0.8.1-py3-none-any.whl
| Download URL | cfdb_ingest-0.8.1-py3-none-any.whl |
|---|---|
| Size | 125.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8647d225acd1d83ac55ad7d875d0beef4b6dc27179eaef50cf4916dfcb6d43f8
|
|
BLAKE2b-256 checksum How to use checksums |
dc45476da4b8ce0504adb54ca0692e525b22ef07a68c3b256f491754efcd396b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.8.7
|
Security for cfdb-ingest 0.8.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.