h2mare 0.4.0

Downloader and data management tools for climate and ocean datasets.

H2MARE - Geospatial Processing for Climate and Ocean Data

A Python pipeline for downloading and preprocessing multi-source oceanographic and atmospheric data into analysis-ready formats. H2MARE streamlines the acquisition and harmonization of data from major climate and ocean observation services, optimized for large-scale spatiotemporal analysis.

Features

• Multi-source data integration: Download and process data from CMEMS, AVISO, and ERA5.
• Format conversion: Automated conversion from NetCDF/GRIB to optimized Zarr and Parquet formats.
• Data compilation: Regrid and interpolate multi-resolution datasets to a common grid.
• Point and geometry extraction: Extract time series for specific locations or spatial features.

Data Sources

H2MARE supports the following data providers. API keys and authentication are required for each.

• CMEMS — Copernicus Marine Service: satellite and in-situ ocean observations
• AVISO — Archiving, Validation and Interpretation of Satellite Oceanographic data
• CDS-ERA5 — ERA5 hourly atmospheric reanalysis (1940–present)

Refer to each provider's documentation for authentication setup before use.

Installation

Prerequisites

• Python >= 3.11
• uv — fast Python package and project manager

Install from PyPI

pip install h2mare
# or
uv add h2mare

Install from source

git clone https://github.com/h2ugoparra/h2mare.git
cd h2mare
uv sync

Configuration

H2MARE requires two configuration files in your working directory before first use.

1. config.yaml

Defines variables, dataset IDs, bounding boxes, and processing parameters. Copy the template from the repository as a starting point and edit it to match your needs.

2. .env

# Path to external or large-capacity storage for processed Zarr files
STORE_ROOT=/path/to/your/storage

# AVISO credentials (required for FSLE, Eddies)
AVISO_USERNAME=your_username
AVISO_PASSWORD=your_password
AVISO_FTP_SERVER=ftp-access.aviso.altimetry.fr

CMEMS credentials are configured via the copernicusmarine client. ERA5 / CDS credentials are configured via the cdsapi client. See the CDS documentation for setup.

Note: Both files must be present in the directory where you run h2mare. You can also set the H2MARE_ROOT environment variable to point to a different directory containing them.

Quick Start

# Download and process a single variable for a specific date range
uv run h2mare run -v sst --start-date 2021-01-01 --end-date 2021-12-31

# Multiple variables at once
uv run h2mare run -v seapodym -v mld -v o2 -v chl

# Infer missing dates from the existing store and download what's new
uv run h2mare run -v sst

# Download only (skip Zarr conversion)
uv run h2mare run -v sst --no-convert

# Validate configuration without downloading
uv run h2mare run -v sst --dry-run

# Process all configured variables
uv run h2mare run

Development

# Run the full test suite
uv run pytest tests/

# Run a single test file
uv run pytest tests/test_zarr_catalog.py -v

# Lint and format
uv run ruff check h2mare/
uv run ruff format h2mare/

Built with

Library	Role
xarray	N-dimensional labelled arrays and NetCDF/Zarr I/O
zarr	Chunked, compressed array storage
dask	Parallel and out-of-core computation
polars	Fast DataFrame engine for Parquet I/O
duckdb	In-process SQL for Parquet overlap resolution and scanning
geopandas	Geometry-based spatial extraction
plotly	Interactive time-series and spatial visualizations
copernicusmarine	CMEMS dataset access
cdsapi	ERA5 / CDS dataset access

Contributing

Contributions are welcome! Please feel free to submit issues or pull requests on GitHub.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

This project was developed under the framework of COSTA project. This project relies on data from Copernicus Marine Service, AVISO, Copernicus Climate Data Store, and NOAA NCEI. We gratefully acknowledge these organizations for providing open access to their datasets.