netcdf2region 0.1.8

Aggregate NetCDF data to regions using spatial weights

netcdf2region

netcdf2region is a Python package providing command-line tools for aggregating gridded NetCDF files over regions defined by shapefiles. It is designed to facilitate spatial analysis and extraction of regional statistics from large climate or geospatial datasets.

Features

• Aggregate gridded NetCDF data by custom regions (e.g., administrative boundaries, ecozones).
• Support for shapefile-based region definitions.
• Efficient processing of large datasets.
• Command-line interface for easy integration into workflows.

Installation

pip install netcdf2region

Disclaimer

This package is still in the early stages of development. Please check that all outputs are valid, as errors during calculation may still occur.

Usage

1. Compute Grid Weights

The compute_grid_weights command estimates the overlap between NetCDF grid cells and regions defined in a shapefile via point sampling. This step is performed separately to enable efficient repeated aggregations. The package also provides precomputed weights for common grids, such as ERA5, and shapefiles, such as GADM 3.6. See the section 'Precomputed weights' for more information.

compute_grid_weights <netcdf> <shapefile> <output> [options]

Positional Arguments

• <netcdf>: Path to the NetCDF file with latitude/longitude grid.
• <shapefile>: Path to the shapefile defining regions (e.g., .shp).
• <output>: Path to the output CSV file for computed grid weights.

Options

• --points_per_cell <int>: Number of random points per grid cell (default: 100).
• --point_distribution <uniform|random>: Distribution of points within grid cells (default: uniform)
• --simplify: Simplify region geometries for faster computation.
• --land_sea_mask <file>: Path to a land-sea mask NetCDF file for faster computation (optional).
• --shapefile_layer <int>: Layer number in the shapefile to use (default: first layer).
• --backend <str>: Parallel backend to use (threading by default).
• --n_jobs <int>: Number of parallel jobs (default: 8).
• --coordinate_origin <str>: Coordinate origin of the grid (default: centered).
• --gid <str>: Geo identifier of the shapefile (default: GID_1)
• --shift_longitudes: Shift longitudes from [0, 360] to [-180, 180]
• --yes: Automatically answer yes to prompts.

See --help for more details.

2. Aggregate NetCDF Data

The netcdf2region command aggregates gridded NetCDF data over regions using precomputed grid weights, which are estimated via the compute_grid_weights command. The package also provides precomputed weights for common grids, such as ERA5, and shapefiles, such as GADM 3.6. These can be used directly with the --precalc_weights option (see the section 'Precomputed weights').

netcdf2region \
    --netcdf <input.nc> \
    (--weights <weights.csv> | --precalc_weights <precalc_weights_name>) \
    --method <mean|sum|std> \
    --output <output.csv> \
    --var <variable_name> \
    [--correct_weights] \
    [--time_var <time_variable>] \
    [--secondary_weights_var <secondary_weight_variable>] \
    [--secondary_weights_netcdf <secondary_weights.nc>]

Arguments

• --netcdf: Path to the input NetCDF file.
• --weights: Path to the CSV file with grid weights (lat,lon,region_id,percent_overlap). Mutually exclusive with --precalc_weights.
• --precalc_weights: Name of a predefined weights file (e.g., ERA5_to_GADM36_admin1_weights.csv). Mutually exclusive with --weights.
• --method: Aggregation method (mean, sum, or std).
• --output: Path to the output CSV file.
• --var: Name of the variable in the NetCDF file to aggregate.
• --correct_weights: (Optional) If set, corrects weights to ensure no values are attributed to the sea.
• --time_var: (Optional) Name of the time variable in the NetCDF file (if not automatically detected).
• --secondary_weights_var: (Optional) Name of a secondary weight variable in the secondary NetCDF file. Must have dimensions (lat, lon). Values are multiplied with percent_overlap from the weights CSV during aggregation. NaNs are set to 1.
• --secondary_weights_netcdf: Path to the NetCDF file containing --secondary_weights_var. Required when --secondary_weights_var is provided.

See the command-line help (--help) for more options and details.

Precomputed weights

The package provides precomputed weights for standard grids and shapefiles that can be used immediately with the netcdf2region command.

Grid	Shapefile	Weights file
ISIMIP 0.5°	GADM 3.6 Admin 0	ISIMIP_to_GADM36_admin0_weights.csv
ISIMIP 0.5°	GADM 3.6 Admin 1	ISIMIP_to_GADM36_admin1_weights.csv
ISIMIP 0.5°	DOSE V2 Admin 1	ISIMIP_to_DOSE_admin1_weights.csv
ERA5 0.25°	GADM 3.6 Admin 0	ERA5_to_GADM36_admin0_weights.csv
ERA5 0.25°	GADM 3.6 Admin 1	ERA5_to_GADM36_admin1_weights.csv
ERA5 0.25°	DOSE Admin 1	ERA5_to_DOSE_admin1_weights.csv

Example: Using Precomputed Weights

To aggregate NetCDF data using precomputed weights, specify the --precalc_weights option with the name of the weights file. For example, to aggregate ERA5 data over GADM 3.6 Admin 1 regions:

netcdf2region \
    --netcdf ERA5_sample.nc \
    --precalc_weights ERA5_to_GADM36_admin1_weights.csv \
    --method mean \
    --output ERA5_admin1_mean.csv \
    --var temperature

This command computes the mean of the temperature variable for each region defined in the GADM 3.6 Admin 1 shapefile using the precomputed weights.

Example of Executing Weight Calculations on a HPC with Slurm Workload Manager

This example assumes that the Python environment 'netcdf2region_env' is available, with the package installed:

conda create -n netcdf2region_env -c anaconda
pip install /path/to/netcdf2region

SLURM script to be executed with sbatch:

#!/bin/bash 
#SBATCH --qos=priority
#SBATCH --account=<account_name>
#SBATCH --nodes=1 
#SBATCH --ntasks-per-node=1 
#SBATCH --cpus-per-task=16
#SBATCH --mem=60000

# Important to be able to find netcdf2region
export PATH="$HOME/.local/bin:$PATH"

# Setup python environment
module purge
module load anaconda
source activate netcdf2region_env

# Setup paths
NETCDF_GRID="ERA5_sample.nc"
SHAPEFILE="gadm36_levels.gpkg"
SHAPEFILE_LAYER=1
WEIGHTS_OUTPUT="ERA5_to_GADM36_admin1_weights.csv"

# Settings for parallel computation
NUMBER_OF_JOBS=16 # Number of parallel jobs set to the number of CPU cores

# Run the Python script
compute_grid_weights $NETCDF_GRID $SHAPEFILE $WEIGHTS_OUTPUT --yes --shapefile_layer=$SHAPEFILE_LAYER --n_jobs=$NUMBER_OF_JOBS

Requirements

• Python 3.7+
• numpy
• pandas
• xarray
• rasterio
• geopandas
• matplotlib
• shapely
• joblib
• scipy
• tqdm
• netCDF4

Contributing

Contributions are welcome! Please open issues or pull requests.