earthkit-hydro 0.1.4

A Python library for common hydrological functions

Installation • Documentation

[!IMPORTANT] This software is Emerging and subject to ECMWF's guidelines on Software Maturity.

earthkit-hydro is a Python library for common hydrological functions. It is the hydrological component of earthkit.

Main Features

• Support for PCRaster, CaMa-Flood and HydroSHEDS river networks
• Computing statistics over catchments and subcatchments
• Finding catchments and subcatchments
• Calculating distances along river networks
• Calculation of upstream or downstream fields
• Handle arbitrary missing values
• Handle N-dimensional fields

Installation

For a default installation, run

pip install earthkit-hydro

For a developer setup (includes linting and test libraries), run

conda create -n hydro python=3.12
conda activate hydro
conda install -c conda-forge rust
git clone https://github.com/ecmwf/earthkit-hydro.git
cd earthkit-hydro
pip install -e .[dev]
pre-commit install

Documentation

An example notebook showing how to use the earthkit-hydro is provided in addition to the documentation below.

earthkit-hydro can be imported as following:

import earthkit.hydro as ekh

The package contains different ways of constructing or loading a RiverNetwork object. A RiverNetwork object is a representation of a river network on a grid. It can be used to compute basic hydrological functions, such as propagating a scalar field along the river network or extracting a catchment from the river network.

Mathematical Details

Given a discretisation of a domain i.e. a set of points $\mathcal{D}={ (x_i, y_i)}_{i=1}^N$, a river network is a directed acyclic graph $\mathcal{R}=(V,E)$ where the vertices $V \subseteq \mathcal{D}$. The out-degree of each vertex is at most 1 i.e. each point in the river network points to at most one downstream location.

For ease of notation, if an edge exists from $(x_i, y_i)$ to $(x_j, y_j)$, we write $i \rightarrow j$.

Readers

ekh.river_network.load(domain, version)

Loads a precomputed RiverNetwork. Current options can be listed with ekh.river_network.available() and are:

domain	version	Details	Note	Attribution
"efas"	"5"	1arcmin European		1
"efas"	"4"	5km European	Smaller domain than v5	1
"glofas"	"4"	3arcmin global	60° South to 90° North	2
"glofas"	"3"	6arcmin global	60° South to 90° North	2
"cama_03min"	"4"	3arcmin global		3
"cama_05min"	"4"	5arcmin global		3
"cama_06min"	"4"	6arcmin global		3
"cama_15min"	"4"	15arcmin global		3
"hydrosheds_05min"	"1"	5arcmin global	56° South to 84° North	4
"hydrosheds_06min"	"1"	6arcmin global	56° South to 84° North	4

ekh.river_network.create(path, river_network_format, source="file")

Creates a RiverNetwork. Current options are

• river_network_format: "esri_d8", "pcr_d8", , "merit_d8", "cama" or "precomputed"
• source: An earthkit-data compatable source. See list.

Computing Metrics Over River Networks

There are four high-level ways to compute metrics depending on the use-case.

Metrics Over Upstream Nodes

ekh.upstream.sum(river_network, field, weights=None)
ekh.upstream.max(river_network, field, weights=None)
ekh.upstream.min(river_network, field, weights=None)
ekh.upstream.mean(river_network, field, weights=None)
ekh.upstream.prod(river_network, field, weights=None)
ekh.upstream.std(river_network, field, weights=None)
ekh.upstream.var(river_network, field, weights=None)

Given an input field, returns as output a new field with the upstream metric calculated for each cell.

Metrics Over Downstream Nodes

ekh.downstream.sum(river_network, field, weights=None)
ekh.downstream.max(river_network, field, weights=None)
ekh.downstream.min(river_network, field, weights=None)
ekh.downstream.mean(river_network, field, weights=None)
ekh.downstream.prod(river_network, field, weights=None)
ekh.downstream.std(river_network, field, weights=None)
ekh.downstream.var(river_network, field, weights=None)

Given an input field, returns as output a new field with the downstream metric calculated for each cell.

Metrics Over Catchments

ekh.catchments.sum(river_network, field, points, weights=None)
ekh.catchments.max(river_network, field, points, weights=None)
ekh.catchments.min(river_network, field, points, weights=None)
ekh.catchments.mean(river_network, field, points, weights=None)
ekh.catchments.prod(river_network, field, points, weights=None)
ekh.catchments.std(river_network, field, points, weights=None)
ekh.catchments.var(river_network, field, points, weights=None)

Given a field and a series of points, calculates the metric over all upstream nodes for each of the points.

Metrics Over Subcatchments

ekh.subcatchments.sum(river_network, field, points, weights=None)
ekh.subcatchments.max(river_network, field, points, weights=None)
ekh.subcatchments.min(river_network, field, points, weights=None)
ekh.subcatchments.mean(river_network, field, points, weights=None)
ekh.subcatchments.prod(river_network, field, points, weights=None)
ekh.subcatchments.std(river_network, field, points, weights=None)
ekh.subcatchments.var(river_network, field, points, weights=None)

Given a field and a series of points, finds the subcatchments defined by the points and computes the metric for each subcatchment.

Metrics Over Arbitrary Zones

ekh.zonal.sum(field, labels, weights=None, return_field=False)
ekh.zonal.max(field, labels, weights=None, return_field=False)
ekh.zonal.min(field, labels, weights=None, return_field=False)
ekh.zonal.mean(field, labels, weights=None, return_field=False)
ekh.zonal.prod(field, labels, weights=None, return_field=False)
ekh.zonal.std(field, labels, weights=None, return_field=False)
ekh.zonal.var(field, labels, weights=None, return_field=False)

Calculates a metric over the input field for each zone defined by the labels field. If return_field is True, returns a field otherwise returns a dictionary of {label: metric} pairs.

(for advanced users) Similarly, one can also use a low-level API via

ekh.calculate_upstream_metric(river_network, field, metric, weights=None)
ekh.calculate_catchment_metric(river_network, field, points, metric, weights=None)
ekh.calculate_subcatchment_metric(river_network, field, points, metric, weights=None)
ekh.calculate_zonal_metric(field, labels, metric, weights=None)

# applies the ufunc on the field starting from the sources all the way down to the sinks
ekh.flow_downstream(river_network, field, ufunc=np.add)
# applies the ufunc on the field starting from the sinks all the way up to the sources
ekh.flow_upstream(river_network, field, ufunc=np.add)

These are analagous to above.

Finding Catchments and Subcatchments

ekh.catchments.find(river_network, field)

Finds the catchments (all upstream nodes of specified nodes, with overwriting).
$$v_i^{\prime} = v_j^{\prime} ~ \text{if} ~ v_j^{\prime} \neq 0 ~ \text{else} ~ v_i, ~j ~ \text{s.t.} ~ i \rightarrow j$$

ekh.subcatchments.find(river_network, field)

Finds the subcatchments (all upstream nodes of specified nodes, without overwriting).
$$v_i^{\prime} = v_j^{\prime} ~ \text{if} ~ (v_j^{\prime} \neq 0 ~ \text{and} ~ v_j = 0) ~ \text{else} ~ v_i, ~j ~ \text{s.t.} ~ i \rightarrow j$$

Calculating Distances or Lengths

ekh.distance.min(river_network, points, weights=None, upstream=False, downstream=True)
ekh.distance.max(river_network, points, weights=None, upstream=False, downstream=True)

Given a set of input points, computes the min or max distance (upstream and/or downstream depending on input) from those points for all cells in the field. Unreachable points are given a value np.inf. Weights represents the distance to a downstream cell (weights at sinks are ignored). By default, it is assumed the distance to any downstream cell is 1 regardless if the connection is diagonal or not.

ekh.length.min(river_network, points, weights=None, upstream=False, downstream=True)
ekh.length.max(river_network, points, weights=None, upstream=False, downstream=True)

Given a set of input points, computes the min or max length (upstream and/or downstream depending on input) of the network starting from those points for all cells in the field. Unreachable points are given a value np.inf. Weights represents the length of the river in the grid cell. By default, this is assumed to be one.

We also provide some analagous convenience functions for calculating shortest/longest distances/lengths to sources or sinks.

ekh.distance.to_source(river_network, weights, path='shortest')
ekh.distance.to_sink(river_network, weights, path='shortest')

ekh.length.to_source(river_network, weights, path='shortest')
ekh.length.to_sink(river_network, weights, path='shortest')

Calculating Upstream or Downstream Fields

ekh.move_downstream(river_network, field, ufunc=np.add)

Updates each node with the sum of its upstream nodes.
$$v_i^{\prime}=\sum_{j \rightarrow i}~v_j$$

ekh.move_upstream(river_network, field)

Updates each node with its downstream node.
$$v_i^{\prime} = v_j, ~j ~ \text{s.t.} ~ i \rightarrow j$$

Exporting or Masking a River Network

river_network.create_subnetwork(field)

Computes the river subnetwork defined by a field mask of the domain.

river_network.export(filename)

Exports the RiverNetwork as a joblib pickle.

Migrating from PCRaster

earthkit-hydro provides many functions with PCRaster equivalents, summarised below:

PCRaster	earthkit-hydro	Note
accuflux	upstream.sum
catchmenttotal	upstream.sum
areatotal	zonal.sum	return_field=True
areaaverage	zonal.mean	return_field=True
areamaximum	zonal.max	return_field=True
areaminimum	zonal.min	return_field=True
downstream	move_upstream
upstream	move_downstream
catchment	catchments.find
subcatchment	subcatchments.find
path	upstream.max
ldddist	distance.min	friction input is slightly different to weights, and by default ekh takes distance between two nodes to be one regardless if on diagonal or not
downstreamdist	distance.to_sink	Same caveats as for ldddist
slopelength	distance.to_source	path="longest", same caveats as for ldddist
abs, sin, cos, tan, ...	np.abs, np.sin, np.cos, np.tan, ...	any numpy operations can be directly used

Points of difference

• earthkit-hydro treats missing values as np.nans i.e. any arithmetic involving a missing value will return a missing value. PCRaster does not always handle missing values exactly the same.
• earthkit-hydro can handle vector fields and fields of integers, floats, bools. PCRaster supports a restricted subset of this.

Attributions

¹ The EFAS river network is available under the conditions set out in the European Commission Reuse and Copyright Notice and is available at https://data.jrc.ec.europa.eu/dataset/f572c443-7466-4adf-87aa-c0847a169f23.

Choulga, Margarita; Moschini, Francesca; Mazzetti, Cinzia; Grimaldi, Stefania; Disperati, Juliana; Beck, Hylke; Salamon, Peter; Prudhomme, Christel (2023): LISFLOOD static and parameter maps for Europe. European Commission, Joint Research Centre (JRC) [Dataset] PID: http://data.europa.eu/89h/f572c443-7466-4adf-87aa-c0847a169f23

² The GloFAS river network is available under the conditions set out in the European Commission Reuse and Copyright Notice and is available at https://data.jrc.ec.europa.eu/dataset/68050d73-9c06-499c-a441-dc5053cb0c86.

Choulga, Margarita; Moschini, Francesca; Mazzetti, Cinzia; Disperati, Juliana; Grimaldi, Stefania; Beck, Hylke; Salamon, Peter; Prudhomme, Christel (2023): LISFLOOD static and parameter maps for GloFAS. European Commission, Joint Research Centre (JRC) [Dataset] PID: http://data.europa.eu/89h/68050d73-9c06-499c-a441-dc5053cb0c86

³ The CaMa river networks are available under CC-BY 4.0 Licence and are available at http://hydro.iis.u-tokyo.ac.jp/~yamadai/cama-flood/.

Yamazaki, Dai; Ikeshima, Daiki; Sosa, Jeison; Bates, Paul D.; Allen, George H.; Pavelsky, Tamlin M. (2019): MERIT Hydro: A high-resolution global hydrography map based on latest topography datasets. Water Resources Research, vol.55, pp.5053-5073, 2019, DOI: 10.1029/2019WR024873

⁴ The HydroSHEDS river networks are available under the conditions set out in the HydroSHEDS Version One Licence Agreement and are available at https://www.hydrosheds.org.

Lehner, Bernhard; Verdin, Kristine; Jarvis, Andy (2008): New global hydrography derived from spaceborne elevation data. Eos, Transactions, 89(10): 93-94. Data available at https://www.hydrosheds.org.

Licence

Copyright 2024, European Centre for Medium Range Weather Forecasts.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

In applying this licence, ECMWF does not waive the privileges and immunities
granted to it by virtue of its status as an intergovernmental organisation
nor does it submit to any jurisdiction.