pyESD 1.0.7

Python Package for Empirical Statistical Downscaling. pyESD is under active development and all colaborators are welcomed. The purpose of the package is to downscale any climate variables e.g. precipitation and temperature using predictors from reanalysis datasets (eg. ERA5) to point scale. pyESD adopts many ML and AL as the transfer function.

Python Package for Empirical Statistical Downscaling (v1.01) :sun_behind_rain_cloud: :cloud_with_snow: :cloud_with_rain: :fire: :thermometer:

PyESD is an open-source framework of the Perfect Prognosis approach of statistical downscaling of any climate-related variable such as precipitation, temperature, and wind speed using reanalysis products eg. ERA5 as predictors. The package features all the downscaling cycles including data preprocessing, predictor selection, constructions (eg. using transformers), model selection, training, validation and evaluation, and future prediction. The package serves as the means of downscaling General Circulation Models of future climate to high resolution relevant for climate impact assessment such as droughts, flooding, wildfire risk, and others. The main specialties of the pyESD include:

• Well designed in an OOP style that considers weather stations as individual objects and all the downscaling routines as attributes. This ensures fewer lines of code that cover the end-to-end downscaling of climate change variable products.
• PyESD features many machine learning algorithms and predictor selection techniques that can be experimented with toward the selection and design of robust transfer functions that can be coupled with GCM to generate future estimates of climate change
• And many other functionalities that are highlighted in the paper description of the package (to be submitted).

The main component and the work flow of the package are summarised in the modeling outline:

Installation :hammer_and_wrench:

1. Install the standard version: pip install pyESD from PyPI or git clone [e-mail]:Dan-Boat/PyESD.git cd to the folder | pip install .

2. Install in editable mode: pip install -e pyESD or pip install -e . in the package base folder clone from github The installation might require some dependencies that must be installed if not successful from the distribution from PyPI: cartopy, xarray, sciki-learn, scipy and the other scientific frameworks such as NumPy, pandas, Matplotlib, and seaborn

3. Alternatively, to ensure the installation in an isolated environment, virtual python environment using conda or virtualenv can be used to create a separate env for the package installation

Documentation :blue_book:

The package documentation is still in progress. The initial structure is accessible at github-pages

Examples

The package has been used for downscaling precipitation and temperature for a catchment located in southwestern Germnany. We have also used it for generating future rainfall products for all the synoptic weather stations in Ghana. Their respective control scripts are located in the examples folder. Generally, the control scripts follow the modeling workflow as shown in: . For instance, the downscaling framework show below can be experimented with to select the robust predictor selection method and emprical transfer function for a specific location and predictand variable.

Workflow demonstration: To use the PP-ESD model to downscale climate model, weather station and reanalysis datasets are required. The predictors are loaded in as netCDF files and the predictand as csv file. Let assume that the various predictor variables are stored locally in the era5_datadir directory /home/daniel/ERA5/ and the predictand variable eg. precipitation is stored in station_dir The files should have the same timestamp as the interested predictand variable

1. import all the required modules

from pyESD.Weatherstation import read_station_csv
from pyESD.standardizer import MonthlyStandardizer, StandardScaling
from pyESD.ESD_utils import store_pickle, store_csv
from pyESD.splitter import KFold
from pyESD.ESD_utils import Dataset
from pyESD.Weatherstation import read_weatherstationnames

import pandas as pd 

2. Read the datasets

ERA5Data = Dataset('ERA5', {
    't2m':os.path.join(era5_datadir, 't2m_monthly.nc'),
    'msl':os.path.join(era5_datadir, 'msl_monthly.nc'),
    'u10':os.path.join(era5_datadir, 'u10_monthly.nc'),
    'v10':os.path.join(era5_datadir, 'v10_monthly.nc'),
    'z250':os.path.join(era5_datadir, 'z250_monthly.nc'),

3. define potential predictors and radius of predictor construction, time range for model training and evaluation

radius = 100 #km
predictors = ["t2m", "tp","msl", "v10", "u10"]

from1958to2010 = pd.date_range(start="1958-01-01", end="2010-12-31", freq="MS") #training and validation

from2011to2020 = pd.date_range(start="2011-01-01", end="2020-12-31", freq="MS")  # testing trained data 

4. Read weather stations as objects and apply the downscaling cycle attributes. Note that running the model the first time for a specific location extract the regional means using the define radius and location of the station. The extracted means are stored in a pickel files in the directory called predictor_dir

variable = "Precipitation"
SO = read_station_csv(filename=station_dir, varname=variable)
        
        
# USING ERA5 DATA
# ================


#setting predictors 
SO.set_predictors(variable, predictors, predictordir, radius,)

#setting standardardizer
SO.set_standardizer(variable, standardizer=MonthlyStandardizer(detrending=False,
                                                               scaling=False))

scoring = ["neg_root_mean_squared_error",
            "r2", "neg_mean_absolute_error"]

#setting model
regressor = "RandomForest"
SO.set_model(variable, method=regressor, 
               daterange=from1958to2010,
               predictor_dataset=ERA5Data, 
               cv=KFold(n_splits=10),
               scoring = scoring)


# MODEL TRAINING (1958-2000)
# ==========================


SO.fit(variable, from1958to2010, ERA5Data, fit_predictors=True, predictor_selector=True, 
            selector_method="Recursive" , selector_regressor="ARD",
            cal_relative_importance=False)
   
score_1958to2010, ypred_1958to2010 = SO.cross_validate_and_predict(variable, from1958to2010, ERA5Data)
   
score_2011to2020 = SO.evaluate(variable, from2011to2020, ERA5Data)

ypred_1958to2010 = SO.predict(variable, from1958to2010, ERA5Data)
   
ypred_2011to2020 = SO.predict(variable, from2011to2020, ERA5Data)

ypred_2011to2020.plot()
plt.show()

Package testing

The package is tested using the unittest framework with synthetic generated data. The testing scripts are located in the test folder. Running the various scripts with -v flag (higher level of verbose), would validate the modified version of the package.

Publications

The package description and application paper is currently under preparation (to be submitted to GMD) Its application for weather station in Ghana was presented at the AGU22 Link and the paper is under preparation

Citation: Upload on zenodo: https://doi.org/10.5281/zenodo.7748769

Future versions

The pacakage is still in planing stage (v.0.0.1) (The stable version would be uploaded on pypi with the version number v.1.0.1)

Collaborators are welcomed: interms of model application, model improvement, documentation and expansion of the package!

@ Daniel Boateng (linkedin) : University of Tuebingen :incoming_envelope: [e-mail]

======= History

1.0.7 (2023-06-30)

Bug fixes

* Fix the issues with installation on pypi

New indicators
--------------
* added additional teleconnection indicies for Southern Hemisphere
* Test the framework with daily data but reqiure additional models like wet day classifiers
* missing data in the predictand values can be fill with the implemented impute methods

To do
------
* Extend the package to use spatial datasets
* Implement wet and dry day classifiers before model training
* implement the various calibration metric for daily data 
* Add simple MOS method (quantile mapping)