mET-weap 1.1

ERA5/MODIS Preprocessor for WEAP 21

mET_weap

Trough the ANR Amethyst's project, a python package has been developed to process MODIS-NDVI and Meteo ERA5 long time series to produce the input data for WEAP 21.

Documentation

• Read the documentation "mET_weap Installation Procedure & User guide"

Quick Installation of mET weap

For more informations, refer to the documentation "mET_weap Installation Procedure & User guide".

Requirements

1. Download mET_weap on Github
2. Copy/paste the test file “mET_weap_test” on your computer
3. Subscription to earth data
4. Subscription to the CDS (Climate Data Store)
5. OSGeo4w installation
6. Create an environment variable for GDAL (Windows)
7. Download required files
8. Installation of Anaconda

Installation

a) Users installation from local file:

For example, if the source file of the setup.py = "C:\Users\VI\Documents\mET_weap". In the anaconda prompt, execute:

- cd "C:\Users\VI\Documents\mET_weap"

Then, execute:

- python setup.py sdist bdist_wheel
- cd .\dist
- pip install mET_weap-0.1.tar.gz

Finally, because the installation of gdal doesn't work with install_requires in the setup.py, execute:

- conda install gdal

b) Users installation from the PyPI

In the anaconda prompt, execute:

- pip install mET_weap

Test folder

Recover the “mET_weap_test” folder in the package downloaded on Pypi or Github. This folder can be also recovered in the “sites-packages” folder ("C:\Users\VI\Anaconda3\Lib\site packages\mET_weap" for instance) if the installation is made directly from the Pypi. To begin, copy/paste this test folder “mET_weap_test” on your desktop. The program will be executed from this “work” directory.

Configuration

The configuration is made with the userconfig.py file. This is the only file that the user has to fill in to make his project works.

First, the path root and the path to osgeo has to be defined :

mET_weap.path_root = os.path.join('C:'+os.sep+'HAOUZ'+os.sep+'mET_weap') # The root directory for output files
mET_weap.path_to_osgeo = os.path.join('C:'+os.sep+'OSGeo4W64'+os.sep+'bin') # the path for gdal binaries

For instance, in the example above, path_root = 'C:\HAOUZ\mET_weap' and path_to_osgeo = 'C:\OSGeo4W64\bin'. The path root creates the directory for the outputs.

The next part of the config file is the switchs for the modules. Before to execute the usermain, fill in "True" to activate and "False" to deactivate a module. For instance, on the example below, 'do_get_modis' is the only module activated.

# ---- Switchs to activate or deactivate the modules ----

# Each Modis tile covers an area of 5°x5°, it may not be necessary to download it again to process another area.

'do_get_modis'           : True,         # download modis files

# If you already downloaded ERA5 files for your area and domain, don't do it again.

'do_download_ERA5_cds'   : False,        # download weather dataset from CDS ERA5

# The following toggles are processing steps,
# Toggle the steps that already went ok to False

'do_project_subset'      : False,        # projection and subset of the modis files
'do_classification'      : False,        # yearly classification
'do_kc_fc'               : False,        # compute Kc/Fc
'do_ET0_rain_ERA5'       : False,        # compute daily ET0 and rainfall from ERA5
'do_interp_ndvi_ERA5'    : False,        # temporal interpolation of NDVI and KC*ET0 with ERA5 data
'do_synthesis_ERA5'      : False,        # spatio-temporal synthesis with ERA5 data

After the login to the server CDS API. Accept the license copernicus product. Then, click on your name near the logout button to retrieve your API key on your user profile. Use the format "UID:API key" for your CDSAPI key.

# ----- The CDSAPI key to download ERA5 data -----
'CDSAPI':{
	'url':"https://cds.climate.copernicus.eu/api/v2",
	'key':"xxxx:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
	},

Next, fill in your MODIS username and password in the the userconfig.py file.

# ---- The MODIS login to download MODIS data ----
'MODIS':{
	'username':"xxxxxxxx",
	'password':"xxxxxxxx"
	},

Then, the spatial domain has to be defined. Fill in the geographic coordinates of your studied area in the userconfig.py.

# ----- the spatial domain to be processed -----
'domain' : {
	"uly":31.7,          # Upper left Y in dd (decimal degrees)
	"ulx":-8.6,          # Upper left X in dd
	'lry':30.85,         # Lower right Y in dd
	'lrx':-7             # Lower Right X in dd
	},

After, the scales are defined. Normally, the user doesn't need to change these values.

'scale' : {
	"echelle" : 0.25,
	"resolution_ERA5": 0.25            
	},

Choose your temporal domain. For instance, from September 2015 to September 2016 in the example above.

# -----  the temporal domain to be processed -----
'period'  : {
	'begin': datetime(2015, 9, 1),       # the date of beginning as a datetime (yyyy,m,j)
	'end': datetime(2016, 8, 30)         # the date of ending as a datetime (yyyy,m,j)
	},

Define the agronomic year:

# -----  definition of the agronomic year with the beginning month and ending month -----
'agroyear'  : {
	'begin': 9,           # generally the agronomic year begins in september (9)
	'end': 8              # generally the agronomic year begins in august of the next year (8)
	},

The next part allows to choose the functions use to compute the Kc and Fc factors.

# -----  Kc and Fc to NDVI relationships -----
'Relation_NDVI': {
	'scale_factor' : 0.0001,                                            # A scale factor to apply to NDVI data, typically 0.001 for MODIS
	'Ndvi_KcKcb' : os.path.join('Tab_ndvi_kc_fc', 'ndvi_kckcb.xlsx'),   # the XLS file for relations NDVI->Kc (Crop Coefficient)
	'Ndvi_Fc' : os.path.join('Tab_ndvi_kc_fc', 'NDVI-FC.xlsx'),         # the XLS file for relations NDVI->Fc (Fraction Cover)
	'do_senescence' : True,                                             # Optional: To compute or not the fraction cover plateau for annual crops. The computation is time consuming
	'do_stack':False,                                                   # Optional: To output or not annual stacks. Those stacks may be useful for data vizualisation

	# rel OS is the table of relations between NDVI/Kcb and NDVI/Fraction covers
	# the syntax is: 'class_name' : [output_value, line of xls file for Kcb, line of xls file for Fc]
	# None if Kc is not computed for a particular class

	'rel_OS' : {
		'No_data' : [0, None, None],
		'Advent'  : [2, None, None],
		'Arb_Veg' : [3,    6,    5],
		'Arb_SN'  : [4,    0,    5],
		'C_Anu'   : [5,    6,    0],
		'Sol_Nu'  : [8, None, None]
		}
	},

This module allows to configure the link between the shapefile of the studied area and the synthesis module. This shapefile has to be in the mET_weap directory. For his project, the user need to fill in the name of his shapefile in ['Synth']['shapefile']. In the test file, on the example above, 'shapefile' : 'zonage\Atlas_Piemont_Haouz.shp'. In 'shp_attrib', fill in the column name of the attribute table. In the example 'c_zone_irr'. Finally, define the 'timerange' and the 'mode'.

# -----  Synthesis -----
'Synth': {
	'shapefile' : os.path.join('zonage'+os.sep+'Atlas_Piemont_Haouz.shp'),
	'shp_attrib' : 'c_zone_irr',    # column name of the attribute table
	'timerange' : "MONTH",		# an accumulation time which is either DAY, WEEK or MONTH
	'mode' : "SUM",			# MEAN or SUM
},

Normally, the paths don't need to be changed because the directory Gtopo30 is located in the mET_weap_test directory. 'geopotential_cdsapi.nc' is downloaded in 'C:\HAOUZ'

# -----  The path for the DEM input file -----
'DEM': {
	'Description':'GTOPO30',
	'DEM_Filename': os.path.join('Gtopo30', 'gt30w020n40.tif'),
	'ERA_Filename_CDSAPI': os.path.join(''.join(os.path.split(mET_weap.path_root)[0]), 'geopotential_cdsapi.nc')
},

These paths define the outputs directory. They are all base on the path_root. Normally, the user doesn't need to change them.

# ----- The paths of the output files -----
'path': {
	'Temp_ERA5': os.path.join(''.join(mET_weap.path_root), "fishnet_era5"),                  # directory for temporary data

	'Data' : os.path.join(''.join(mET_weap.path_root), "data"),                              # the root directory to download data
	'MOD13Q1': os.path.join(''.join(mET_weap.path_root), "data"+os.sep+ "MOD13Q1"),          # the directory for downloading MODIS MOD13Q1
	'ERA5': os.path.join(''.join(mET_weap.path_root), os.path.join("data", "ERA5")),         # the directory to download ERA5 data

	'Tif' : os.path.join(''.join(mET_weap.path_root), "tif"),                                # the root directory for output data
	'MOD13Q1_tif': os.path.join(''.join(mET_weap.path_root), "tif"+os.sep+"MOD13Q1"),        # the  directory for NDVI Subsets
	'ERA5_tif': os.path.join(''.join(mET_weap.path_root), "tif"+os.sep+"ERA5"),              # the  directory for ERA5 processed input files

	'outputs_ERA5': os.path.join(''.join(mET_weap.path_root), "tif"+os.sep+"outputs_ERA5"),  # The directory for results used for synthesis ERA5

	'synthesis_ERA5': os.path.join(''.join(mET_weap.path_root), "synthesis_ERA5")            # The directory for synthesized results based on ERA5 meteorological data
},

License

mET_weap uses the GPLv3 license. Copyright (C) 2014-2018 Michel Le Page. For more information see COPYING.