hdsr-wis-config-reader 1.4

Read HDSR FEWS WIS config into python objects

Context

• Created: November 2021
• Author: Renier Kramer, [e-mail]
• Python version: >3.5

Description

A python project that you can use to read the HDSR FEWS-WIS configuration. It serves an interface (python objects) for configuration files (.xmls and .csv).

Usage (examples)

pip install hdsr-wis-config-reader
>>> from hdsr_wis_config_reader import XmlReader, FewsConfigReader
>>> from hdsr_wis_config_reader import LocationSetCollection, IdMappingCollection
>>> import pathlib  # dependency of hdsr-wis-config-reader

>>> config_dir = pathlib.Path(<path_to_fews_config_dir>)  # this points to the FEWS <region_home>/config dir
>>> fews_config = FewsConfigReader(path=config_dir)
>>> location_sets = LocationSetCollection(fews_config=fews_config)
>>> id_mappings = IdMappingCollection(fews_config=fews_config)

# Get the absoluate path to a config file
>>> fews_config.RegionConfigFiles["Filters"]
# ../<your_config_dir>/RegionConfigFiles/Filters.xml'

# Get hoofdlocationset names
>>> location_sets.hoofdloc.name
# 'hoofdlocaties'
>>> location_sets.hoofdloc.csv_filename
'oppvlwater_hoofdloc'
>>> location_sets.hoofdloc.fews_name
'OPVLWATER_HOOFDLOC'

# Extract a dictionary of parameter(groups)
>>> fews_config.get_parameters(dict_keys="groups")["Neerslag"]
# {
#   "parameterType": "accumulative",
#   "unit": "mm", 
#   "valueResolution": "0.001",
#   "parameter": [{"shortName: ...
#   etc..
# }
>>> fews_config.get_parameters(dict_keys="parameters")["RHdef.d"]
# {
#    "shortName": "Neerslagtekort [mm] - dag",
#    "name": "Neerslagtekort [mm] - dag",
#    "parameterType": "accumulative",
#    "unit": "mm",
#    "valueResolution": "0.001",
#    "groupId": "Neerslag",
# }
       
# Get a geodataframe with hoofd locations (note that geomeometry height (z) gets -9999 when if Z is nan)
>>> location_sets.hoofdloc.geo_df
# LOC_ID      LOC_NAME                            X       Y       Z       ALLE_TYPES          START       ...     geometry
# KW100110    WIJKERSLOOT_1001-K_WIJKERSLOOT      150501  442988  2.87    krooshek/pompvijzel 19970101    ...     POINT Z (150501 442988 2.87)
# KW100120    WIJKERSLOOT_1001-K_WIJKERSLOOT STUW 150439  442885  nan     stuw                20120321    ...     POINT Z (150439 442885 -9999)
# etc...

# Get the validation rules for sublocations
>>> location_sets.subloc.validation_rules
# [
#   {'parameter': 'H.R.', 'extreme_values': {'hmax': 'HR1_HMAX', 'hmin': 'HR1_HMIN'}}, 
#   {'parameter': 'H2.R.', 'extreme_values': {'hmax': 'HR2_HMAX', 'hmin': 'HR2_HMIN'}},
#   etc..
# ]

# Get the attribute files for waterstand locations 
>>> location_sets.waterstandloc.attrib_files
# [
#   {   'csvFile': 'oppvlwater_langsprofielen', 
#       'id': '%LOC_ID%', 
#       'attribute': [
#           {'number': '%Langsprofiel_Kromme_Rijn%', 'id': 'Langsprofiel_Kromme_Rijn'}, 
#           {'number': '%Langsprofiel_Caspargouwse_Wetering%', 'id': 'Langsprofiel_Caspargouwse_Wetering'},
#   etc...

# Get filtered idmapping (you can filter on one or more args: {ex_loc, ex_par, int_loc, int_par}): 
>>> df_idmap = id_mappings.idmap_all.get_filtered_df(int_loc="KW761001")
# externalLocation  externalParameter   internalLocation    internalParameter   source              histtag
# 610               Q1                  KW761001            Q.G.0               IdOPVLWATER         610_Q1
# 7610              Q1                  KW761001            Q.G.0               IdOPVLWATER         7610_Q1
# 610               Q1                  KW761001            Q.G.0               IdOPVLWATER_HYMOS   610_Q1

# Get all external parameters in the groundwater idmapping
>>> from hdsr_wis_config_reader.idmappings.columns import IdMapColumnChoices
>>> id_mappings.idmap_grondwater_caw.get_filtered_column_values(
        make_result_unique=True,
        target_column=IdMapColumnChoices.ex_par,
    )  
# ['1GW', 'GW1', 'GW2', 'GW3', 'HB1', 'HB2', 'HB3', 'HB4', 'HB5', 'HB6', 'HB7', 'HB8']

License

MIT

Contributions

All contributions, bug reports, bug fixes, documentation improvements, enhancements and ideas are welcome on https://github.com/hdsr-mid/hdsr_wis_config_reader/issues

Test Coverage

---------- coverage: platform win32, python 3.7.11-final-0 -----------
Name                                                    Stmts   Miss  Cover
---------------------------------------------------------------------------
hdsr_wis_config_reader\__init__.py                         10      0   100%
hdsr_wis_config_reader\constants.py                         6      1    83%
hdsr_wis_config_reader\idmappings\collection.py            72      4    94%
hdsr_wis_config_reader\idmappings\columns.py               80     16    80%
hdsr_wis_config_reader\idmappings\custom_dataframe.py      28      1    96%
hdsr_wis_config_reader\idmappings\files.py                  7      0   100%
hdsr_wis_config_reader\idmappings\sections.py               8      0   100%
hdsr_wis_config_reader\location_sets\base.py               98     10    90%
hdsr_wis_config_reader\location_sets\collection.py         54      4    93%
hdsr_wis_config_reader\location_sets\hoofd.py              18      1    94%
hdsr_wis_config_reader\location_sets\msw.py                12      1    92%
hdsr_wis_config_reader\location_sets\ow.py                 13      1    92%
hdsr_wis_config_reader\location_sets\ps.py                 12      1    92%
hdsr_wis_config_reader\location_sets\sub.py                27      2    93%
hdsr_wis_config_reader\readers\config_reader.py           134     14    90%
hdsr_wis_config_reader\readers\xml_reader.py               48     10    79%
hdsr_wis_config_reader\validation_rules\__init__.py         4      0   100%
hdsr_wis_config_reader\validation_rules\files.py           35     10    71%
hdsr_wis_config_reader\validation_rules\logic.py           28      0   100%
---------------------------------------------------------------------------
TOTAL                                                     694     76    89%

Conda general tips

Build conda environment (on Windows) from any directory using environment.yml:

> conda env create --name <conda_env_name> --file <path_to_project>/environment.yml python=<python_version>
> conda info --envs  # verify that <conda_env_name> is in this list 

Start the application from any directory:

> conda activate <conda_env_name>
At any location:
> (<conda_env_name>) python <path_to_project>/main.py

Test the application:

> conda activate <conda_env_name>
> cd <path_to_project>
> pytest  # make sure pytest is installed (conda install pytest)

List all conda environments on your machine:

At any location:
> conda info --envs

Delete a conda environment:

Get directory where environment is located 
> conda info --envs
Remove the enviroment
> conda env remove --name <conda_env_name>
Finally, remove the left-over directory by hand

Write dependencies to environment.yml:

The goal is to keep the .yml as short as possible (not include sub-dependencies), yet make the environment reproducible. Why? If you do 'conda install matplotlib' you also install sub-dependencies like pyqt, qt icu, and sip. You should not include these sub-dependencies in your .yml as:

• including sub-dependencies result in an unnecessary strict environment (difficult to solve when conflicting)
• sub-dependencies will be installed when dependencies are being installed

> conda activate <conda_env_name>

Recommended:
> conda env export --from-history --no-builds | findstr -v "prefix" > --file <path_to_project>/environment_new.yml   

Alternative:
> conda env export --no-builds | findstr -v "prefix" > --file <path_to_project>/environment_new.yml 

--from-history: 
    Only include packages that you have explicitly asked for, as opposed to including every package in the 
    environment. This flag works regardless how you created the environment (through CMD or Anaconda Navigator).
--no-builds:
    By default, the YAML includes platform-specific build constraints. If you transfer across platforms (e.g. 
    win32 to 64) omit the build info with '--no-builds'.

Pip and Conda:

If a package is not available on all conda channels, but available as pip package, one can install pip as a dependency. Note that mixing packages from conda and pip is always a potential problem: conda calls pip, but pip does not know how to satisfy missing dependencies with packages from Anaconda repositories.

> conda activate <conda_env_name>
> conda install pip
> pip install <pip_package>

The environment.yml might look like:

channels:
  - defaults
dependencies:
  - <a conda package>=<version>
  - pip
  - pip:
    - <a pip package>==<version>

You can also write a requirements.txt file:

> pip list --format=freeze > <path_to_project>/requirements.txt