fews-3di 2.0

FEWS-3di coupling

fews-3di

Program to start 3Di simulations from FEWS.

Installation and usage

We can be installed using python 3.6+ with:

$ pip install fews-3di

The script is called run-fews-3di, you can pass --help to get usage instructions and --verbose to get more verbose output in case of problems.

run-fews-3di looks for a run_info.xml in the current directory by default, but you can pass a different file in a different location with --settings:

$ run-fews-3di
$ run-fews-3di --help
$ run-fews-3di --settings /some/directory/run_info.xml

Configuration and input/output files

The expected information in run_info.xml is:

<?xml version="1.0" encoding="UTF-8"?>
<Run xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xmlns="http://www.wldelft.nl/fews/PI"
     xsi:schemaLocation="http://www.wldelft.nl/fews/PI
                         http://fews.wldelft.nl/schemas/version1.0/pi-schemas/pi_run.xsd"
                         version="1.5">
    <startDateTime date="2020-01-26" time="10:00:00"/>
    <endDateTime date="2020-01-30" time="12:00:00"/>
    <properties>
        <string key="username" value="pietje"/>
        <string key="password" value="onder-de-deurmat"/>
        <string key="organisation" value="12345678abcd"/>
        <string key="modelrevision" value="abcd123456787"/>
        <string key="simulationname" value="Simulation name"/>
        <string key="save_state" value="True"/>
        <string key="save_state_time" value="1400" />
        <string key="use_last_available_state" value="False" />
        <string key="saved_state_expiry_days" value="5"/>
        <string key="rain_type" value="radar"/>
        <string key="rain_input" value="730d6675-35dd-4a35-aa9b-bfb8155f9ca7"/>
        <string key="fews_pre_processing" value="True"/>
        <string key="lizard_results_scenario_name" value="Testsimulatie"/>
        <string key="lizard_results_scenario_uuid" value=""/>
        <string key="initial_waterlevel" value=""/>
        <string key="api_host" value=""/>
    </properties>
</Run>

Note: saved_state_expiry_days used to be spelled as save_state_expiry_days, without a “d”. The example radar uuid is the Dutch rainfall radar (NRR).

Using saved states: To use a warm state provide a text file with id in the states folder using the name states/3di-saved-state-id.txt. A cold state is supplied in a similar way with the name: states/3di-cold-state-id.txt.

Use last available state: To overpass the state management system and directly take the last available state in the 3Di database the option: use_last_available_state can be set to True.

Saving state: When saving a state a save_state_time can be specified. This parameter defines the time in the simulation (in seconds) when the state should be saved. If left empty the end of the simulation is used.

Rain_type: multipe rain-types can be used in the configuration:

• constant
• radar
• custom

Rain_input: according to the chosen rain-type, a rain input must be given

in the configuration:

• constant –> integer [m/s]
• radar –> lizard uuid
• custom –> two options: rain_csv or rain_netcdf. These files must be stored in the input directory as input/rain.csv and input/precipitation.nc

fews_pre_processing: can be True or False. Must be True if the

results are needed in fews: additional pre_processing of the results is needed.

initial_waterlevel: can be min, max, or mean. When specified

the initial waterlevel raster is taken into account. If left empty no initial waterlevel is used in the simulation.

initial_waterlevel: if you want to use the initial waterlevel raster as

defined in the settings (leave empty if no initial waterlevel is predefined):

• min
• mean
• max

api_host: (optional) api_host address can be added here. If not provided

the default api_host address (”https://api.3di.live/v3.0”) will be used.

Several input files are needed, they should be in the input directory relative to the run_info.xml:

• run_info.xml
• input/lateral.csv
• input/precipitation.nc
• input/evaporation.nc
• input/ow.nc
• model/gridadmin.h5

Output is stored in the output directory relative to the run_info.xml:

• output/simulation.log (unavailable, but included in the zip)
• output/flow_summary.log (idem)
• output/log_files_sim_ID.zip
• output/results_3di.nc
• output/dischages.csv
• output/ow.nc

Development

Development happens on github. See DEVELOPMENT.rst for more information.

Changelog of fews-3di

2.0 (2022-06-21)

• Requiring threedi-api-client 4.0.1 or higher. This has several import and functionality changes, which we’ll have to use (and compensate for). Some items to watch out for:
  • The api host setting should not include the api version number.
  • Preferrably, don’t add a trailing slash to the api host url.
• .login() no longer needs to be called. threedi-api-client handles it automatically. If you call it, you get a deprecation warning.
• --allow-missing-saved-state also works if there are no states found.

1.15 (2022-06-10)

• Added optional api_host parameter to the settings file.

1.14 (2022-02-08)

• Added missing requests dependency to package setup.

1.13 (2021-09-01)

• Added possibility to write state at specific time intervall.

1.12 (2021-04-28)

• Fixes for 1.11, initial waterlevel should work now.

1.11 (2021-04-28)

• Added possibility to add initial waterlevel raster.

1.10 (2021-02-09)

• Added the functionality to provide a cold state file. Place next to original state file with the name: 3di-cold-state-id.txt.

1.9 (2021-01-27)

• Added new rainfall modules, constant, csv and radar rain.
• Processing results into fews is now optional.

1.7 (2020-11-13)

• Checks for crashed status and queue’s model.

1.6 (2020-10-19)

• Using a “streaming” download of large files to prevent partial downloads.

1.5 (2020-09-21)

• Added more resilience to local network errors. The loop that waits for results to be ready checks the state every 30 seconds and is thus the most vulnerable to wifi issues, a flaky VPN and local network hickups. We now detect such a socket.gaierror there and simply try again in 30 seconds.

1.4 (2020-07-21)

• A minor bugfix in the result files which are downloaded after the simulation

1.3 (2020-07-16)

• A minor bugfix in the location where the script searches for the saved-state file

1.2 (2020-07-09)

• The code has been set-up to look for specific filenames in predefined folders.
• All inputs (rain, evaporation etc.) have now become optional, if one is absent a logging message is returned but the code will run. This allows for flexibility in the usage of the code with different kinds of input.
• Two new optional parameters have been added: lizard_results_scenario_uuid and lizard_results_scenario_name. If a Lizard results scenario name is provided, results will be processed in Lizard. If it is not provided, the simulation runs as usual without processing.

1.1 (2020-05-04)

• When an existing saved state isn’t found, it can be because it is the first time the script is run. Or the previous saved data has expired. The error message now points at the --allow-missing-saved-state possibility. This can be used to allow the missing of the saved state: a new one will be created.
• Fixed bug: two lines were accidentally swapped, leading to an early crash.

1.0 (2020-05-04)

• Code cleanup + more coverage.
• Improved the documentation, including a separate DEVELOPMENT.rst to keep those details out of the generic readme.

0.4 (2020-04-30)

• Reading and storing saved states added.

0.3 (2020-04-23)

• Release mechanism fix.

0.2 (2020-04-23)

• Added lateral upload.
• Added rain upload.
• Added evaporation upload.
• Simulation is actually being run now.
• Added processing of the results.
• Added usage instructions.

0.1 (2020-04-09)

• Started copying code from the old project.
• Got 3Di api connection to work, including creating an (empty) simulation.
• Initial project structure created with cookiecutter and https://github.com/nens/cookiecutter-python-template

Development instructions

Development installation

To install:

$ make install

If you want to use a specific version of python, you can first run something like this:

$ python3.9 -m venv .

There will be a script you can run like this:

$ bin/run-fews-3di

It runs the main() function in fews-3di/scripts.py, adjust that if necessary. The script is configured in setup.py (see entry_points).

If you need a new dependency (like requests), add it in setup.py in install_requires. Local development tools, like “black”, can be added to the requirements.txt file. In both cases, run make install again.

Code structure

• fews_3di/scripts.py: the run-fews-3di code. Should only handle the commandline stuff and logging setup.
• fews_3di/simulation.py: the main ThreediSimulation class. The various steps like “add rain” and “start simulation” are separate methods on that class: this way you can keep the overview of what’s happening. It is a class to make it easier to share common data like “simulation id”.
• fews_3di/utils.py: reading the settings plus some helper functions like timestamps_from_netcdf().
• fews_3di/tests/*: the tests, including sample data.

Error handling and logging

Try/excepts are only used when strictly necessary. Unexpected errors will simply be shown as a traceback.

Some errors are expected, like a missing setting or a missing netcdf file. For these, there’s an explicit error class like MissingSettingException. These are caught in scripts.py and shown as a neat error message. With --verbose, you also get the traceback.

Debug logging is used to make it easy to figure out what the program is doing in case of a problem or an unexpected result.

Info level logging is for feedback to the user. Don’t log too much on this level.

Warning/error are the usual. An error when something is wrong and we’re stopping the script. A warning for when something seems wrong, but when we’ll continue execution anyway.

Neatness and tests

In order to get nicely formatted python files without having to spend manual work on it, run the following command periodically:

$ make beautiful

If you don’t have “make” installed, look in the Makefile for the commands it runs (black, flake8, isort).

Run the tests regularly, this includes pyflakes and black checks:

$ make test

Running pytest by itself is also possible, for instance if you want to pass specific options:

$ bin/pytest --disable-warnings

The tests are also run automatically on “github actions” for “master” and for pull requests. So don’t just make a branch, but turn it into a pull request right away:

• Important: it is easy to give feedback on pull requests. Little comments on the individual lines, for instance. So use it to get early feedback, if you think that’s useful.
• On your pull request page, you also automatically get the feedback from the automated tests.

There’s also coverage reporting on coveralls.io.

As an experiment, python type hints are sprinkled throughout the code. When running the tests, errors are often found. The reason for the experiment was some confusion in the original version of fews-3di:

• A string “True” from the settings instead of a proper boolean True/False value.
• Timestamps that were sometimes datetime objects and sometimes iso-formatted datetime strings.

With type hints, it is perfectly clear what something is supposed to be.

Releases

Before releasing, make sure the changelog is up to date, otherwise zest.releaser complains :-) Then run fullrelease:

$ bin/fullrelease

Github detects the new tag and automatically uploads a new release to https://pypi.org/project/fews-3di/