threedi-raster-edits 0.27

Threedi Raster Edits provides python tooling for threedi such as raster-conversion, alignment, fillers, checks and others

threedi-raster-edits

Introduction

Threedi Raster Edits is used internally at NENS for raster conversion, edits and checks. Currently it includes the functions as alignment, replace_nodata, alignment checks, property checks and raster conversions to friction, interception and others.

Usage

We can download rasters:

>>> import threedi_raster_edits as tre
>>> landuse = tre.UUID.THREEDI_LANDUSE
>>> vector = tre.Vector(vector_path)
>>> feature = vector[0]
>>> rextract = tre.RasterExtraction(lizard_api_key)
>>> rextract.run("ouput/raster.tif", landuse_uuid, feature.geometry, cellsize=0.5, threads=1)

Raster conversion:

>>> from threedi_raster_edits import ThreediRasterGroup
>>> group = ThreediRasterGroup(dem_path, landuse_path=landuse_path, soil_path=soil_path)
>>> group.load_landuse_conversion_table(csv_landuse_path)
>>> group.load_soil_conversion_table(csv_soil_path)
>>> group.generate_friction()
>>> group.generate_permeability()
>>> group.generate_interception()
>>> group.generate_crop_type()
>>> group.generate_max_infiltration()
>>> group.generate_infiltration()
>>> group.generate_hydraulic_conductivity()
>>> group.friction.write("friction.tif")

Raster checks:

>>> group = ThreediRasterGroup(dem_path, friction_path="friction.tif")
>>> align_checks = group.check_alignemnt()
>>> property_checks = group.check_properties()

Building interception can also be generated:

>>> from threedi_raster_edits import Vector
>>> buildings = Vector("buildings.shp")
>>> group = ThreediRasterGroup(dem_path, buildings=buildings)
>>> group.generate_building_interception(10)
>>> group.interception.write("interception.tif")

One can also use the internal raster class for easy raster processing as reproject, clipping, resampling, aligning, reading geometries and creating copies:

>>> from threedi_raster_edits import Raster
>>> import numpy as np
>>> raster = Raster("dem.tif")
>>> raster.reproject(28992)
>>> raster.resample(1,1)
>>> raster.clip(Vector("clip.shp"))
>>> raster.align(Raster("template.tif"))
>>> copy = raster.copy(shell=True)
>>> clip = Vector("clip.shp")
>>> geometry = clip[0].geometry # geometry of first feature
>>> clip_values = np.nansum(raster.read(clip))

For more advanced/larger processing you can also use raster loops:

>>> output = raster.copy(shell=True)
>>> for tile in raster:
>>>     array = tile.array
>>>     array[array==1] = 20
>>>     output.array = array, tile.location
>>> output.write("tiles_adding.tif")

Or just do something with the array:

>>> output = raster.copy(shell=True)
>>> array = raster.array
>>> array = array *100
>>> output.array = array
>>> output.write("output.tif")

Installation

We can be installed with:

$ pip install threedi-raster-edits

(TODO: after the first release has been made)

Development installation of this project itself

GDAL is not automatically installed, hence not available in the requirement,so please install gdal 3.2.0 yourself or use anaconda:

$ conda create -n threedi_raster_edits python=3 gdal=3.2.0 rtree black pytest

We use python’s build-in “virtualenv” to get a nice isolated directory. You only need to run this once:

$ python3 -m venv --system-site-packages .

A virtualenv puts its commands in the bin directory. So bin/pip, bin/pytest, etc. Set up the dependencies like this:

$ bin/pip install -r requirements.txt

There will be a script you can run like this:

$ bin/run-threedi-raster-edits

It runs the main() function in threedi-raster-edits/scripts.py, adjust that if necessary. The script is configured in setup.py (see entry_points).

In order to get nicely formatted python files without having to spend manual work on it, run the following command periodically:

$ bin/black threedi_raster_edits

Run the tests regularly. This also checks with pyflakes, black and it reports coverage. Pure luxury:

$ bin/pytest

The tests are also run automatically on “github actions” for “master” and for pull requests. So don’t just make a branch, but turn it into a pull request right away:

• Prepend the title with “[WIP]”, work in progress. That way you make clear it isn’t ready yet to be merged.

• Important: it is easy to give feedback on pull requests. Little comments on the individual lines, for instance. So use it to get early feedback, if you think that’s useful.

• On your pull request page, you also automatically get the feedback from the automated tests.

There’s also coverage reporting on coveralls.io (once it has been set up).

If you need a new dependency (like requests), add it in setup.py in install_requires. Local development tools, like “black”, can be added to the requirements.txt directoy. In both cases, run install again to actuall install your dependency:

$ bin/pip install -r requirements.txt

Releasing the project

To make a release, first install zest.releaser (globally):

$ pip install zest.releaser

Make sure you’re on master and that you’ve updated the changelog. Then release it with “fullrelease”. You can press “enter” at every question:

$ fullrelease

The github action ought to upload the fresh package to pypi.

Steps to do after generating with cookiecutter

• Add a new project on https://github.com/nens/ with the same name. Set visibility to “public” and do not generate a license or readme.

  Note: “public” means “don’t put customer data or sample data with real persons’ addresses on github”!

• Follow the steps you then see (from “git init” to “git push origin master”) and your code will be online.

• Go to https://github.com/nens/threedi-raster-edits/settings/collaboration and add the teams with write access (you might have to ask someone with admin rights to do it).

• Update this readme. Use .rst as the format.

• Ask Reinout to configure travis and coveralls.

• Remove this section as you’ve done it all :-)

Changelog of threedi-raster-edits

0.27 (2022-08-29)

• Bugfix: For raster tiles everything is not done in memory anymore.

• Cleanup for lizard including dependency installation.

• Updated for postgres upload. Works again.

• Updated blocksize estimation.

• Differences between stacked and adjacent rasters.

• No automatic console logging.

0.26 (2022-07-11)

• Geometry field changes of connection nodes.

0.25 (2022-07-08)

• Hotfix: removed aggregation_in_space in the v2_aggregation_settings

0.24 (2022-07-08)

• Hotfix: requirements.txt

0.23 (2022-07-07)

• Added two extra dependencies.

0.22 (2022-07-07)

• Removed 3Di results.

• Removed 3Di scenarios.

• Klondike support.

0.21 (2022-03-22)

• Scenarios > 3.8.0

0.20 (2022-03-22)

• QGIS changes for raster conversions.

• Include global cache, for storage in a pickle.

• Added easy threedi scenario downloads.

0.19 (2022-02-28)

• Increased speed of vector loops

• Added examples in scripts

• Added intuitive progress bars inclusing for gdal

0.18 (2021-11-29)

• Changed input for rextract.

0.17 (2021-11-29)

• Added threading to lizard rextract

• Also qgis compatible way of showing progress

• Simulations now in queue

0.16 (2021-10-28)

• Fixed release error

0.15 (2021-10-28)

• More efficient loading of threedirastergroup

0.14 (2021-10-28)

• ThreediAPI and ThreediResults are now optional.

0.13 (2021-10-19)

• Fixed vector reprojections

• Added inflated rasterloops for rasters

• Added rasterloops for rastergroups

• Added inflated rasterloops for rastergroups

• Added bbox clipping for rasters

0.12 (2021-09-13)

• Improved difference algorithm

• Remove geometry fixed at every call, now call once with veotor.fix()

• fid= -1 will result in a fid which is the count

0.11 (2021-09-06)

• Token release

0.10 (2021-09-06)

• Added sqlite-model support

• Added api support

0.9 (2021-05-06)

• Changed black format.

0.8 (2021-05-06)

• Added clips for rasters

• Added custom line string lengths

• Added vector interpolation

• Added (partly) fix threedi rasters

0.7 (2021-03-26)

• Fixed release process (same for 0.6/0.6).

0.4 (2021-03-26)

• Fixed release process.

• Fixed tests.

• Added logging.

• Better memory usage of rasters.

• Small changes in vector, geometries.

0.3 (2021-03-25)

• Automated pypi release.

0.2 (2021-03-12)

• Changed the syntax of raster class

• Changed the imports to the main script: E.g., from threedi_raster_edits import raster, rastergroup etc.

• Changed the readme.

• Rewritten the geometry structure.

0.1 (2021-03-11)

• Initial project structure created with cookiecutter and https://github.com/nens/cookiecutter-python-template