iceberg-rivers.search 0.1.5

The ICEBERG Rivers usecase package

Rivers (Arctic hydrology)

We provide a classification algorithm for ice surface features from high-resolution imagery. This algorithm was developed by convolutional neural network training to detect regions of large and small rivers and to distinguish them from crevasses and non-channelized slush. We also provide a detection algorithm to extract polyline water features from the classified high-probability river areas.

Prerequisites - all available on bridges via the commands below

• Linux
• Python 3
• CPU and NVIDIA GPU + CUDA CuDNN

Software Dependencies - these will be installed automatically with the installation below.

• numpy
• scipy
• tifffile
• keras==2.3.1
• opencv-python
• rasterio
• affine
• pygdal==3.0.4
• tensorflow-gpu==1.15.0

Installation

Preliminaries:
These instructions are specific to XSEDE Bridges2 but other resources can be used if cuda, python3, and a NVIDIA P100 GPU are present, in which case 'module load' instructions can be skipped, which are specific to Bridges.

For Unix or Mac Users:
Login to Bridges 2 via ssh using a Unix or Mac command line terminal. Login is available to bridges directly or through the XSEDE portal. Please see the Bridges 2 User's Guide.

For Windows Users:
Many tools are available for ssh access to Bridges 2. Please see Ubuntu, MobaXterm or PuTTY

PSC Bridges 2

Once you have logged into Bridges 2, you can follow one of two methods for installing iceberg-rivers.

Method 1 (Recommended):

The lines below following a '$' are commands to enter (or cut and paste) into your terminal (note that all commands are case-sensitive, meaning capital and lowercase letters are differentiated.) Everything following '#' are comments to explain the reason for the command and should not be included in what you enter. Lines that do not start with '$' or '[rivers_env] $' are output you should expect to see.

$ pwd
/home/username
$ cd $PROJECT                      # switch to your working space.
$ mkdir Rivers                      # create a directory to work in.
$ cd Rivers                         # move into your working directory.
$ module load AI/anaconda3-tf1.2020.11
$ export PATH=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/bin:$PATH
$ export LD_LIBRARY_PATH=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/lib:$LD_LIBRARY_PATH
$ export GDAL_DATA=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/share/gdal
$ conda create --prefix iceberg_rivers --clone $AI_ENV
$ source activate iceberg_rivers/
[iceberg_rivers] $ pwd
/ocean/projects/group/username/Rivers
[iceberg_rivers]$ export PYTHONPATH=/ocean/projects/group/username/iceberg_rivers/lib/python3.7/site-packages/
[iceberg_rivers]$ pip install iceberg_rivers

Method 2 (Installing from source; recommended for developers only):

$ cd $PROJECT                      # switch to your working space.
$ mkdir Rivers                      # create a directory to work in.
$ cd Rivers                         # move into your working directory.
$ module load AI/anaconda3-tf1.2020.11
$ export PATH=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/bin:$PATH
$ export LD_LIBRARY_PATH=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/lib:$LD_LIBRARY_PATH
$ export GDAL_DATA=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/share/gdal
$ conda create --prefix iceberg_rivers --clone $AI_ENV
$ source activate iceberg_rivers/
[rivers_env] $ pwd
/ocean/projects/group/username/Rivers
$ git clone https://github.com/iceberg-project/Rivers.git
[iceberg_rivers] $ export PYTHONPATH=/ocean/projects/group/username/iceberg_rivers/lib/python3.7/site-packages/
[iceberg_rivers] $ pip install .

To test

[iceberg_rivers] $ deactivate       # exit your virtual environment.
$ interact --gpu  # request a compute node.  This package has been tested on P100 GPUs on bridges, but that does not exclude any other resource that offers the same GPUs. (this may take a minute or two or more to receive an allocation).
$ cd $PROJECT/Rivers                # make sure you are in the same directory where everything was set up before.
$ module load AI/anaconda3-tf1.2020.11
$ export PATH=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/bin:$PATH
$ export LD_LIBRARY_PATH=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/lib:$LD_LIBRARY_PATH
$ export GDAL_DATA=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/share/gdal
$ source activate iceberg_rivers/    # activate your environment, no need to create a new environment because the Rivers tools are installed and isolated here.
[iceberg_rivers] $ export PYTHONPATH=/ocean/projects/group/username/iceberg_rivers/lib/python3.7/site-packages/
[iceberg_rivers] $ iceberg_rivers.tiling --help  # this will display a help screen of available usage and parameters.

Prediction

• Download a pre-trained model at:

You can download to your local machine and use scp, ftp, rsync, or Globus to transfer to bridges.

Rivers predicting is executed in three steps: First, follow the environment setup commands under 'To test' above. Then create tiles from an input GeoTiff image and write to the output_folder. The scale_bands parameter (in pixels) depends on the trained model being used. The default scale_bands is 299 for the pre-trained model downloaded above. If you use your own model the scale_bands may be different.

[iceberg_rivers] $ iceberg_rivers.tiling --tile_size=224 --step=112 --input=<image_abspath> --output=./test/

Then, detect rivers on each tile and output counts and confidence for each tile.

[iceberg_rivers] $ iceberg_rivers.predict --input <tile_folder> -o <output_folder> -w <model>

Finally, mosaic all the tiles back into one image

[iceberg_rivers] $ iceberg_rivers.mosaic --input_WV image --input <masks_folder> --tile_size 224 --step 112 --output_folder ./mosaic