Skip to main content

The ICEBERG Rivers usecase package

Project description

Rivers (Arctic hydrology)

We provide a classification algorithm for ice surface features from high-resolution imagery. This algorithm was developed by convolutional neural network training to detect regions of large and small rivers and to distinguish them from crevasses and non-channelized slush. We also provide a detection algorithm to extract polyline water features from the classified high-probability river areas.

Prerequisites - all available on bridges via the commands below

  • Linux
  • Python 3
  • CPU and NVIDIA GPU + CUDA CuDNN

Software Dependencies - these will be installed automatically with the installation below.

  • numpy
  • scipy
  • tifffile
  • keras==2.3.1
  • opencv-python
  • rasterio
  • affine
  • pygdal==3.0.4
  • tensorflow-gpu==1.15.0

Installation

Preliminaries:
These instructions are specific to XSEDE Bridges2 but other resources can be used if cuda, python3, and a NVIDIA P100 GPU are present, in which case 'module load' instructions can be skipped, which are specific to Bridges.

For Unix or Mac Users:
Login to Bridges 2 via ssh using a Unix or Mac command line terminal. Login is available to bridges directly or through the XSEDE portal. Please see the Bridges 2 User's Guide.

For Windows Users:
Many tools are available for ssh access to Bridges 2. Please see Ubuntu, MobaXterm or PuTTY

PSC Bridges 2

Once you have logged into Bridges 2, you can follow one of two methods for installing iceberg-rivers.

Method 1 (Recommended):

The lines below following a '$' are commands to enter (or cut and paste) into your terminal (note that all commands are case-sensitive, meaning capital and lowercase letters are differentiated.) Everything following '#' are comments to explain the reason for the command and should not be included in what you enter. Lines that do not start with '$' or '[rivers_env] $' are output you should expect to see.

$ pwd
/home/username
$ cd $PROJECT                      # switch to your working space.
$ mkdir Rivers                      # create a directory to work in.
$ cd Rivers                         # move into your working directory.
$ module load AI/anaconda3-tf1.2020.11
$ export PATH=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/bin:$PATH
$ export LD_LIBRARY_PATH=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/lib:$LD_LIBRARY_PATH
$ export GDAL_DATA=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/share/gdal
$ conda create --prefix iceberg_rivers --clone $AI_ENV
$ source activate iceberg_rivers/
[iceberg_rivers] $ pwd
/ocean/projects/group/username/Rivers
[iceberg_rivers]$ export PYTHONPATH=/ocean/projects/group/username/iceberg_rivers/lib/python3.7/site-packages/
[iceberg_rivers]$ pip install iceberg_rivers

Method 2 (Installing from source; recommended for developers only):

$ cd $PROJECT                      # switch to your working space.
$ mkdir Rivers                      # create a directory to work in.
$ cd Rivers                         # move into your working directory.
$ module load AI/anaconda3-tf1.2020.11
$ export PATH=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/bin:$PATH
$ export LD_LIBRARY_PATH=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/lib:$LD_LIBRARY_PATH
$ export GDAL_DATA=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/share/gdal
$ conda create --prefix iceberg_rivers --clone $AI_ENV
$ source activate iceberg_rivers/
[rivers_env] $ pwd
/ocean/projects/group/username/Rivers
$ git clone https://github.com/iceberg-project/Rivers.git
[iceberg_rivers] $ export PYTHONPATH=/ocean/projects/group/username/iceberg_rivers/lib/python3.7/site-packages/
[iceberg_rivers] $ pip install .

To test

[iceberg_rivers] $ deactivate       # exit your virtual environment.
$ interact --gpu  # request a compute node.  This package has been tested on P100 GPUs on bridges, but that does not exclude any other resource that offers the same GPUs. (this may take a minute or two or more to receive an allocation).
$ cd $PROJECT/Rivers                # make sure you are in the same directory where everything was set up before.
$ module load AI/anaconda3-tf1.2020.11
$ export PATH=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/bin:$PATH
$ export LD_LIBRARY_PATH=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/lib:$LD_LIBRARY_PATH
$ export GDAL_DATA=/ocean/projects/mcb110096p/paraskev/gdal-3.0.4/share/gdal
$ source activate iceberg_rivers/    # activate your environment, no need to create a new environment because the Rivers tools are installed and isolated here.
[iceberg_rivers] $ export PYTHONPATH=/ocean/projects/group/username/iceberg_rivers/lib/python3.7/site-packages/
[iceberg_rivers] $ iceberg_rivers.tiling --help  # this will display a help screen of available usage and parameters.

Prediction

  • Download a pre-trained model at:

You can download to your local machine and use scp, ftp, rsync, or Globus to transfer to bridges.

Rivers predicting is executed in three steps: First, follow the environment setup commands under 'To test' above. Then create tiles from an input GeoTiff image and write to the output_folder. The scale_bands parameter (in pixels) depends on the trained model being used. The default scale_bands is 299 for the pre-trained model downloaded above. If you use your own model the scale_bands may be different.

[iceberg_rivers] $ iceberg_rivers.tiling --tile_size=224 --step=112 --input=<image_abspath> --output=./test/

Then, detect rivers on each tile and output counts and confidence for each tile.

[iceberg_rivers] $ iceberg_rivers.predict --input <tile_folder> -o <output_folder> -w <model>

Finally, mosaic all the tiles back into one image

[iceberg_rivers] $ iceberg_rivers.mosaic --input_WV image --input <masks_folder> --tile_size 224 --step 112 --output_folder ./mosaic

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

iceberg_rivers.search-0.1.5.tar.gz (21.2 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page