Sandpyper

- Sandy beaches SfM-UAV analysis tools -

Sandpyper performs an organised and automated extraction of color and elevation profiles from as many DSM and orthophotos as you like. It is thought to be used when a considerable number of DSMs and orthophotos from many different locations and coordinate reference systems need to be processed. Then, it computes volumetric and behavioural analysis of sandy beachfaces, speeding up an otherwise long and difficult to handle job. It has some specialised functions to help dealing with the common limitations found in working with Unoccupied Aerial Vehicles (UAVs) and Structure from Motion (SfM) in beach environments, which are:

1. Swash zone: the water motion of waves washing in and out of the swash zone prevents SfM algorithm to reliably model elevation. It is commonly discarded.
2. Vegetation: both dune vegetation and beach wracks (macroalgae, woody debris) should be removed or filtered as anything that is not sand would compromise sediment volumetric computation and behavioural analysis.
3. File size: a few km long beach surveyed with a DJI Phantom 4-Advanced at 100 meters altitude creates roughly 10 Gb (uncompressed) of data, which can be cumbersome for some GIS to handle.

From user-defined cross-shore transects, Sandpyper helps with:

1. cleaning profiles from unwanted non-sand points
2. computing period-specific limits of detection to obtain reliable estimates of changes
3. detecting statistically significant clusters of change (also referred to hotspots/coldspots) of beach change
4. computing multiscale volumetric analysis
5. modeling multiscale Beachface Cluster Dynamics indices
6. visualising beach changes, limits of detection, transects and BCDs with a variety of in-built plotting methods

Additionally, Sandpyper has some useful functions that can come at hand, such as: automatic transect creation from a vector line, grid creation along a line and subsequent tiles extraction and others.

1. automatic transects creation from a vector line
2. spatial grid creation of specified tile size along a line
3. tiles extraction from spatial grid

Sandpyper is very easy to use.

As the above image shows, Sandpyper processing pipeline is mainly composed of 3 main components:

1. Raw data extraction
2. Data correction
3. Sediment dynamics analysis

To achieve this in an organised and coherent way, Sandpyper provides two core objects, the ProfileSet and the ProfileDynamics classes.
The ProfileSet class sets up the monitoring global parameters, extracts the change data and Limit of Detections, implements iterative silhouette analysis with inflexion point search and facilitates point cleaning by using user-provided watermasks, shoremasks and class dictionaries.
The ProfileDynamics class is the core processor for beach dynamics. It computes multitemporal and multiscale elevation change, performs Hotspot/Coldspot analysis at the location level, discretises the data into classes of magnitude of change, models multiscale behavioural dynamics and provides many different plotting options.
After instantiation, both classes will gradually store more and more beach change information in their attributes.

Follow the Jupyter Notebook tutorials to understand how it works!

This code has supported the analysis and publication of the article "Citizen science for monitoring seasonal-scale beach erosion and behaviour with aerial drones", in the open access Nature Scientific Report journal.

About The Project

Background

Sandpyper has been originally developed to facilitate the analysis of a large dataset coming from more than 300 Unoccupied Aerial Vehicles (UAV) surveys, performed by Citizen Scientist in Victoria, Australia. This is the Eureka Award Winning World-first beach monitoring program powered by volunteers, who autonomously fly UAVs on 15 sensitive sites (erosional hotspots) across the Victorian coast, every 6 weeks for 3 years, since 2018. This project is part of a broader marine mapping program called The Victorian Coastal Monitoring Program (VCMP), funded by the Victorian Department of Environment, Land, Water and Planning, co-funded by Deakin University and The University of Melbourne.

Each survey creates Digital Surface Models (DSMs) and orthophotos of considerable size (5-10 Gb uncompressed), which can be troublesome for some GIS to render, let alone perform raster-based computations.

Modules

• sandpyper: main module where the ProfileSet and ProfileDynamics classes and methods are defined.
• common: where all the functions are stored.

Getting Started

Currently Sandpyper is tested on Windows, MacOS and Ubuntu with Python 3.8 and 3.9. To get a local copy up and running follow these simple steps.

Prerequisites

• Install Conda in your local machine. We need it to create the sandpyper_env virtual environment.

• then, if you do not have it already, install Visual Studio C++ build tools . You can download it here.

• If you don't have it already, add conda-forge channel to your anaconda config file, by typing this in your Anaconda Prompt terminal (base environment):

  conda config --add channels conda-forge
  

• Now, always in the (base) environment, create a new environment called sandpyper_env using python=3.9 and install the required packages by typing:

  conda create --name sandpyper_env python=3.9 geopandas=0.8.2 matplotlib=3.3.4 numpy=1.20.1 pandas=1.2.2 tqdm=4.56.2 pysal=2.1 rasterio=1.2.0 richdem=0.3.4 scikit-image=0.18.1 scikit-learn=0.24.1 scipy=1.6.0 seaborn=0.11.1 tqdm=4.56.2 pooch=1.4.0 fuzzywuzzy
  

• If rasterio package cannot be installed due to GDAL binding issues, follow the instructions in rasterio installation webpage.

• If you want to test the package using the provided notebooks, download the test data (test_data.rar) HERE

Installation

1. conda activate sandpyper_env
   

2. pip install sandpyper
   

3. Install Jupyter Notebooks:

   conda install jupyter
   

4. Once you open a Jupyter Notebook with the sandpyper_env, import it to test it works.

   import sandpyper
   

Usage

To see Sandpyper in action, follow the Jupyter Notebooks provided here.
Download the test data file (test_data.rar) HERE.
For the API reference, see the Documentation.

Roadmap

1. Update CRS definition to new CRS object standard in order to upgrade Geopandas version.
2. Relax all requirements to make it future-proof and available in Anaconda.org.
3. Add raster support for Dems of Differences (DoDs) and LoDs.
4. Add shoreline analysis from space.
5. Add [leafmap](https://github.com/giswqs/leafmap) to better (interactive) plotting.
6. Add automatic check for overlapping label correction polygons.

See the open issues for a list of proposed features (and known issues).

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Nicolas Pucino: @NicolasPucino - npucino@deakin.edu.au
Project Link: https://github.com/npucino/sandpyper

Publications

• Pucino, N., Kennedy, D. M., Carvalho, R. C., Allan, B. & Ierodiaconou, D. Citizen science for monitoring seasonal-scale beach erosion and behaviour with aerial drones. Sci. Rep. 11, 3935 (2021)

Acknowledgements

• Funding provided by Deakin University and the Victorian Department of Environment, Land, Water and Planning.
• Deakin University Citizen Science program webpage.
• My PhD supervisor A/Prof Daniel Ierodiaconou, and the Deakin Marine Mapping team.