Skip to main content

A data visualization tool for PyCCD time-series model results and Landsat ARD

Project description

Abstract

The Timeseries Analysis and Plotting (TAP) Tool is being developed to provide exploratory data analysis for the Land Change Monitoring, Assessment, and Projection (LCMAP) project at USGS EROS. TAP is an open-source project written in python that uses PyQt5 and matplotlib for generating interactive GUIs and plots. Currently, it provides the following functionality:

  • Plot a timeseries of Landsat Analysis Ready Data (ARD)
    • Landsat 4-7 Bands 1, 2, 3, 4, 5, 7 Surface Reflectance
    • Landsat 8 Bands 2-7 Surface Reflectance
    • Landsat 4-7 Band 6 Brightness Temperature
    • Landsat 8 Band 10 Brightness Temperature
  • Plot model parameters and curve-fits from the Continuous Change Detection and Classification (CCDC) algorithm
  • Calculate indices on the fly for plotting and visualization
    • NDVI
    • MSAVI
    • SAVI
    • EVI
    • NDMI
    • NBR
    • NBR-2
  • Display RGB visualizations of Landsat ARD
  • Export plotted Landsat ARD surface reflectance, brightness temperature, and index values to a .csv file
  • Save a plot figure as a .png file
  • Save a Landsat ARD RGB display as a .png file
  • Save ESRI point shapefile for plotted coordinates

Installation

These instructions work for both Windows and Linux systems.

lcmap_tap has been installed and tested successfully on Windows 7 & 10, and CentOS Linux (version 7).

Note: some functionality could be unstable or intermittent as development of LCMAP operations is ongoing.

System Requirements

  • python >= 3.6
  • GDAL == 2.2.2
  • poppler == 0.60.1 (Linux only)
  • PyQt5 == 5.10.1*
  • cython
  • cytoolz
  • matplotlib
  • numpy
  • pandas
  • pyyaml
  • requests
  • lcmap-merlin

* Note on PyQt5: For development any version of PyQt5>=5.9 is currently suitable. For running TAP, it is recommended to use PyQt5==5.10.1 because of an issue in later versions in which Qtwebengineprocess does not close on TAP exit. If building a stand-alone executable with PyInstaller, PyQt5==5.10.1 seems to work best at the moment.


The tool currently requires the following data inputs:

  • Serialized change and cover results generated by PyCCD.
  • Landsat ARD (obtained via HTTP request)

Landsat ARD and the PyCCD algorithm are publicly available. However, TAP currently requires access to ARD via a web-service that is currently available to on-site personnel at USGS EROS only.

In addition, the pre-generated PyCCD results must be stored locally and are expected to be in a specific format.

For these reasons, TAP is not intended for general public use until these datasets become widely available at a future date.


Note

It is recommended to use an Anaconda or Miniconda virtual environment since it provides an easier method of installation for GDAL, cython, and cytoolz. Otherwise, information for installing GDAL manually can be found here.


Create a Virtual Environment

  • Install Anaconda or Miniconda
  • Create a Conda virtual environment
  • Linux

    conda env create --name tap python=3.6.7 gdal=2.2.2 poppler=0.60.1 cython cytoolz
    
  • Windows

    conda env create --name tap python=3.6.7 gdal=2.2.2 cython cytoolz
    
  • If you experience SSL cert errors, ssl cert verification can be temporarily disabled for conda via:

    conda config --set ssl_verify false
    
  • Activate the environment

    conda activate tap
    

Install TAP

  • Ensure the virtual environment is activated from previous creation steps.

  • Use pip to install TAP and the remaining dependencies

    pip install lcmap-tap
    
  • If you are having trusted host/certificate issues during pip install, you can explicitly trust pypi

    pip install lcmap-tap --trusted-host python.pypi.org --trusted-host pypi.org --trusted-host files.pythonhosted.org
    
  • Create the configuration file required by tap

  • Linux Example

    • Create: “$HOME/.conda/envs/tap/lib/python3.6/site-packages/lcmap_tap/config.yaml”

      URL: http://lcmap-test.cr.usgs.gov/ARD_CU_C01_V01
      CCD: /lcmap_data/bulk/tiles
      
  • Windows Example

    • Create: “$HOME\AppData\Local\Continuum\miniconda3\envs\tap\Lib\site-packages\lcmap_tap\config.yaml”
    URL: http://lcmap-test.cr.usgs.gov/ARD_CU_C01_V01
    CCD: Z:\bulk\tiles
    

Run the Tool

Once installed, lcmap_tap can be executed directly from the command line if the virtual environment is activated:

  • Linux
lcmap_tap
  • Windows
python.exe $HOME\AppData\Local\Continuum\miniconda3\envs\tap\Scripts\lcmap_tap.exe

Packaging

Packaging tap-tool using PyInstaller for distribution of an executable binary.

Pre-Reqs

  • The following instructions assume the above Installation steps have already been completed.
  • Conda virtual environment is activated
  • Download a ZIP archive of the TAP source code for the target tag/branch.
  • Extract the zipped folder.
  • From the command line, cd into the extracted folder.
  • Current working directory is inside the extracted lcmap-tap ZIP archive

Instructions

  • Install the most recent version of PyInstaller with pip directly from github:

    pip install --trusted-host python.pypi.org --trusted-host pypi.org --trusted-host files.pythonhosted.org --trusted-host github.com --trusted-host codeload.github.com https://github.com/pyinstaller/pyinstaller/archive/develop.zip
    
  • Create a new file - a short script that calls lcmap-tap, save as “run_lcmap_tap.py”

    from lcmap_tap.__main__ import main
    main()
    
  • Freeze the script – this will recursively search all imported modules in lcmap_tap to include in the build environment:

    pyinstaller run_lcmap_tap.py
    
  • Copy the file “config.yaml” (created in previous Installation section) from the conda environment into the downloaded/extracted lcmap-tap top level directory (should contain setup.py).

  • Once complete, open the run_lcmap_tap.spec file to edit.

  • We need to tell PyInstaller to include certain non-python data files. Add the following to the ‘datas’ list.

    datas=[( '.\\config.yaml', 'lcmap_tap'),
           ('.\\lcmap_tap\\Auxiliary\\icon.png', 'lcmap_tap/Auxiliary'),
           ('.\\lcmap_tap\\MapCanvas\\UseWebEngineView', 'lcmap_tap/MapCanvas/UseWebEngineView'),
           ('.\\lcmap_tap\\MapCanvas\\UseWebView', 'lcmap_tap/MapCanvas/UseWebView')],
    
    • Note: the spec file is read as a python file. Datas is a list of tuples, each tuple contains two strings, the first is the path to the data file or folder, the second is the destination for the data file in the build module.
  • Update the hidden imports in the run_lcmap_tap.spec file to include:

    hiddenimports=['PyQt5.sip', 'gdal', 'ogr', 'osr', 'merlin'],
    
    • Note: Doing this seems to fix an issue with not being able to import gdal later on. I think it also helps avoid errors with importing these other modules as well.
  • Re-run PyInstaller, but point to the spec file rather than the script file

    pyinstaller run_lcmap_tap.spec
    
  • The resulting ‘dist’ folder contains the stand-alone TAP tool and this is what should be distributed on to users’ systems.

    • Zip up this directory for distribution: dist/run_lcmap_tap
  • Run TAP simply by double-clicking the executable file, or a shortcut that points to this executable:

    dist/run_lcmap_tap/run_lcmap_tap.exe
    

Project details


Release history Release notifications

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for lcmap-tap, version 1.2.1
Filename, size File type Python version Upload date Hashes
Filename, size lcmap_tap-1.2.1-py3-none-any.whl (112.4 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size lcmap_tap-1.2.1.tar.gz (95.6 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page