deep-data-package 1.0.0

Hyperion and AVIRIS: file reader, multiple band indici calculator, and interactive mapper

################################################################################

Copyright (C) 2019 Team Deep-Data, Colorado School of Mines

Date created: 1/22/19

Date last updated:

Creators:

- Sean Smith [e-mail]

- Tyler Murphy [e-mail]

- Tyler Blount [e-mail]

- Sydney Nelson [e-mail]

- Josh Grego [e-mail]

- Jake Steiner [e-mail]

- Marcelo Gonzales [e-mail]

- Daniel Hanuszczak [e-mail]

################################################################################

Included below are file descriptions for all files in deep_data_package module along will descriptions of functionality.

################################################################################

file_reader.py

################################################################################

Module for reading AVIRIS flight data and mapping

OVERVIEW =====================================================================

Classes ====================================================================
    hyperion_band_reader
    aviris_band_reader
    pickled_aviris_band_reader

Functions ==================================================================
    select_closest_band
    create_pickled_aviris_flight
    get_available_aviris_flights
    get_available_pickled_aviris_flights
    get_available_hyperion_passes

CLASS hyperion_band_reader ===================================================

  Class for reading Hyperion data from GEOtiff L1 files

  NOTE: bands 1-70 and VNIR, bands 71-242 are SWIR

Reference ==================================================================
    https://archive.usgs.gov/archive/sites/eo1.usgs.gov/EO1userguidev2pt320030715UC.pdf
    https://crisp.nus.edu.sg/~research/tutorial/eo1.htm
    https://crisp.nus.edu.sg/~research/tutorial/eo1.htm

Inputs =====================================================================
    Optional:
        hyperion_pass - name of Hyperion pass to load (Default EO1H0410362008289110KF)
        scale_factor - number to divide bands by (Defaul 80 to get radiance values)

Returns ====================================================================
    class to load radiance bands as np arrays

class Attributes ===========================================================
    band_reader.hyperion_pass - Hyperion flight loaded
    band_reader.scale_factor - number to divide bands by (Defaul 80 to get radiance values)
    band_reader.mode - self identified mode for reference ('hyperion')
    band_reader.approx_band_centers - string refence to what band centers correspond to what files in Hyperion GEOtif files
    self.swir_actual_band_centers - caluclated band centers for SWIR bands (for reference)


METHODS ====================================================================

    load_band ==============================================================
        By defualt loads Hyperion pass EO1H0410362008289110KF over sunshine canyon
        landfill

        Reference ==========================================================
            N/A

        Inputs =============================================================
            band_req - band wavelength to load
            adjust_for_gain - N/A

        Returns ============================================================
            2D np array of radiance data for band wavelength called

            - To load a row

                band_array[row_number, :]

            - To load a column

                band_array[:, col_number]

            - To load a pixel

                band_array[row_number, col_number]


    load_pixel =============================================================
        Function for reading all bands of an individual pixel from AVIRIS
        flight data

        Reference ==========================================================
            N/A

        Inputs =============================================================
            row - int of row number of requested pixel
            col - int of column number of requested pixel
            adjust_for_gain - N/A

        Returns ==========================================================
            Ordered dict of wavelengths vs irradiance for requested pixel

CLASS aviris_band_reader =====================================================

  Class for reading AVIRIS flight data from uncompressed AVIRIS flight
  folder. No files need to be modified after downloading from AVIRIS data
  portal. Just uncompress in the aviris_flight_folder and go.

  By defualt loads AVIRIS flight f111115t01p00r08 over sunshine canyon
  landfill

Reference ==============================================================
    N/A

Inputs =================================================================
    Optional:
        aviris_flight_number - AVIRIS flight number to load
        aviris_filename - filename to load in AVIRIS flight number folder
        samples - from hdr file in AVERIS flight folder (Defualt 780)
        lines - from hdr file in AVERIS flight folder (Defualt 8355)
        bands - from hdr file in AVERIS flight folder (Defualt 224)
        aviris_calibration_filename - name of AVIRIS file for reflectance calibration
        aviris_gain_filename - name of AVIRIS gain file for reflectance calibration

Returns ================================================================
    class to load radiance bands as np arrays

class Attributes =======================================================
    band_reader.aviris_flight -  AVIRIS flight loaded
    band_reader.n_cols - number of columns in AVIRIS band file (equals input samples)
    band_reader.n_rows - number of rows in AVIRIS band file (equals input lines)
    band_reader.n_bands - number of bands in AVIRIS band file (equals input bands)
    band_reader.scale_factor - multiplier of bands (equals input scale_factor)
    band_reader.aviris_flight_folder_filepath - full filepath for AVIRIS flight folder
    band_reader.bands_filepath - full filepath for AVIRIS band file

    band_reader.info - dict with useful info from AVIRIS flight
        Contains:
            # General info
            ['info']['n_rows'] = number of rows in .npy files
            ['info']['n_cols'] = number of columns in .npy files
            ['info']['n_bands'] = number of bands from AVIRIS flight
            ['info']['AVIRIS_flight'] = aviris_flight

            # Band info
            ['band_calibration']['band_quantity'] = 'Wavelength'
            ['band_calibration']['band_unit'] = 'nm'
            ['band_calibration']['centers'] = mathmatical center of wavelength
            ['band_calibration']['bandwidths'] = bandwidths of wavelengths
            ['band_calibration']['bands'] = wavelengths callable

            # Gain values
            ['gain_values'] = np array of gain values for each band

        Call any of above w/ band_reader.info['INSERT_AS_SHOWN_ABOVE']

METHODS ====================================================================

    parse_aviris_flight ====================================================
        Function for loading AVIRIS flight files of the first time
        By defualt loads AVIRIS flight f111115t01p00r08 over sunshine canyon landfill

        Inputs =============================================================
            info_file_name - name for info file to be written by parser
            aviris_calibration_filename - name of AVIRIS file for reflectance calibration
            aviris_gain_filename - name of AVIRIS gain file for reflectance calibration

        Returns ============================================================
            flight_dict - dict with useful info from AVIRIS flight
                Contains:
                    # General info
                    ['info']['n_rows'] = number of rows in .npy files
                    ['info']['n_cols'] = number of columns in .npy files
                    ['info']['n_bands'] = number of bands from AVIRIS flight
                    ['info']['AVIRIS_flight'] = aviris_flight

                    # Band info
                    ['band_calibration']['band_quantity'] = 'Wavelength'
                    ['band_calibration']['band_unit'] = 'nm'
                    ['band_calibration']['centers'] = mathmatical center of wavelength
                    ['band_calibration']['bandwidths'] = bandwidths of wavelengths
                    ['band_calibration']['bands'] = wavelengths callable

                    # Gain values
                    ['gain_values'] = np array of gain values for each band


    get_memmap =============================================================
        Function for loading memmap of AVIRIS band file

        Inputs =============================================================
            N/A

        Returns ============================================================
            np.memmap of AVIRIS band file


    load_band ==============================================================
        Function for reading AVIRIS flight data from .npy files pickled by
        create_pickled_aviris_flight()

        By defualt loads AVIRIS flight f111115t01p00r08 over sunshine canyon
        landfill

        Reference ==========================================================
            N/A

        Inputs =============================================================
            band_req - band wavelength to load
            adjust_for_gain - If want to divide band array by gain value in .gain file (Default: False)

        Returns ============================================================
            2D np array of radiance data for band wavelength called

            - To load a row

                band_array[row_number, :]

            - To load a column

                band_array[:, col_number]

            - To load a pixel

                band_array[row_number, col_number]


    load_pixel =============================================================
        Function for reading all bands of an individual pixel from AVIRIS
        flight data

        Reference ==========================================================
            N/A

        Inputs =============================================================
            row - int of row number of requested pixel
            col - int of column number of requested pixel
            adjust_for_gain - N/A

        Returns ============================================================
            Ordered dict of wavelengths vs irradience for requested pixel

CLASS pickled_aviris_band_reader =============================================

  Class for reading AVIRIS flight data from .npy files pickled by
  create_pickled_aviris_flight()

  By defualt loads AVIRIS flight f111115t01p00r08 over sunshine canyon
  landfill

Reference ==============================================================
    N/A

Inputs =================================================================
    Optional:
        aviris_flight - AVIRIS flight to load

Returns ================================================================
    class to load radiance bands as np arrays

class Attributes =======================================================
    band_reader.aviris_flight -  AVIRIS flight loaded
    band_reader.np_path - path will load .npy files from

    band_reader.info - dict with useful info from AVIRIS flight
        Contains:
            # General info
            ['info']['n_rows'] = number of rows in .npy files
            ['info']['n_cols'] = number of columns in .npy files
            ['info']['n_bands'] = number of bands from AVIRIS flight
            ['info']['AVIRIS_flight'] = aviris_flight

            # Band info
            ['band_calibration']['band_quantity'] = 'Wavelength'
            ['band_calibration']['band_unit'] = 'nm'
            ['band_calibration']['centers'] = mathmatical center of wavelength
            ['band_calibration']['bandwidths'] = bandwidths of wavelengths
            ['band_calibration']['bands'] = wavelengths callable

            # Gain values
            ['gain_values'] = np array of gain values for each band

        Call any of above w/ band_reader.info['INSERT_AS_SHOWN_ABOVE']

METHODS ====================================================================

    load_band ==============================================================
        Function for reading AVIRIS flight data from .npy files pickled by
        create_pickled_aviris_flight()

        Reference ==========================================================
            N/A

        Inputs =============================================================
            band_req - band wavelength to load

        Returns ============================================================
            2D np array of radiance data for band wavelength called

            - To load a row

                band_array[row_number, :]

            - To load a column

                band_array[:, col_number]

            - To load a pixel

                band_array[row_number, col_number]


    load_pixel =============================================================
        Function for reading all bands of an individual pixel from AVIRIS
        flight data from .npy files pickled by create_pickled_aviris_flight()

        Reference ==========================================================
            N/A

        Inputs =============================================================
            row - int of row number of requested pixel
            col - int of column number of requested pixel
            adjust_for_gain - N/A

        Returns ============================================================
            Ordered dict of wavelengths vs irradience for requested pixel

FUNCTION select_closest_band =================================================

Returns closest band available to band_req.

Inputs =============================================================
    band_req - band wavelength to match
    band_req - list of band wavelengths available

Returns ============================================================
    band available, if two numbers are equally close, returns the
    smallest number.

FUNCTION create_pickled_aviris_flight ========================================

Function for creating pickled AVIRIS flight data using spectral python library

By defualt loads AVIRIS flight f111115t01p00r08 over sunshine canyon landfill

Reference ==============================================================
    N/A

Inputs =================================================================
    Optional:
        aviris_flight - AVIRIS flight number
        aviris_flight_folder - AVIRIS flight folder to load
        aviris_bands_filename - filename to load in AVIRIS flight number folder
        samples - from hdr file in AVERIS flight folder (Defualt 780)
        lines - from hdr file in AVERIS flight folder (Defualt 8355)
        bands - from hdr file in AVERIS flight folder (Defualt 224)
        aviris_calibration_filename - name of AVIRIS file for reflectance calibration
        aviris_gain_filename - name of AVIRIS gain file for reflectance calibration
        adjust_for_gain - if should devide bands by associated gain value to true values (Default False)
        scale_factor - if want to dived all values by some value (Default 10000.0)

Returns ================================================================
    N/A

FUNCTION get_available_aviris_flights ========================================

Function listing all AVIRIS flights available

Inputs =================================================================
    N/A

Returns ================================================================
    List of AVIRIS flights available

FUNCTION get_available_pickled_aviris_flights ================================

Function listing all pickled AVIRIS flights available

Inputs =================================================================
    N/A

Returns ================================================================
    List of pickled AVIRIS flights available

FUNCTION get_available_hyperion_passes =======================================

Function listing all Hyperion passes available

Inputs =================================================================
    N/A

Returns ================================================================
    List of Hyperion passes available

################################################################################

################################################################################

mapper.py

################################################################################

Module for mapping band ratio output arrays

OVERVIEW =====================================================================

Functions ==================================================================
    create_heatmap_interactive

FUNCTION create_heatmap_interactive ==========================================

Function for creating an interactive heatmap

Reference ==============================================================
    N/A

Inputs =================================================================
    band_reader - band_reader class instance
    bandratio_array - bandratio result array for heatmap
    bandratio_array_name - name of bandratio calculated
    flight_name - name of the flight
    alpha - transparency value of overlay (Defualt 0.6)
    gamma_1 - threshold value for map (Default mean of normalized bandratio_array)
    interval - step value of threshold slider (Default 0.00001)
    std_multiplier - multiplier of std for range of values of threshold slider (Default 3)
    clipped - if want clipped heatmap instead of default treshhold (Default False)
    display - show plt (Default True)
    save - save plt when (Default False)
    cmap - color scheme for heatmap (Default 'hot' for threshold, 'jet' for clipped)
    quiet - if should not log info about bandratio_array (Defualt False)
    rgb_scale_value - float value to increase brightness of rgb image

Returns ================================================================
    N/A

################################################################################

################################################################################

band_ratios.py

################################################################################

Module for calculating band ratios

OVERVIEW =====================================================================

Functions ==================================================================
  Deep_Data
  HI_1453
  HI_1215
  HI_1732
  GM
  NDVI
  NDPI

FUNCTION Deep_Data ===========================================================

Deep-Data Hydrocarbon Index ================================================
    Convert numpy arrays of radiance data to a Deep-Data hydrocarbon
    index map

Inputs =====================================================================
    L_1197 : At-sensor radiance at 1197 nm wavelength (numpy array)
    L_1216 : At-sensor radiance at 1216 nm wavelength (numpy array)
    L_1235 : At-sensor radiance at 1235 nm wavelength (numpy array)
    L_1373 : At-sensor radiance at 1373 nm wavelength (numpy array)
    L_1453 : At-sensor radiance at 1453 nm wavelength (numpy array)
    L_1503 : At-sensor radiance at 1503 nm wavelength (numpy array)

Returns ====================================================================
    Deep-Data Hydrocarbon Index Map

FUNCTION HI_1453 =============================================================

1453 Hydrocarbon Index =====================================================
    Convert numpy arrays of radiance data to a 1453 hydrocarbon index map

Inputs =====================================================================
    L_1373 : At-sensor radiance at 1373 nm wavelength (numpy array)
    L_1453 : At-sensor radiance at 1453 nm wavelength (numpy array)
    L_1503 : At-sensor radiance at 1503 nm wavelength (numpy array)

Returns ====================================================================
    1453 Hydrocarbon Index Map

FUNCTION HI_1215 =============================================================

1215 Hydrocarbon Index =====================================================
    Convert numpy arrays of radiance data to a 1215 hydrocarbon index map

Inputs =====================================================================
    L_1197 : At-sensor radiance at 1197 nm wavelength (numpy array)
    L_1216 : At-sensor radiance at 1216 nm wavelength (numpy array)
    L_1235 : At-sensor radiance at 1235 nm wavelength (numpy array)

Returns ====================================================================
    1215 Hydrocarbon Index Map

FUNCTION HI_1732 =============================================================

1732 Hydrocarbon Index =================================================
    Convert numpy arrays of radiance data to a 1732 hydrocarbon index map

Inputs =====================================================================
    L_1705 : At-sensor radiance at 1705 nm wavelength (numpy array)
    L_1729 : At-sensor radiance at 1729 nm wavelength (numpy array)
    L_1741 : At-sensor radiance at 1741 nm wavelength (numpy array)

Returns ====================================================================
    1732 Hydrocarbon Index Map

FUNCTION GM ==================================================================

Goddijn-Murphy Index =======================================================
    From: "Proof of concept for a model of light reflectance of plastics
    floating on natural waters" by Lonneke Goddijn-Murphy and Juvenal
    Dufaur. - See Table 5

    Convert numpy arrays of radiance data to a Goddijn-Murphy map

Inputs =====================================================================
    L_850 : At-sensor radiance at 850 nm wavelength (numpy array)
    L_1660 : At-sensor radiance at 1660 nm wavelength (numpy array)

Returns ====================================================================
    Goddijn-Murphy Map

FUNCTION NDVI ================================================================

NDVI =======================================================================
    Convert numpy arrays of radiance data to a NDVI map

Inputs =====================================================================
    L_1006 : At-sensor radiance at 1006 nm wavelength (numpy array)
    L_675 : At-sensor radiance at 675 nm wavelength (numpy array)

Returns ====================================================================
    NDVI Map

FUNCTION NDPI ================================================================

NDPI =======================================================================
    Convert numpy arrays of radiance data to a NDPI map

Inputs =====================================================================
    L_1263 : At-sensor radiance at 1263 nm wavelength (numpy array)
    L_2417 : At-sensor radiance at 2417 nm wavelength (numpy array)

Returns ====================================================================
    NDPI Map

################################################################################