A Sparse Matched Filter Algorithm for Atmospheric Trace Gas Concentration Estimation
Project description
MAG1C: Matched filter with Albedo correction and reweiGhted L1 sparsity Code
Fast concentration estimation and detection of trace gas absorption from imaging spectrometer data.
Installation
pip install mag1c
Requirements
mag1c depends on these software packages for math routines and data I/O.
Python 3.6 (or newer) and the following python packages and versions:
numpy
spectral
torch
1.3+scikit-image
scipy
(optional: Only required if you want to load target spectra from.mat
files.)
Read on for a step-by-step setup guide of how to satisfy these requirements if they are not already met.
GPU Processing
If available, this code uses a compatible GPU for accelerated computation. See https://pytorch.org/get-started/locally/ for details on how to install pytorch with gpu support for your system. You will then need to install the other dependencies.
You must use the option --gpu
when running to enable GPU acceleration.
If you know that you will not use a GPU, you can install the CPU-only version of pytorch by specifying the cpuonly
package to conda in addition to pytorch
. See https://pytorch.org/get-started/locally/#no-cuda-1 for how to install the CPU-only version using pip.
Setup
The simplest way to obtain this program is through pip
.
pip install mag1c
This will install required dependencies (most notably, pytorch on linux systems. For Windows, or specific installation flavors of pytorch (like CPU-only), follow your choice of instructions on PyTorch's website, then install mag1c. Mag1c is compatible with PyTorch installed through conda
, just make sure you are using the environment's pip
to install mag1c, and activate the conda environment whenver you wish to run mag1c
.
Advanced Setup with git clone of this repository
These instructions are targeted towards *nix systems. There are analagous steps on Windows platforms.
1. Clone this repository:
git clone https://github.com/markusfoote/mag1c.git
cd mag1c
2. Select one of these sub-steps to install dependencies:
2a. pip
You can install all necessary dependencies by using the included requirements.txt file in this repository.
pip install -r requirements.txt
If you do not have administrator access on your system, you may need to request installation of the packages to a location that your user has installation privileges with the --user
flag:
pip install --user -r requirements.txt
2b. python virtual environment and pip.
Alternatively, you can satisfy the dependencies in a virtual environment that is independent of your system-wide python installation.
python3 -m venv my-env
source my-env/bin/activate
pip install -r requirement.txt
This snippet first creates a new (and empty) virtual environment, then activates that environment. The required dependencies are then installed within this environment.
To leave the virtual environment after finishing this how-to, type deactivate
.
You will need to activate this virtual environment anytime you return in a new shell with source /path/to/<your-environment-name>/bin/activate
before running the mag1c
processing algorithm.
2c. Anaconda
Python 3.6 or newer is required. If your system does not have a suitable python version, installing Anaconda is usually an acceptable and simple method to obtain a newer python version (as well as the other dependencies). Conda will create an environment and install dependencies in one step:
conda create -n myenv
conda install -n myenv pytorch cudatoolkit=10.0 spectral scipy -c defaults -c conda-forge -c pytorch
2d. Install via your package manager
If you are on a linux system, these packages may be provided by your distribution's package manager (apt
or yum
or zypper
...). This method is typically very platform specific and some packages may not be available and therefore must still be installed by one of the above methods. Ask your system administrator for assistance if you want to go this route.
3. Test your installation
Invoking the mag1c
script will attempt to load all the dependencies before parsing arguments. A simple test that everything is set up correctly is to report the version information:
python mag1c.py --version
or, if you installed mag1c
through pip:
mag1c --version
If this command reports errors, something is wrong with your setup. Check that you've followed the above instructions, then consider opening an issue or contacting the author(s).
4. Process some data!
See below in Usage and Examples for how to process your own data. Datasets are input and output in ENVI format.
Usage
This program can be invoked in multiple ways:
python /path/to/mag1c.py
works whenever you have a copy of themag1c.py
script.python -m mag1c
works when you install the python package (i.e. viapip
).mag1c
is a direct entry point to the program when you install viapip
.sparsemf
is exactly the same asmag1c
, just with a debatably-more-readable name.
Runtime Options
There are numerous options/flags that can be provided to the mag1c script. Run magic.py --help
for a full description of the available arguments.
usage: mag1c [--spec TARGET_SPEC_FILE] [--out OUTPUT_FILE] [--gpu] [-i N]
[-g N] [-b N] [-s] [-t N] [-o] [-d] [-p] [--asap] [--noalbedo]
[--no-albedo-output] [--nonnegativeoff] [--onlypositiveradiance]
[--outputgeo GLT] [--geo GLT] [--optimize] [-q] [--version]
[-T THRESHOLD] [-W LOW HIGH] [-M [RADIUS]] [-A [PX_AREA]]
[--hfdi] [--saturation-processing-block-length N] [--no-sparsity]
[--covariance-update-scaling COVARIANCE_UPDATE_SCALING]
[--covariance-lerp-alpha COVARIANCE_LERP_ALPHA]
[--use-wavelength-range MIN MAX]
[--visible-mask-growing-threshold FLOAT] [-S] [-h]
RADIANCE_FILE
M atched filter with
A lbedo correction and
rewei G hted
L 1 sparsity
C ode
University of Utah Albedo-Corrected Reweighted-L1 Matched Filter
v1.2.0.alpha1
positional arguments:
RADIANCE_FILE ENVI format radiance file to process -- Provide the
data file itself, not the header
optional arguments:
--spec TARGET_SPEC_FILE
Target spectrum file to use. If no file is specified,
a target spectrum will be calculated from an internal
methane absorption spectrum that is convolved to the
band centers and FWHMs stored in the radiance image
header.
--out OUTPUT_FILE File to write output into. Provide the data file
itself, not the header.
--gpu Use GPU for accelerated computation. (default: False)
-i N, --iter N Number of iterations to run. (default: 30)
-g N, --group N Grouping size for detectors. (default: 1)
-b N, --batch N Number of groups processed simultaneously. (default:
1)
-s, --single Use single/float32 instead of double/float64 in
computation. (default: False)
-t N, --threads N Number of additional threads for loading data.
(default: 0)
-o, --overwrite Force the output files to overwrite any existing
files. (default: False)
-d, --writedouble Write output data file as double precision instead of
single. Does not affect compute. (default: False)
-p, --noprefill Don't write NO_DATA value (-9999) to the output file
upon creation. (default: False)
--asap Force the results to be written to disk ASAP after
each batch, instead of with some additional delay from
the memory-mapped file caching. (default: False)
--noalbedo Calculate results withOUT albedo correction. (default:
False).
--no-albedo-output Do not include albedo band in result file.
--nonnegativeoff Do not apply non-negativity constraint to MF result.
(default: False)
--onlypositiveradiance
Only process pixels that have strictly non-negative
radiance for all wavelengths. This flag overrides any
batch size setting to 1.
--outputgeo GLT Use the provided GLT file to perform geocorrection to
the resulting ENVI file. This result will have the
same name as the provided output file but ending with
'_geo'.
--geo GLT If using an orthocorrected radiance file, provide the
corresponding GLT file. The output will also be
geocorrected -- Do not use with --outputgeo to apply
glt.
--optimize Use Pytorch torchscript optimizations. Experimental.
-q, --quiet Do not write status updates to console.
--version show program's version number and exit
-T THRESHOLD, --saturationthreshold THRESHOLD
specify the threshold used for classifying pixels as
saturated (default: 5.0)
-W LOW HIGH, --saturationwindow LOW HIGH
specify the wavelength window, as "-W LOW HIGH",
within which to detect saturation (default: 1945, 2485
nanometers)
-M [RADIUS], --maskgrowradius [RADIUS]
radius to use for expanding the saturation mask to
cover (and exclude) flare-related anomalies. This
value must include units: meters (abbreviated as m) or
pixels (abbreviated as px). If flag is given without a
value, 100m will be used. This is a combined flag for
enabling mask growing.
-A [PX_AREA], --mingrowarea [PX_AREA]
minimum number of pixels that must constitute a
2-connected saturation region for it to be grown by
the mask-grow-radius value. If flag is provided
without a value, 6 pixels will be assumed as the
value.
--hfdi calculate the Hyperspectral Fire Detection Index (doi:
10.1016/j.rse.2009.03.010) and append band to output.
--saturation-processing-block-length N
--no-sparsity
--covariance-update-scaling COVARIANCE_UPDATE_SCALING
--covariance-lerp-alpha COVARIANCE_LERP_ALPHA
--use-wavelength-range MIN MAX
--visible-mask-growing-threshold FLOAT
Restrict mask growing to only occur when 500 nm
radiance is less than this value.
-S, --saturation enable saturation detection (and avoid processing such
pixels). This flag overrides any batch size setting to
1. Required if any saturation related flags are
provided. (default: False)
-h, --help show this help message and exit
When using this software, please cite:
Foote et al. 2019, "Title Here" doi:xxxx.xxxx
Examples
Process a single file with defaults:
python mag1c.py /my/radiance --spec /my/target --out /some/output
Process a single file with bash variables and some custom options, including GPU:
export CUDA_VISIBLE_DEVICES=0 # Restrict processing to the first GPU in the system
RDNFILE="/path/to/my/radiance data with spaces in filename"
TEMPLATE=/path/to/my/templatespectrum
OUTPUT=/path/to/outputfile_date_time_iteration25_grouping5
GLT=/path/to/my/gltfile
python mag1c.py "$RDNFILE" --spec $TEMPLATE --out $OUTPUT --outputgeo $GLT --iter 25 --group 5 --gpu -t 2 -b16
Process all files in a folder:
TEMPLATE=/path/to/template.txt
for f in /path/to/folder/ang*_rdn_*_clip; do \
python mag1c.py "${f}" "${TEMPLATE}" "/output/folder/$(basename "${f/rdn/mag1c}")" --iter 20
done;
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.