Satellite Data Preprocessor

A package built for preprocessing netCDF, netCDF4, GeoJSON, or atomic CSV files using spatial joins and concatenation to create time-series numpy and csv datasets.

Keyword Arguments

Usage:

• Define a directory structure and create the required configuration files. (see Configuration Files)
• Create an instance of Preprocessor and provide the path to a config json file along with keyword arguments.
• Call Preprocessor.preprocess to preprocess data according to the provided config.
• To define configurations for different data sources, use Preprocessor.preprocess_multi. Config_list[i] and Kwargs_list[i] will apply to the ith data source.

import mlossp as sp

config_path = "config.json"
extra_config_path = "nc_config.json"
f = lambda : "_1"

preprocessor = sp.Preprocessor(
    config_path,
    extra_config=extra_config_path,
    suffix_out=f
)
preprocessor.preprocess()

config_list = ["config_1.json", "../configs/config_2.json"]
kwargs_list = [{"extra_config": extra_config_path, "out_out": f}, {}]
preprocessor.preprocess_multi(config_list, kwargs_list)

Configuration Files

Every category of data, i.e., data within a directory should have a json configuration file. The following keys should be defined within the configuration file:

Given the following directory structure:

- Data
    - Visualizations
    - EO
        - data1.nc
        - data2.nc
    - EO_OUT
    - Countries
        - gadm41_LKA_1.json
        - gadm41_USA_1.json
- preprocesser.py

And region and satellite data with the following features:

Region: NAME, NAME_1, NAME_2, POINT, ...
Satellite Data: longitude, latitude, precipitationCal, ...

A sample configuration would be defined as follows:

{
  "compress_to": "Data/Visualizations",
  "crs": "EPSG:4326",
  "regions_dir": "Data/Countries",
  "data_dir": "Data/EO",
  "out_dir": "Data/EO_OUT",
  "regions_file_map":  {
    "Sri Lanka": "gadm41_LKA_1.json",
    "USA": "gadm41_USA_1.json"
  },
  "selected_regions": ["Sri Lanka"],
  "join_on": "NAME_1",
  "joins": {
    "precipitationCal": ["mean", "min", "max"]
  },
  "file_extension": "*.nc",
  "chunk": 5
}

Preprocessing netCDF data

To use this package with netCDF data, a separate configuration must be provided for each data source. A sample configuration is shown below.

{
	"latVar":"latitude",
	"lonVar":"longitude",
	"is360":false,
	"extraVars": ["NDVI"]
}

Configuration Keys:

Preprocessing CSV data

To use this package to preprocess CSV data, a configuration file must be defined for each data source. CSV data provided will be aggregated across the given time period; Information across higher temporal resolutions will be lost.

The number of csv files provided will be the final number of timestamps. If you would like to atomicize your csv files, see [Formatting CSVs](#Formatting CSVs).

{
	"latVar":"Lattitude",
	"lonVar":"Longitude",
	"is360":false,
	"filter": [["Disease Name", "Dengue Fever", "e"], ["Location Name", "SRILANKA", "n"], ["Location Name", "Kalmune", "n"]],
	"extraVars": ["Cases", "Location Name"]
}

Configuration Keys:

Formatting CSVs

CSVs are inherently designed to support tabular, 2D data. Hence, storing higher-dimensional data such as tensors and ndarrays as CSVs results in inconsistent formats.

We offer mlossp.formatters as a solution to parse a (or a list of) 3D CSV files into a 3D numpy tensor, which is formatted then written into a .npy file and a .csv file that is supported by align_csv and align_npy. We also support vertical alignment of pandas dataframes and csvs through mlossp.formatters. These must be manually sliced to have an equal number of timestamps.

Citations

This package incorporates modified code from https://github.com/podaac/netcdf_to_geojson_vectors, which is licensed under Apache 2.0. You may obtain a copy of the license at: http://www.apache.org/licenses/LICENSE-2.0.