eniccs 0.2.4

EnMAP Improved Cloud and Cloud Shadow (EnICCS) masking pipeline

EnICCS is a tool for generating improved cloud and cloud shadow masks for EnMAP L2A scenes over land surfaces.

For details please refer to the accompanying paper.

Table of Contents

• Background
• About EnICCS
• Example
• Installation
• Usage
• Note on Transferability
• Customization
• Contributions
• Planned Features
• EnMAP Cloud and Cloud Shadow Benchmarking Dataset
• Citation
• Funding and Acknowledgements
• License

Background

Current operational cloud and cloud shadow masks often fail to detect small clouds and cloud shadows and lack proper cloud boundary delineation. Residual clouds and cloud shadows can significantly distort spectral signatures, such that the recorded signal no longer corresponds to the expected ‘clear-sky’ measurement of the observed surface. This, in turn, compromises any downstream analysis.

About EnICCS

EnICCS intends to improve existing cloud and cloud shadow masks through a series of steps.

1. Operational masks are refined using a combination of narrowband indices, thresholding and masking.
2. The refined masks are then used to fit a simple PLS-DA model to classify the respective scene.
3. Predictions are post-processed with a simple cloud-to-shadow matching routine.

The overall workflow is illustrated below:

EnICCS is simple to use with a single function call and requires only the directory path of the EnMAP L2A data as input.

Various parameters can be adjusted to improve performance while implementation level changes can help adapt the tool to different regions and surface types. For more details see the paper and customization section.

Example

Some exemplary pairs of EnMAP images with operational (left) and EnICCS masks (right) respectively:

Installation

You can install EnICCS from GitHub or using pip:

pip install eniccs
# or
pip install git+https://github.com/leleist/eniccs.git

Usage

To use EnICCS, you can import it and run the main wrapper function with default parameters.
Just provide the directory path of the EnMAP L2A data on a tile-by-tile basis, i.e., as provided by DLR.

from eniccs import run_eniccs

dir_path = r"path/to/your/EnMAP/TIFFS"  

# simple call with default parameters
run_eniccs(dir_path)

# new cloud and cloudshadow masks will be saved to "dir_path"

Please note: EnICCS currently accepts two file types, TIFF and BSQ, with extensions (.TIFF, .TIF, .tiff, .tif, .BSQ, .bsq)

Note on Transferability:

EnICCS was developed and tested on EnMAP scenes over tropical western Kenya. Application to regions with differing surface characteristics may require adjustments. The code structure allows for some optimization with available parameters and simple adaptation, leveraging expert knowledge and/or visual inspection. For more details see the accompanying paper and Customization.

Customization

EnICCS has two points of contact for customization:

1. Parameter Adjustments

The following parameters can be adjusted when calling the run_eniccs function:

run_eniccs(
     dir_path: str,                     # path to EnMAP L2A data
     save_output: bool = True,          # save output masks
     return_mask_obj: bool = False,     # return mask object
     auto_optimize: bool = False,       # optimize the number of latent variables for PLS-DA automatically
     verbose: bool = False,             # print progress messages
     plot: bool = False,                # plot informative plots
     smooth_output: bool = True,        # apply conservative morphological processing for smooting the output masks
     contamination: float = 0.25,       # contamination parameter for LOF outlier detection
     percentile: int = 85,              # percentile for cloud-to-shadow matching routine distance threshold
     num_samples: int = 3000,           # number of samples for PLS-DA training
     buffer_size: int = 1,              # Buffer size for dilation of CCS mask outputs.
     n_jobs: int = -1,                  # number of parallel jobs (CPU)
     random_state: int = 42,            # random state for class balancing and data splitting
     output_dir: str = None,            # alterbative output directory for saving the new masks (if None, saves to dir_path)
)

2. Implementation Level Changes

The main module contains functions for mask refinement i.e. improve_cloud_mask_over_land and improve_cloud_shadow_mask as well as a wrapper refine_ccs_masks that integrates the two prior functions into the overall workflow. These functions can be modified. Here, bands, indices and thresholds can be changed to suit the surfaces of interest. Classification and post-processing steps can stay untouched in this scenario.

In the context of supervised ML classification, we recall the garbage-in-garbage-out principle. Thus, the quality of the refined masks used for training is crucial for the performance of the PLS-DA model, despite the available post-processing steps.

Contributions

Contributions are welcome!
Specifically regarding band indices and thresholds for different surface types (Desert, Snow, Urban).

Planned Features

• "no reference data" mode for using EnICCS as a standalone cloud masking tool, without existing operational masks.
• Selectable surface-type presets (e.g., tropical, desert, snow, urban) with band indices and thresholds for mask refinement.

EnMAP Cloud and Cloud Shadow Benchmarking Dataset

We provide the hand-drawn "gold standard" reference masks for five EnMAP scenes used in the accompanying paper as a benchmarking dataset on Zenodo 10.5281/zenodo.17350339.
For tile-wise performance metrics, please consult the paper supplemental information.

Citation

Please cite the accompanying paper and Zenodo sources:

Leander Leist, Boris Thies, Jörg Bendix, Evaluation and improvement of EnMAP’s cloud and cloud-shadow masks – An application in tropical western Kenya, International Journal of Applied Earth Observation and Geoinformation, Volume 144, 2025, 104914, ISSN 1569-8432, https://doi.org/10.1016/j.jag.2025.104914, https://www.sciencedirect.com/science/article/pii/S1569843225005618.

Funding and Acknowledgements

This work was funded by the German Space Agency at DLR via the German Federal Ministry of Economic Affairs and Climate Action under Grant 50EE2303A.

Illustrations contain EnMAP data and modified EnMAP data © DLR [2023, 2024]. All rights reserved.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.