nrt 0.3.0

Online monitoring with xarray

Python package for near real time detection of change in spatio-temporal datasets

nrt provides a standardized interface for Near Real Time monitoring of disturbances on satellite image time-series. The package is optimized for fast computation and suitable for operational deployment at scale. A typical operational use case of such package would be a system constantly receiving new satellite based acquisitions and generating alerts when an anomaly is detected. Five monitoring frameworks from scientific literature on change detection are implemented and exposed via a common API. All five monitoring framework share a common general approach which consists in modelling the “normal” behavior of the variable through time by fitting a linear model on a user defined stable history period and monitoring until a “break” is detected. Monitoring starts right after the stable history period, and for each new incoming observation the observed value is compared to the predicted “normal” behavior. When observations and predictions diverge, a “break” is detected. A confirmed “break” typically requires several successive diverging observations, this sensitivity or rapid detection capacity depending on many variables such as the algorithm, its fitting and monitoring parameters, the noise level of the history period or the magnitude of the divergence. The five monitoring frameworks implemented are:

• Exponentially Weighted Moving Average (EWMA) (Brooks et al., 2013)

• Cumulative Sum of Residual (CuSum) (Verbesselt et al., 2012; Zeileis et al., 2005). CuSum is one of the monitoring option of the bfastmonitor function available in the R package bfast.

• Moving Sum of Residuals (MoSum) (Verbesselt et al., 2012; Zeileis et al., 2005). MoSum is one of the monitoring option of the bfastmonitor function available in the R package bfast.

• Continuous Change Detection and Classification of land cover (CCDC, CMFDA) (Zhu et al., 2012, 2014) - Partial implementation only of the original published method.

• InterQuantile Range (IQR) - Simple, unpublished outlier identification strategy described on stackexchange.

Parts of this package are derived from Chris Holden’s pybreakpoints and yatsm packages. Please see the copyright statements in the respective modules.

Documentation

Learn more about nrt in its official documentation at https://nrt.readthedocs.io/en/latest/

Installation

pip install nrt

The main dependencies, which should be automatically resolved by pip, are:

• numpy

• scipy

• xarray

• numba

• rasterio

• netCDF4

Example usage

The snippet below presents a near real time monitoring simulation. The input data is split in stable history and monitoring period; the monitoring class is instantiated (EWMA algorithm), a simple harmonic model is fitted on the history period, and new acquisition are passed to the monitor method one at the time. Note that in a real operational scenario where new observations come at a less frequent interval (e.g. every 5 or 8 days which coorespond to the revisit frequency of sentinel 2 and Landsat constellations respectively), the monitoring state can be saved on disk and reloaded when required.

import datetime

from nrt.monitor.ewma import EWMA
from nrt import data

# Forest/non-forest mask
mask = (data.romania_forest_cover_percentage() > 30).astype('int')

# NDVI training and monitoring periods
s2_cube = data.romania_20m()
s2_cube['ndvi'] = (s2_cube.B8A - s2_cube.B04) / (s2_cube.B8A + s2_cube.B04)
s2_cube = s2_cube.where(s2_cube.SCL.isin([4,5,7]))
ndvi_history = s2_cube.ndvi.sel(time=slice('2015-01-01', '2018-12-31'))
ndvi_monitoring = s2_cube.ndvi.sel(time=slice('2019-01-01', '2021-12-31'))

# Instantiate monitoring class and fit stable history
EwmaMonitor = EWMA(trend=False, mask=mask)
EwmaMonitor.fit(dataarray=ndvi_history)

# Monitor new observations
for array, date in zip(ndvi_monitoring.values,
                       ndvi_monitoring.time.values.astype('M8[s]').astype(datetime.datetime)):
    EwmaMonitor.monitor(array=array, date=date)

# At any time a monitoring report can be produced with EwmaMonitor.report(filename)
# and state of the monitoring instance can be saved as netcdf with
# EwmaMonitor.to_netcdf(filename)

Contributing

Any type of contribution is welcome. Please see the contributing guidelines at CONTRIBUTING.md.

Citing nrt

If you use nrt in your research or project, please consider citing it using the following BibTeX entry.

@article{dutrieux2024nrt,
  year = {2024},
  publisher = {The Open Journal},
  volume = {9},
  number = {100},
  pages = {6815},
  author = {Lo\"{i}c Dutrieux and Jonas Viehweger},
  title = {nrt: operational monitoring of satellite image time-series in Python},
  journal = {Journal of Open Source Software},
  doi = {10.21105/joss.06815},
}

About the authors

Loïc Dutrieux works as a remote sensing researcher at the Joint Research Center (JRC) in Ispra, Italy. His work focuses on forest disturbances mapping and characterization from satellite image time-series.

Jonas Viehweger is a young researcher with a MSc in remote sensing from the university of Marburg, Germany. He developped a large part of the nrt package during his traineeship period at the Joint Research Center (JRC) in Ispra, Italy.

Chris Holden implemented many time-series change detection algorithms in python during his PhD at Boston university.

References

Brooks, E.B., Wynne, R.H., Thomas, V.A., Blinn, C.E. and Coulston, J.W., 2013. On-the-fly massively multitemporal change detection using statistical quality control charts and Landsat data. IEEE Transactions on Geoscience and Remote Sensing, 52(6), pp.3316-3332. https://doi.org/10.1109/TGRS.2013.2272545

Verbesselt, J., Zeileis, A. and Herold, M., 2012. Near real-time disturbance detection using satellite image time series. Remote Sensing of Environment, 123, pp.98-108. https://doi.org/10.1016/j.rse.2012.02.022

Zeileis, A., Leisch, F., Kleiber, C. and Hornik, K., 2005. Monitoring structural change in dynamic econometric models. Journal of Applied Econometrics, 20(1), pp.99-121. https://doi.org/10.1002/jae.776

Zhu, Z., Woodcock, C.E. and Olofsson, P., 2012. Continuous monitoring of forest disturbance using all available Landsat imagery. Remote sensing of environment, 122, pp.75-91. https://doi.org/10.1016/j.rse.2011.10.030

Zhu, Z. and Woodcock, C.E., 2014. Continuous change detection and classification of land cover using all available Landsat data. Remote sensing of Environment, 144, pp.152-171. https://doi.org/10.1016/j.rse.2014.01.011