`check-distribution`

A Python utility for analyzing and logging the statistical distribution of values in arrays and raster images during model runs. Particularly useful for debugging geospatial data processing pipelines and monitoring intermediate variables in scientific computing workflows.

Author

Gregory H. Halverson (they/them)
gregory.h.halverson@jpl.nasa.gov
NASA Jet Propulsion Laboratory 329G

Features

📊 Statistical Analysis: Automatically computes min, max, mean, and NaN proportion
🎨 Color-Coded Logging: Uses colored output to highlight warnings and important values
🗺️ Raster Support: Native support for Raster objects from the rasters package
⚠️ Blank Detection: Optionally raises errors when outputs are completely blank
📈 Smart Output: Detailed distribution for categorical data, statistics for continuous data
🕒 Temporal Tracking: Optional date and location parameters for tracking values over time

Installation

From PyPI

pip install check-distribution

From Source

git clone https://github.com/JPL-Evapotranspiration-Algorithms/check-distribution.git
cd check-distribution
pip install -e .

Development Installation

pip install -e ".[dev]"

Quick Start

import numpy as np
from check_distribution import check_distribution

# Analyze a simple array
data = np.random.rand(100, 100)
check_distribution(data, "temperature")

Detailed Usage

Basic Usage

The check_distribution function analyzes array or raster data and logs statistical information:

from check_distribution import check_distribution
import numpy as np

# Create sample data
temperature = np.random.normal(25, 5, (100, 100))

# Basic check
check_distribution(temperature, "temperature")

Output:

variable temperature min: 10.234 mean: 25.123 max: 39.876 nan: 0.00% (nan)

Working with Categorical Data

For arrays with fewer than 10 unique values, the function displays counts for each value:

# Land cover classification
land_cover = np.array([
    [1, 1, 2, 3],
    [1, 2, 2, 3],
    [2, 2, 3, 3],
    [3, 3, 3, 0]
])

check_distribution(land_cover, "land_cover")

Output:

variable land_cover (int64) with 4 unique values
* 0: 1
* 1: 3
* 2: 5
* 3: 7

Adding Temporal Context

Track variable distributions over time:

from datetime import date

# Daily temperature data
for day in range(1, 8):
    temp = np.random.normal(20 + day, 3, (100, 100))
    check_distribution(
        temp, 
        "temperature",
        date_UTC=date(2025, 1, day)
    )

Output:

variable temperature on 2025-01-01 min: 15.234 mean: 21.123 max: 27.876 nan: 0.00% (nan)
variable temperature on 2025-01-02 min: 16.123 mean: 22.456 max: 28.234 nan: 0.00% (nan)
...

Adding Spatial Context

Include location information for spatial data:

# Process multiple tiles
tiles = ["h08v05", "h09v05", "h08v06"]

for tile in tiles:
    ndvi = np.random.rand(2400, 2400)
    check_distribution(
        ndvi,
        "NDVI",
        date_UTC=date(2025, 6, 15),
        target=tile
    )

Output:

variable NDVI on 2025-06-15 at h08v05 min: 0.001 mean: 0.512 max: 0.999 nan: 0.00% (nan)
variable NDVI on 2025-06-15 at h09v05 min: 0.003 mean: 0.498 max: 0.997 nan: 2.34% (nan)
...

Working with Raster Objects

The function natively supports Raster objects from the rasters package:

from rasters import Raster

# Load a raster
dem = Raster("elevation.tif")

# Check distribution (automatically uses raster's nodata value)
check_distribution(dem, "elevation")

Detecting Blank Outputs

By default, the function raises BlankOutputError if an array is completely NaN:

from check_distribution import check_distribution, BlankOutputError

# This will raise an error
try:
    blank_data = np.full((100, 100), np.nan)
    check_distribution(blank_data, "missing_data")
except BlankOutputError as e:
    print(f"Error: {e}")

Output:

Error: variable missing_data is a blank image

To allow blank outputs:

blank_data = np.full((100, 100), np.nan)
check_distribution(blank_data, "missing_data", allow_blank=True)

Handling Arrays with NaN Values

The function automatically detects and reports NaN proportions:

# Create data with NaN values
data = np.random.rand(100, 100)
data[data < 0.2] = np.nan

check_distribution(data, "partial_data")

Output:

variable partial_data min: 0.201 mean: 0.612 max: 0.999 nan: 20.15% (nan)

High NaN proportions (>50%) are highlighted in yellow, and 100% NaN triggers a red warning.

Detecting All-Zero Arrays

Arrays with all zeros are automatically flagged:

zeros = np.zeros((100, 100))
check_distribution(zeros, "empty_result", allow_blank=True)

Output (warning):

variable empty_result all zeros min: 0.000 mean: 0.000 max: 0.000 nan: 0.00% (nan)

Complete Example: Processing Pipeline

import numpy as np
from datetime import date
from check_distribution import check_distribution, BlankOutputError

def process_satellite_image(tile_id, acquisition_date):
    """Example processing pipeline with distribution checks."""
    
    # Load raw data
    raw_dn = np.random.randint(0, 16384, (2400, 2400))
    check_distribution(raw_dn, "raw_DN", acquisition_date, tile_id)
    
    # Convert to reflectance
    reflectance = raw_dn * 0.0001
    check_distribution(reflectance, "reflectance", acquisition_date, tile_id)
    
    # Calculate NDVI
    nir = reflectance * np.random.uniform(0.8, 1.2, reflectance.shape)
    red = reflectance * np.random.uniform(0.1, 0.3, reflectance.shape)
    ndvi = (nir - red) / (nir + red + 1e-10)
    
    try:
        check_distribution(ndvi, "NDVI", acquisition_date, tile_id)
    except BlankOutputError:
        print(f"Warning: NDVI calculation failed for {tile_id}")
        return None
    
    # Apply cloud mask
    cloud_mask = np.random.choice([0, 1], reflectance.shape, p=[0.15, 0.85])
    check_distribution(cloud_mask, "cloud_mask", acquisition_date, tile_id)
    
    ndvi_masked = ndvi.copy()
    ndvi_masked[cloud_mask == 1] = np.nan
    check_distribution(ndvi_masked, "NDVI_masked", acquisition_date, tile_id)
    
    return ndvi_masked

# Run pipeline
result = process_satellite_image("h08v05", date(2025, 6, 15))

API Reference