OpenEO-LIB

OpenEO-LIB is a Python library for satellite-based Earth observation workflows, including data providers, geospatial analysis, plume simulation, and reusable visualization components.

The package is designed so that reusable library functionality is kept separate from application-specific validation and research code.

Current architecture

openeolib/
├── __init__.py
├── analyzer.py          Analysis pipeline orchestration
├── animation.py         Generic scalar-field animation (GridAnimationEngine)
├── engines.py           Inference-engine interfaces and built-ins
├── eo_cache.py          Persistent scientific-data cache (ScienceCacheStore)
├── eo_grid.py           Grid regridding, Gaussian-weighted interpolation, NaN-aware Gaussian filter/derivative
├── eo_types.py          Shared data structures and type definitions
├── eo_utils.py          Geospatial and wind utilities
├── palettes.py          Domain color tables (e.g. JMA rainfall-intensity palette)
├── panels.py            Generic Matplotlib visualization panels
├── report.py            Generic figure/report composition
├── roc.py               ROC construction
├── simulator.py         Synthetic plume generation
├── theme.py             Theme definition only (Theme, DEFAULT_THEME)
├── atmosphere/
│   └── pwv_layers.py    Decompose GNSS total PWV into AGL layers from an ERA5 profile
└── providers/
    ├── base.py               BaseProvider ABC (raw Dataset retrieval)
    ├── provider.py           AnalyzerProvider / EOProvider (S2+ERA5+S5P integration)
    ├── openeo_client.py      openEO connection client
    ├── _http_utils.py        Shared retry/local-file-cache helpers
    ├── _protocols.py         Cross-provider provenance accessor (get_provenance())
    ├── sentinel2.py          Sentinel-2 L2A band retrieval
    ├── sentinel5p.py         Sentinel-5P L2 CH4 column retrieval
    ├── era5.py               ERA5 wind/pressure/temperature retrieval
    ├── msm.py                JMA MSM (mesoscale model) wind retrieval
    ├── radar_gpv_client.py   JMA nationwide composite radar GPV download
    ├── rain.py               JMA rainfall / XRAIN providers + animate_rain()
    ├── amedas.py             JMA AMeDAS surface pressure/temperature retrieval
    ├── radiosonde.py         University of Wyoming radiosonde sounding retrieval
    ├── pwv.py                GPT3 / VMF3 blind-grid PWV retrieval
    ├── unr_trop.py           Nevada Geodetic Laboratory (NGL/UNR) GNSS tropospheric products
    ├── himawari.py           Himawari-8/9 AHI cloud-top brightness temperature
    └── _s2_utils.py          Sentinel-2 coordinate/time conversion utilities

Provider responsibilities are deliberately separated. BaseProvider and AnalyzerProvider are independent interfaces with different responsibilities; AnalyzerProvider does not extend BaseProvider. BaseProvider is the minimal file/data-opening interface for providers that return an xarray.Dataset without application-level analysis. AnalyzerProvider is the higher-level interface used by EOAnalyzer and returns an ObservationBundle for a site and observation time -- a typing.Protocol (structural typing), so a provider conforms by having a matching get_bundle(site, dt) method, not by explicitly inheriting from it. EOProvider implements it for the openEO/Sentinel-2/Sentinel-5P workflow and contains the workflow-specific quality and time-alignment logic. AnalyzerProvider/ ObservationBundle is not a general-purpose provider contract -- it is EOAnalyzer's own methane/plume-detection input shape (it reads bands["B11"]/["B12"] directly), produced only by EOProvider and PlumeSimulator.

For code that wants to treat any provider's result uniformly regardless of its own return shape (xr.Dataset, a Dict, or None) -- e.g. for logging or an audit trail -- providers._protocols.get_provenance(result) extracts a normalized {source, provider_class, datetime_utc, provenance_key} dict from whichever of the three metadata conventions this package's providers happen to use (.attrs["source"]/["provider_class"], .attrs["provider"], or dict keys "backend"/"datetime_utc"). It requires no changes to any existing provider.

The visualization layer (panels.py, theme.py, report.py, animation.py) is kept domain-agnostic. Domain providers may know the schema and units of their own source data, but they do not depend on application-specific research pipelines.

Likewise, theme.py contains only the Theme data structure and DEFAULT_THEME.

Installation

pip install openeolib

Development dependencies are optional and are intended for the private test suite:

pip install -e ".[dev]"

Optional provider dependencies are grouped by feature:

pip install -e ".[provider]"
pip install -e ".[era5]"
pip install -e ".[jma]"
pip install -e ".[himawari]"

The core package supports Python 3.9 or later. Provider-specific optional dependencies may have their own Python-version requirements.

The JMA extra (openeolib[jma]) is required for MSMProvider. The module openeolib.providers.msm remains importable without the optional dependencies, but constructing MSMProvider raises a clear ImportError listing the missing dependencies and the installation command. This keeps optional JMA support from breaking the core package import.

Public visualization API

The reusable visualization API consists of five panels:

• VectorFieldPanel — arbitrary geospatial vector fields with optional scalar contours.
• BasemapPanel — an already prepared RGB image with an optional marker.
• RawBandsPanel — arbitrary grids of 2D images with a shared scale.
• ScalarMapPanel — arbitrary 2D scalar fields.
• DetectionMaskPanel — boolean, probability, or coverage masks.

Example:

import matplotlib.pyplot as plt
import numpy as np
from openeolib import ScalarMapPanel

field = np.random.default_rng(0).normal(size=(100, 100))

fig = plt.figure(figsize=(6, 5))
gs = fig.add_gridspec(1, 1)[0]
ScalarMapPanel().draw(
    fig,
    gs,
    field,
    title="Scalar field",
    axis_mode="none",
)
fig.savefig("scalar_field.png", dpi=150, bbox_inches="tight")

Vector fields

VectorFieldPanel is intentionally not ERA5-specific. It accepts latitude/longitude grids and arbitrary vector components. A caller may supply any bounding box through extent=(west, east, south, north).

from openeolib import VectorFieldPanel

panel = VectorFieldPanel(quiver_stride=3)
panel.draw(
    fig,
    gs,
    lats=lats,
    lons=lons,
    u=u,
    v=v,
    extent=(120, 150, 20, 50),
    title="Wind field",
)

There is no Japan-specific bounding-box constant in the library.

Theme

Use Theme when a caller needs to customize the visual appearance:

from openeolib import Theme

light = Theme(
    bg="#ffffff",
    panel_bg="#ffffff",
    grid_color="#cccccc",
    text_primary="#222222",
)

Theme does not contain plotting operations or application-specific labels, flags, or data extraction rules.

Generic reports

SiteReportBuilder composes caller-supplied panels into a single-site report Figure. It does not know about methane, ROC curves, quality flags, or a particular inference engine -- it only handles grid layout, the report title, and file saving.

Each entry in panels is (label, factory, height_ratio). factory is called once with the report's Theme and must return a (fig, gs) -> None draw function; any panel-specific data (the field to plot, its title, ...) is bound into that closure by the caller, not inspected by SiteReportBuilder itself.

import numpy as np
from openeolib import SiteReportBuilder, ScalarMapPanel

field = np.random.default_rng(0).normal(size=(100, 100))

report = SiteReportBuilder(
    panels=[
        (
            "Scalar field",
            lambda theme: lambda fig, gs: ScalarMapPanel(theme=theme).draw(
                fig, gs, field, title="Scalar field", axis_mode="none",
            ),
            1.0,
        ),
    ],
    report_title="Example report",
)

report.build_site({"site": {"id": "SITE-01"}}, save_path="report.png")

Application-specific validation reports can be built with the components under examples/validation_panels.py and examples/validation_report.py without adding those domain concepts to the reusable package.

Analysis and simulation

The main public analysis components include:

from openeolib import EOAnalyzer, PlumeSimulator, InferenceEngine

EOAnalyzer coordinates provider data and an inference engine. PlumeSimulator provides synthetic plume data for testing and demonstrations. InferenceEngine defines the interface for custom detection/quantification algorithms.

Providers

The package contains provider implementations for openEO, Sentinel-2, Sentinel-5P, Microsoft Planetary Computer (no-auth alternative for Sentinel-2/Sentinel-5P), ERA5, MSM, JMA radar rainfall products, JMA AMeDAS surface observations, University of Wyoming radiosonde soundings, GPT3/VMF3 blind-grid PWV, and Himawari-8/9 cloud-top brightness temperature. Provider-specific dependencies are optional where practical.

XrainProvider accepts NetCDF and CSV input. If a CSV has no time column, it creates a single time coordinate containing NaT rather than inventing an observation timestamp. Callers that require a real observation time must provide a time column or pass time_name=. This distinction is intentional and prevents silent fabrication of temporal metadata.

RadarGpvClient is a public retrieval utility, not a BaseProvider or AnalyzerProvider. It resolves JMA nationwide composite radar GPV archive URLs, downloads the archive, and extracts the target GRIB2 file. JmaRainProvider is the dataset-opening provider that parses those extracted files into an xarray.Dataset. This separation keeps network/archive handling distinct from dataset parsing.

openeolib.palettes holds domain color tables tied to a physical quantity, kept separate from theme.py's generic style values. It depends only on matplotlib, so it can be imported without the optional provider dependencies. Currently it provides the JMA rainfall-intensity palette (JMA_RAIN_BOUNDARIES, JMA_RAIN_COLORS, and jma_rain_cmap_norm() for a ready-made (cmap, norm) pair) -- the same table providers.rain uses internally, exposed for callers who want it outside that provider.

ERA5Provider supports surface fields and pressure-level wind processing, including height-aware interpolation for wind products and optional pressure-level specific humidity for moisture-transport calculations such as Integrated Vapor Transport (IVT, computed via eo_utils.compute_ivt() -- trapezoidal pressure-level integration that sorts levels internally so the result's sign never depends on the order the caller passed them in). fetch_grid_specific_humidity_at_pressure_levels() returns the full multi-level specific-humidity + geopotential-height profile over a bbox (rather than a single level), for callers doing AGL-layer water-vapor integration -- see openeolib.atmosphere.pwv_layers, which decomposes a GNSS station's total PWV into AGL layers (e.g. 0-1/1-2/2-3/3-6 km) using that profile's vertical shape. cds_retry_max/cds_sleep_max bound cdsapi's own inner retry loop (library default up to ~16.7 hours per request) separately from ERA5Provider's outer retry loop, so a real outage surfaces on a predictable timescale instead of a background job appearing stuck for hours. Provider modules return data; visualization remains a separate concern.

EOProvider (the S2/ERA5/S5P integration used by EOAnalyzer) persists every fetch() result on disk (ScienceCacheStore, enable_cache=True by default), keyed by (lat, lon, dt, band_set, radius_km, enable_100m_wind). An outright failure (no S2 scene at all) is never cached, so a later call retries instead of replaying the same empty result forever.

PlanetaryComputerProvider, in providers/planetary_computer.py, is a standalone alternative to Sentinel2Provider/Sentinel5PProvider that reads Microsoft Planetary Computer's public STAC catalog instead of OpenEO/CDSE -- no account or authentication required, just a free, auto-renewed SAS token (planetary-computer SDK). Use it directly (fetch_sentinel2() / fetch_sentinel5p()) when OpenEO/CDSE auth is unavailable or undesired; it is independent of OpenEOClient and not wired into EOProvider. Two differences from the OpenEO-backed providers follow from what the PC STAC catalog actually exposes: fetch_sentinel2()'s sza/vza/amf are always None, since PC's sentinel-2-l2a collection has no per-pixel sun/view zenith angle raster (OpenEO's SENTINEL2_L2A synthesizes one server-side); and fetch_sentinel5p() bins the raw swath pixels (irregular per-pixel lat/lon, full-orbit netCDF files rather than per-tile COGs) onto a regular grid itself, since PC has no pre-gridded product. Requires the planetary_computer extra (pystac-client, planetary-computer, stackstac).

MSMProvider retrieves JMA MSM GPV wind data and requires the jma extra. The public facade exposes MSMProvider, MSM_AVAILABLE, and MSM_MISSING_DEPENDENCIES so applications can detect optional support without catching an import failure from the core package.

AMeDASProvider and RadiosondeProvider retrieve historical JMA surface observations and University of Wyoming upper-air soundings respectively, both by scraping the providers' public data-search pages (requests + beautifulsoup4) rather than an official API, and both cache retrieved results indefinitely on disk (ScienceCacheStore), since historical observations never change. Both return xr.Dataset (time-dimensioned for AMeDAS; time/level-dimensioned, NaN-padded across variable-length profiles, for radiosonde soundings), matching the convention used by the rest of the providers rather than a bespoke Dict shape.

PWVProvider (GPT3 blind climatological grid) and VMF3Provider (VMF3_OP per-epoch NWP grid), in providers/pwv.py, are independent, GNSS-external sources of Zenith Hydrostatic/Wet Delay for validating or cross-checking an ERA5-based PWV pipeline. Both ultimately produce a PWV estimate via the Bevis et al. (1994) Pi(Tm) conversion; see the module docstring for the K1/K2 refractivity-constant caveat when comparing against another pipeline's convention.

UNRTropProvider, in providers/unr_trop.py, retrieves 5-minute GNSS tropospheric products (zenith total/wet delay, tropospheric delay gradients Gn/Ge, PWV, weighted mean temperature) from the Nevada Geodetic Laboratory (NGL/UNR) archive -- >18,600 stations worldwide, independent of any national agency's own real-time solution. Same two-stage API (find_stations_in_bbox() -> fetch_day()) and ScienceCacheStore persistent-cache convention as AMeDASProvider/RadiosondeProvider.

eo_grid.py provides regrid_to_grid() (regrid one regular lat/lon grid onto another, e.g. a coarser NWP product's native grid onto a finer analysis grid) and gaussian_weighted_grid_interpolation() (scattered station observations, scalar or multi-component, onto a regular grid via Gaussian-distance weighting). Split out of eo_utils.py, which is intentionally numpy-only, since both need scipy.interpolate.

It also provides nan_aware_gaussian_filter() and nan_aware_gaussian_derivative(), NaN-aware counterparts of scipy.ndimage.gaussian_filter built on normalized convolution: a gap in the input (e.g. a land-only product over a domain that includes sea) only affects pixels with no valid data within the kernel, rather than spreading NaN across the whole kernel footprint. Both reduce exactly to the plain scipy.ndimage call when the input has no NaN.

HimawariCloudClient / HimawariCloudProvider, in providers/himawari.py, download and decode Himawari-8/9 AHI gridded cloud-top brightness temperature from CEReS (Chiba University), requiring the himawari extra (satpy). This module's underlying data has its own license, separate from openeolib's Apache-2.0 code license: non-commercial/research use only, no redistribution, and required attribution when used or published. Both classes emit a UserWarning with the attribution text on instantiation; see the module docstring for the full terms before using it beyond research.

For ERA5, CDS credentials and the current CDS API/client configuration are required for live retrieval. Unit tests mock retrieval where network access is unnecessary.

Animation

GridAnimationEngine in openeolib.animation is the generic animation component. Domain-specific wrappers, such as animate_rain() in providers.rain, configure rainfall-specific variables and color scales before delegating rendering to the generic engine.

from openeolib.animation import GridAnimationEngine

engine = GridAnimationEngine()
engine.animate_scalar_field(
    grids=grids,
    lats=lats,
    lons=lons,
    timestamps=timestamps,
    output_path="animation.gif",
)

Validation examples

Validation-only components are kept outside the package:

from examples.validation_panels import (
    SpectralPanel,
    StatisticalPanel,
    FlagsPanel,
)
from examples.validation_report import ValidationReportBuilder

These modules are useful for project-specific evaluation but are not exported from openeolib and should not be treated as stable library APIs.

Testing

The test suite is maintained for internal development only. tests/ is intentionally excluded from both the published package (see MANIFEST.in) and this repository's git history (see .gitignore), so it is not available to clone or install. This has no bearing on installing or using the library.

License

OpenEO-LIB is distributed under the Apache License 2.0. See LICENSE for the full license text.