Skip to main content

OpenEO-LIB

OpenEO-LIB is a Python library for satellite-based Earth observation workflows, including data providers, geospatial analysis, plume simulation, and reusable visualization components.

The package is designed so that reusable library functionality is kept separate from application-specific validation and research code.

Current architecture

openeolib/
├── __init__.py
├── _plot_utils.py       Shared Matplotlib helpers for the panels (internal)
├── analyzer.py          Analysis pipeline orchestration
├── animation.py         Generic scalar-field animation (GridAnimationEngine)
├── engines.py           Inference-engine interfaces and built-ins
├── eo_cache.py          Persistent scientific-data cache (ScienceCacheStore)
├── eo_grid.py           Grid regridding, Gaussian-weighted interpolation, NaN-aware Gaussian filter/derivative
├── eo_roi.py            Area-weighted ROI (bbox/polygon) statistics of gridded time series
├── eo_types.py          Shared data structures and type definitions
├── eo_utils.py          Geospatial and wind utilities
├── palettes.py          Domain color tables (e.g. JMA rainfall-intensity palette)
├── panels.py            Generic Matplotlib visualization panels
├── report.py            Generic figure/report composition
├── roc.py               ROC construction
├── simulator.py         Synthetic plume generation
├── theme.py             Theme definition only (Theme, DEFAULT_THEME)
├── atmosphere/
│   └── pwv_layers.py    Decompose GNSS total PWV into AGL layers from an ERA5 profile
└── providers/
    ├── base.py               BaseProvider ABC (raw Dataset retrieval)
    ├── provider.py           AnalyzerProvider / EOProvider (S2+ERA5+S5P integration)
    ├── openeo_client.py      openEO connection client
    ├── _http_utils.py        Shared retry/local-file-cache helpers
    ├── _protocols.py         Cross-provider provenance accessor (get_provenance())
    ├── sentinel2.py          Sentinel-2 L2A band retrieval
    ├── sentinel5p.py         Sentinel-5P L2 CH4 column retrieval
    ├── planetary_computer.py Sentinel-2/Sentinel-5P via Microsoft Planetary Computer (no auth)
    ├── earth_search.py       Sentinel-2 via Element84 Earth Search / AWS (no auth)
    ├── _stac_s2.py           Shared Sentinel-2 STAC search/retrieval core (PC + Earth Search)
    ├── era5.py               ERA5 wind/pressure/temperature retrieval
    ├── hrrr.py               NOAA HRRR 10 m wind retrieval (CONUS, no auth)
    ├── msm.py                JMA MSM (mesoscale model) GPV retrieval (MSMProvider), plus a
    │                         subprocess-isolated batch runner (BatchRequest/run_batch_isolated)
    ├── lfm.py                JMA LFM (local forecast model) GPV retrieval scaffold (LFMProvider)
    ├── radar_gpv_client.py   JMA nationwide composite radar GPV download
    ├── rain.py               JMA rainfall / XRAIN providers + animate_rain() + rain_roi_timeseries()
    ├── amedas.py             JMA AMeDAS hourly / 10-minute surface observations
    ├── radiosonde.py         University of Wyoming radiosonde sounding retrieval
    ├── cwop.py               CWOP private weather stations (wind etc.) via NOAA MADIS public archive
    ├── nies_air.py           Air-quality monitoring stations' hourly wind/temp (NIES confirmed values)
    ├── tbeic.py              Tokyo Bay monitoring posts' 15-min over-water wind (TBEIC)
    ├── pwv.py                GPT3 / VMF3 blind-grid PWV retrieval
    ├── unr_trop.py           Nevada Geodetic Laboratory (NGL/UNR) GNSS tropospheric products
    ├── himawari.py           Himawari-8/9 AHI brightness temperature via CEReS (FTP)
    ├── himawari_aws.py       Himawari-8/9 AHI brightness temperature via NOAA AWS (HTTPS, no FTP)
    └── _s2_utils.py          Sentinel-2 coordinate/time conversion utilities

Provider responsibilities are deliberately separated. BaseProvider and AnalyzerProvider are independent interfaces with different responsibilities; AnalyzerProvider does not extend BaseProvider. BaseProvider is the minimal file/data-opening interface for providers that return an xarray.Dataset without application-level analysis. AnalyzerProvider is the higher-level interface used by EOAnalyzer and returns an ObservationBundle for a site and observation time -- a typing.Protocol (structural typing), so a provider conforms by having a matching get_bundle(site, dt) method, not by explicitly inheriting from it. EOProvider implements it for the openEO/Sentinel-2/Sentinel-5P workflow and contains the workflow-specific quality and time-alignment logic. AnalyzerProvider/ ObservationBundle is not a general-purpose provider contract -- it is EOAnalyzer's own methane/plume-detection input shape (it reads bands["B11"]/["B12"] directly), produced only by EOProvider and PlumeSimulator.

For code that wants to treat any provider's result uniformly regardless of its own return shape (xr.Dataset, a Dict, or None) -- e.g. for logging or an audit trail -- providers._protocols.get_provenance(result) extracts a normalized {source, provider_class, datetime_utc, provenance_key} dict from whichever of the metadata conventions this package's providers happen to use (.attrs["source"]/["provider_class"], .attrs["provider"], dict keys "backend"/"datetime_utc", or pwv.py's "epoch_posix_utc_sec" POSIX timestamp, converted to an ISO UTC string). It requires no changes to any existing provider.

The visualization layer (panels.py, theme.py, report.py, animation.py) is kept domain-agnostic. Domain providers may know the schema and units of their own source data, but they do not depend on application-specific research pipelines.

Likewise, theme.py contains only the Theme data structure and DEFAULT_THEME.

Installation

pip install openeolib

Development dependencies are optional and are intended for the private test suite:

pip install -e ".[dev]"

Optional provider dependencies are grouped by feature, e.g. pip install "openeolib[era5]" (or pip install -e ".[era5]" from a source checkout):

pip install "openeolib[provider]"            # openEO / Sentinel-2 / Sentinel-5P
pip install "openeolib[era5]"                # ERA5 via the CDS API
pip install "openeolib[planetary_computer]"  # Microsoft Planetary Computer
pip install "openeolib[earth_search]"        # Element84 Earth Search
pip install "openeolib[hrrr]"                # NOAA HRRR
pip install "openeolib[jma]"                 # MSM, LFM, JMA radar rainfall
pip install "openeolib[himawari]"            # Himawari-8/9

The core package supports Python 3.9 or later. Provider-specific optional dependencies may have their own Python-version requirements.

The core install (pip install openeolib) imports openeolib, openeolib.providers and the non-provider modules. Most provider modules import their extra's packages (for example xarray) at module level, so importing them without that extra raises ImportError.

The JMA extra (openeolib[jma]) is required for MSMProvider. The module openeolib.providers.msm imports without it; constructing MSMProvider, calling MSMProvider.read_pressure_level_profile() / read_surface_fields() or run_batch_isolated() then raises an ImportError naming every missing dependency and the installation command.

Upgrading from 0.2.x

0.3.0 changes some behaviour. Check these when upgrading:

  • ERA5 cache. ERA5Provider caches raw CDS values under cache_dir/raw. Entries written by 0.2.x directly under cache_dir are no longer read, so data is downloaded again on first use; the old files can be deleted.
  • ERA5 boundary-layer height. ERA5-Land has no boundary-layer height, so boundary_layer_height was always None. Pass enable_blh=True to fetch it from ERA5 single levels. Listing it in extra_variables now raises ValueError.
  • ERA5 100 m wind at a point. fetch(..., enable_100m_wind=True) now returns the 100 m keys; they were usually missing before.
  • Timezone-aware datetimes. ERA5Provider and LFMProvider convert them to UTC. Before, a JST time was requested as if its clock time were UTC. Naive datetimes are still taken as UTC.
  • MSM / LFM GRIB2 reading. Both read GRIB2 with eccodes (included in openeolib[jma]). MSMProvider(grib_backend="cfgrib") is deprecated.
  • Tests that patch ERA5 internals. ERA5Provider._download() is gone; replace provider.client (the CDS client) instead.

Public visualization API

The reusable visualization API consists of five panels:

  • VectorFieldPanel — arbitrary geospatial vector fields with optional scalar contours.
  • BasemapPanel — an already prepared RGB image with an optional marker.
  • RawBandsPanel — arbitrary grids of 2D images with a shared scale.
  • ScalarMapPanel — arbitrary 2D scalar fields.
  • DetectionMaskPanel — boolean, probability, or coverage masks.

Example:

import matplotlib.pyplot as plt
import numpy as np
from openeolib import ScalarMapPanel

field = np.random.default_rng(0).normal(size=(100, 100))

fig = plt.figure(figsize=(6, 5))
gs = fig.add_gridspec(1, 1)[0]
ScalarMapPanel().draw(
    fig,
    gs,
    field,
    title="Scalar field",
    axis_mode="none",
)
fig.savefig("scalar_field.png", dpi=150, bbox_inches="tight")

Vector fields

VectorFieldPanel is intentionally not ERA5-specific. It accepts latitude/longitude grids and arbitrary vector components. A caller may supply any bounding box through extent=(west, east, south, north).

from openeolib import VectorFieldPanel

panel = VectorFieldPanel(quiver_stride=3)
panel.draw(
    fig,
    gs,
    lats=lats,
    lons=lons,
    u=u,
    v=v,
    extent=(120, 150, 20, 50),
    title="Wind field",
)

There is no Japan-specific bounding-box constant in the library.

Theme

Use Theme when a caller needs to customize the visual appearance:

from openeolib import Theme

light = Theme(
    bg="#ffffff",
    panel_bg="#ffffff",
    grid_color="#cccccc",
    text_primary="#222222",
)

Theme does not contain plotting operations or application-specific labels, flags, or data extraction rules.

Generic reports

SiteReportBuilder composes caller-supplied panels into a single-site report Figure. It does not know about methane, ROC curves, quality flags, or a particular inference engine -- it only handles grid layout, the report title, and file saving.

Each entry in panels is (label, factory, height_ratio). factory is called once with the report's Theme and must return a (fig, gs) -> None draw function; any panel-specific data (the field to plot, its title, ...) is bound into that closure by the caller, not inspected by SiteReportBuilder itself.

import numpy as np
from openeolib import SiteReportBuilder, ScalarMapPanel

field = np.random.default_rng(0).normal(size=(100, 100))

report = SiteReportBuilder(
    panels=[
        (
            "Scalar field",
            lambda theme: lambda fig, gs: ScalarMapPanel(theme=theme).draw(
                fig, gs, field, title="Scalar field", axis_mode="none",
            ),
            1.0,
        ),
    ],
    report_title="Example report",
)

report.build_site({"site": {"id": "SITE-01"}}, save_path="report.png")

Application-specific validation reports can be assembled from these reusable panels in the consuming application; application-specific validation code is not included in this package.

Analysis and simulation

The main public analysis components include:

from openeolib import EOAnalyzer, PlumeSimulator, InferenceEngine

EOAnalyzer coordinates provider data and an inference engine. PlumeSimulator provides synthetic plume data for testing and demonstrations. InferenceEngine defines the interface for custom detection/quantification algorithms.

Providers

The package contains provider implementations for openEO, Sentinel-2, Sentinel-5P, Microsoft Planetary Computer and Element84 Earth Search (both no-auth alternatives for Sentinel-2), ERA5, NOAA HRRR, JMA MSM and LFM, JMA radar rainfall and XRAIN, JMA AMeDAS surface observations, CWOP private weather stations, NIES air-quality monitoring stations, Tokyo Bay monitoring posts (TBEIC), University of Wyoming radiosonde soundings, GPT3/VMF3 blind-grid PWV, NGL/UNR GNSS tropospheric products, and Himawari-8/9 cloud-top brightness temperature. Provider-specific dependencies are optional where practical.

XrainProvider accepts NetCDF and CSV input. If a CSV has no time column, it creates a single time coordinate containing NaT rather than inventing an observation timestamp. Callers that require a real observation time must provide a time column or pass time_name=. This distinction is intentional and prevents silent fabrication of temporal metadata.

RadarGpvClient is a public retrieval utility, not a BaseProvider or AnalyzerProvider. It downloads the 1 km rainfall GRIB2 of the JMA nationwide composite radar GPV from the Kyoto University RISH archive: the five-minute raw Ggis1km_Prr05lv files from 2024-02-26, or the file extracted from the ten-minute tar archives before that. JmaRainProvider is the dataset-opening provider that parses those GRIB2 files into an xarray.Dataset. This separation keeps network/archive handling distinct from dataset parsing.

rain_roi_timeseries() reduces the Dataset returned by JmaRainProvider.open() or XrainProvider.open() to one row per time step over an ROI: the area-weighted mean rate, the maximum cell rate, the accumulated rainfall (mean rate x time step) and the fraction of the ROI with valid data, plus optional threshold-exceedance area fractions (thresholds=(20, 50) -> frac_ge_20, frac_ge_50). Negative values are treated as missing. The ROI is either the same {"lat": (min, max), "lon": (min, max)} dict as open(region=...) or a polygon (shapely geometry, GeoJSON-like mapping, or a list of (lon, lat) vertices); each grid cell is weighted by its area inside the ROI.

from openeolib.providers.rain import JmaRainProvider, rain_roi_timeseries

ds = JmaRainProvider().open(paths, region={"lat": (34.5, 36.8), "lon": (138.8, 141.2)})
df = rain_roi_timeseries(ds, {"lat": (35.4, 35.8), "lon": (139.9, 140.3)}, thresholds=(20, 50))

The general-purpose function behind it is eo_roi.roi_timeseries() (any (time, latitude, longitude) field, xarray or numpy; statistics mean, std, min, max, area_integral in units x km^2, accum, valid_fraction). eo_roi.roi_cell_weights() returns the per-cell inside-area (km^2) on its own. Cell areas use the spherical formula; bbox overlaps are exact, polygon overlaps are exact with shapely and estimated by sub-cell sampling without it. A bare 4-tuple bbox is rejected on purpose, because openeolib's ERA5 bbox order (N, W, S, E) differs from GeoJSON's (W, S, E, N).

openeolib.palettes holds domain color tables tied to a physical quantity, kept separate from theme.py's generic style values. It depends only on matplotlib, so it can be imported without the optional provider dependencies. Currently it provides the JMA rainfall-intensity palette (JMA_RAIN_BOUNDARIES, JMA_RAIN_COLORS, and jma_rain_cmap_norm() for a ready-made (cmap, norm) pair) -- the same table providers.rain uses internally, exposed for callers who want it outside that provider.

ERA5Provider supports surface fields and pressure-level wind processing, including height-aware interpolation for wind products and optional pressure-level specific humidity for moisture-transport calculations such as Integrated Vapor Transport (IVT, computed via eo_utils.compute_ivt() -- trapezoidal pressure-level integration that sorts levels internally so the result's sign never depends on the order the caller passed them in). fetch_grid_specific_humidity_at_pressure_levels() returns the full multi-level specific-humidity + geopotential-height profile over a bbox (rather than a single level), for callers doing AGL-layer water-vapor integration -- see openeolib.atmosphere.pwv_layers, which decomposes a GNSS station's total PWV into AGL layers (e.g. 0-1/1-2/2-3/3-6 km) using that profile's vertical shape. cds_retry_max/cds_sleep_max bound cdsapi's own inner retry loop (library default up to ~16.7 hours per request) separately from ERA5Provider's outer retry loop, so a real outage surfaces on a predictable timescale instead of a background job appearing stuck for hours. Provider modules return data; visualization remains a separate concern.

Each ERA5Provider method takes one dt; fetch_series(method, start, end, **kwargs) runs any of them over an hour range, e.g. provider.fetch_series("fetch", datetime(2020, 7, 1), datetime(2020, 7, 31, 23), lat=33.59, lon=130.40), and returns {hour: result}. It downloads the range first in a few CDS requests (whole months for ERA5-Land and single levels, about a week at a time for pressure levels, split further if CDS rejects a request as too large) instead of one request per hour. Only raw CDS values are cached, one entry per request shape and hour under cache_dir/raw, so a method's result never depends on the cache state, and methods asking for the same data share it. The requested area is always sent unchanged: pressure-level values differ in the last float32 bits when the area changes. Boundary-layer height is not an ERA5-Land variable; enable_blh=True fetches it from ERA5 single levels (0.25 deg, a separate request), and without it boundary_layer_height is None. Point values from single levels (100 m wind, boundary-layer height) come from the nearest 0.25 deg node.

EOProvider (the S2/ERA5/S5P integration used by EOAnalyzer) persists every fetch() result on disk (ScienceCacheStore, enable_cache=True by default), keyed by (lat, lon, dt, band_set, radius_km, enable_100m_wind). An outright failure (no S2 scene at all) is never cached, so a later call retries instead of replaying the same empty result forever.

PlanetaryComputerProvider, in providers/planetary_computer.py, is a standalone alternative to Sentinel2Provider/Sentinel5PProvider that reads Microsoft Planetary Computer's public STAC catalog instead of OpenEO/CDSE -- no account or authentication required, just a free, auto-renewed SAS token (planetary-computer SDK). Use it directly (fetch_sentinel2() / fetch_sentinel5p()) when OpenEO/CDSE auth is unavailable or undesired; it is independent of OpenEOClient and not wired into EOProvider. Two differences from the OpenEO-backed providers follow from what the PC STAC catalog actually exposes: fetch_sentinel2()'s sza/vza/amf are always None, since PC's sentinel-2-l2a collection has no per-pixel sun/view zenith angle raster (OpenEO's SENTINEL2_L2A synthesizes one server-side); and fetch_sentinel5p() bins the raw swath pixels (irregular per-pixel lat/lon, full-orbit netCDF files rather than per-tile COGs) onto a regular grid itself, since PC has no pre-gridded product. Requires the planetary_computer extra (pystac-client, planetary-computer, stackstac).

fetch_sentinel2() also accepts level="l1c" for top-of-atmosphere data alongside the default level="l2a" (surface reflectance); Sentinel2Provider only exposes L2A. The L1C collection has no SCL band, so cloud/no-data masking is skipped and cloud_ratio is always None for level="l1c" -- only scene_cloud_cover (the STAC item's whole-scene eo:cloud_cover, distinct from cloud_ratio's AOI-local SCL-based fraction) is available. max_cloud_cover excludes scenes above a given eo:cloud_cover from the STAC search entirely, time_delta_days overrides the search window's half-width (default: the same window Sentinel2Provider uses), and preferred_tile (an MGRS tile id, e.g. "14SKA") breaks ties between same-orbit adjacent-tile candidates that otherwise tie on time and cloud cover -- useful when fetching a second, comparison date and avoiding a spurious cross-UTM-zone discrepancy against the first. The returned dict also includes mgrs_tile and a grid dict (crs/transform/shape/ resolution/bounds) describing the returned arrays' geolocation. The preferred_tile tiebreak uses shapely (part of the planetary_computer extra) when available, and otherwise falls back to plain nearest-time/ cloud-cover selection.

PlanetaryComputerProvider.__init__() and EarthSearchProvider.__init__() (see below) both accept scl_mask_classes/scl_cloud_classes to override which SCL classes fetch_sentinel2() masks out and counts as cloud, on top of the Sentinel2Provider.__init__()-level default (0, 9, 10)/(9, 10) (NO_DATA/CLOUD_HIGH_PROBABILITY/THIN_CIRRUS masked; the latter two counted as cloud). All three providers share this same default; pass e.g. scl_mask_classes=(0, 3, 8, 9, 10) to also mask CLOUD_SHADOWS (3) and CLOUD_MEDIUM_PROBABILITY (8) if the default's lighter masking lets too many cloud-affected pixels through for a given use case.

EarthSearchProvider, in providers/earth_search.py, is a second no-auth Sentinel-2 alternative, reading Element84's public Earth Search STAC catalog (the AWS-hosted "Sentinel-2 Cloud-Optimized GeoTIFFs" dataset) instead of Planetary Computer's Azure-hosted one. Both catalogs expose the same community COG convention (asset ids, eo:cloud_cover, s2:mgrs_tile), so EarthSearchProvider.fetch_sentinel2() has the identical signature and return shape as PlanetaryComputerProvider.fetch_sentinel2() (level, max_cloud_cover, preferred_tile, time_delta_days, ...) -- the two providers share their STAC search/tiebreak/retrieval core (providers/_stac_s2.py, not a public module) and differ only in endpoint and the fact that Earth Search's asset URLs are plain public HTTPS, needing no SAS-token signing step. Use it as a fallback when Planetary Computer is unreachable, or when AWS-side latency/egress makes it the better choice for a given deployment. Requires the earth_search extra (pystac-client, stackstac, shapely); unlike planetary_computer, it does not need the planetary-computer SDK.

HRRRProvider, in providers/hrrr.py, retrieves 10 m wind from NOAA's High-Resolution Rapid Refresh model (CONUS domain, ~3 km grid, hourly cycles), read from the public, anonymously-readable AWS Open Data bucket -- no account or authentication required. fetch_wind(lat, lon, dt, radius_km=, forecast_hour=0) downloads only the two 10 m-above-ground UGRD/VGRD GRIB2 messages it needs, located via the archive's own .idx sidecar and fetched with HTTP byte-range requests rather than the full ~100-150 MB GRIB2 file (the same approach the community "Herbie" tool uses). HRRR's native grid is a curvilinear Lambert Conformal projection, not a regular lat/lon grid, so the returned u10/v10/wind_speed/wind_deg arrays come with matching 2D lats/lons coordinate arrays (cropped to roughly radius_km around the AOI) rather than a 1D axis pair, alongside nearest_u10/nearest_v10/nearest_wind_speed/nearest_wind_deg at the single closest grid cell. Requests outside the CONUS domain return None. It is independent of ERA5Provider/OpenEOClient and not wired into EOProvider; use it as a CONUS-only, higher-resolution alternative to ERA5Provider's surface wind. Requires the hrrr extra (cfgrib, xarray). Like MSMProvider below, fetch_wind() results are cached on disk (ScienceCacheStore, enable_cache=True by default, cache_dir= "./data/hrrr_cache") keyed by its own arguments, so a repeat call for the same (lat, lon, dt, radius_km, forecast_hour) skips the idx fetch, both byte-range GRIB2 fetches, and the cfgrib decode entirely.

MSMProvider, in providers/msm.py, retrieves JMA MSM GPV surface and pressure-level fields and requires the jma extra. Its results are also cached (ScienceCacheStore, enable_cache=True by default, cache_dir nested under data_root so distinct instances never share a cache), so a repeat get_wind()/get_surface()/get_wind_forecast()/get_surface_forecast()/ get_pressure_level_scalars() call with the same arguments skips re-downloading and re-reading the GRIB2 file -- the dominant per-call cost once the raw file is local. download_read_timeout_s (default 20) controls how long a single file download will wait on a slow-but-not-closed connection before retry_get() counts it as a failure and retries. This interacts with max_retries multiplicatively (worst case is max_retries * download_read_timeout_s, plus backoff); a caller expecting unusually large/slow bulletins can pass a larger value explicitly.

MSMProvider reads GRIB2 with eccodes directly (grib_backend="eccodes", the default), message by message, releasing each message as soon as its values are copied. The former cfgrib-based reading, grib_backend="cfgrib", is deprecated and will be removed in a future release; selecting it emits a FutureWarning. In a long-running process it can accumulate eccodes memory until it fails with "Memory allocation error", after which every later GRIB read in that process fails too. The eccodes reader preserves cfgrib's variable names, values, dimensions, coordinates, and core CF variable metadata (units, long_name, and standard_name). For MSM, cfgrib is imported only when the deprecated backend is selected.

The same module also exposes a subprocess-isolated batch runner -- BatchRequest/run_batch_isolated(), for resumable, crash-isolated batches. BatchRequest (one (key, method, kwargs) call against MSMProvider) and run_batch_isolated() split many requests into small chunks (default 20) and re-launch a brand-new python -m openeolib.providers.msm --chunk-worker subprocess per chunk. A single request's own failure only marks that request (via a caller-supplied extractor function); a whole chunk crashing (non-zero exit, a timeout, or no result from the subprocess) NaN-fills that chunk's rows instead of aborting the batch. Results are appended to output_csv after every chunk, and a request whose key already has a row there is skipped on a later run -- so an interrupted batch resumes rather than restarting.

For whole-bulletin work (moisture columns, many levels or forecast steps), MSMProvider also has whole-file readers (always eccodes): get_pressure_level_profile() returns t/u/v/r (and gh/w) on every pressure level and every forecast step of an L-pall bulletin in one call (r exists on 1000-300 hPa only), and get_surface_fields_forecast() returns Lsurf surface pressure [hPa] and hourly precipitation [mm] (the field cfgrib names "unknown"; selected by GRIB discipline 0 / category 1 / number 8). get_tpw_ivt() combines them into precipitable water and IVT, cutting the column at MSM's own surface pressure by default -- without the cut, below-ground pressure levels are integrated as if they were air, inflating precipitable water over mountains. The underlying integrals are eo_utils.compute_column_water_vapor() and eo_utils.compute_ivt(..., surface_pressure_hpa=...), with eo_utils.specific_humidity_from_rh() (Bolton 1980) for q. download_range(start, end, products, max_workers) fetches every 3-hourly bulletin in a period in parallel, and the raw_dir constructor argument lets the provider read from and download into an existing flat archive directory (files already there are never re-downloaded). The static read_pressure_level_profile() / read_surface_fields() take a local file path directly.

LFMProvider, in providers/lfm.py, retrieves JMA LFM (local forecast model, 局地モデル) GPV data -- 1 km-resolution surface and model-level fields over Japan -- reusing MSMProvider's download/cache/retry scaffolding and GRIB2-handling conventions. It reads GRIB2 with eccodes directly, message by message, like MSMProvider's default backend; it does not use cfgrib.

Status: unconfirmed API, scaffold only. Unlike MSM (freely mirrored by Kyoto University's RISH archive), LFM is distributed only through JMBSC's "気象庁クラウド環境" (JMA Cloud environment), a contracted, paid service (申込み + 情報提供負担金). The following are not publicly documented and could not be confirmed without a contract; they are documented only in JMBSC's 利用者マニュアル, obtained from JMBSC's 配信事業部 (haisin@jmbsc.or.jp):

  • base_url: the download endpoint is unknown. Pass it explicitly (base_url=) or via the JMBSC_LFM_BASE_URL environment variable once obtained from the manual; any call needing it raises RuntimeError until then.
  • Authentication: deliberately restricted to ~/.netrc (HTTP Basic auth via requests' own netrc lookup) -- this module accepts no api_key/token/header parameter. Put the JMBSC-issued credentials in ~/.netrc under the download host's name.
  • Filename pattern: _build_filename() uses a best-effort guess following JMA's other GPV products' naming convention, substituting "LFM" for the model tag -- unverified against a real LFM filename.
  • Variable/grid naming: get_field() takes variable_candidates/ level/level_coord from the caller rather than hardcoding an LFM schema, since real GRIB2 variable naming for LFM's model-level fields is unconfirmed; use list_variables() against a real downloaded bulletin to discover it. The ecCodes reader preserves either 1-D rectilinear axes or matching 2-D latitude/longitude coordinates and masks records outside the requested box; confirm the actual grid shape and coordinate conventions with a real bulletin before relying on this scaffold.

AMeDASProvider and RadiosondeProvider retrieve historical JMA surface observations and University of Wyoming upper-air soundings respectively, both by scraping the providers' public data-search pages (requests + beautifulsoup4) rather than an official API, and both cache retrieved results indefinitely on disk (ScienceCacheStore), since historical observations never change. Both return xr.Dataset (time-dimensioned for AMeDAS; time/level-dimensioned, NaN-padded across variable-length profiles, for radiosonde soundings), matching the convention used by the rest of the providers rather than a bespoke Dict shape. JMA prints the wind-speed/direction column pair under two different main-label word orders depending on station class -- "風向・風速" for type-A/B (staffed/pressure) stations vs "風速・風向" for type-C (AMeDAS-only, the majority of the network) stations, the latter also using a "平均風速(m/s)" sub-label instead of "風速" -- so AMeDASProvider normalizes both the main and sub header label (stripping any "(unit)" suffix) before matching, rather than recognizing only one word order.

AMeDASProvider also retrieves JMA's 10-minute table (fetch_10min_day() / fetch_10min_range() / fetch_10min(), same station identifiers and xr.Dataset shape as the hourly methods, attrs["interval_minutes"] == 10): pressure (type-A/B only), precipitation, temperature, relative humidity, mean wind (wind_speed_ms, wind_direction), maximum instantaneous wind (wind_gust_ms, wind_gust_direction) and sunshine_minutes. The 10-minute pages use a two-row header for type-A/B stations but a three-row header for type-C stations (平均 / 最大瞬間, each over 風速 / 風向), so the header is parsed as a tree of any depth; a column it does not recognize raises a warning instead of being dropped silently. Type-C pages list every column whatever the station observes; columns a station does not observe are all NaN. 10-minute results are cached separately from hourly ones, keyed by a parser version so that entries written by an older parser are never reused.

PWVProvider (GPT3 blind climatological grid) and VMF3Provider (VMF3_OP per-epoch NWP grid), in providers/pwv.py, are independent, GNSS-external sources of Zenith Hydrostatic/Wet Delay for validating or cross-checking an ERA5-based PWV pipeline. Both ultimately produce a PWV estimate via the Bevis et al. (1994) Pi(Tm) conversion; see the module docstring for the K1/K2 refractivity-constant caveat when comparing against another pipeline's convention.

UNRTropProvider, in providers/unr_trop.py, retrieves 5-minute GNSS tropospheric products (zenith total/wet delay, tropospheric delay gradients Gn/Ge, PWV, weighted mean temperature) from the Nevada Geodetic Laboratory (NGL/UNR) archive -- >18,600 stations worldwide, independent of any national agency's own real-time solution. Same two-stage API (find_stations_in_bbox() -> fetch_day()) and ScienceCacheStore persistent-cache convention as AMeDASProvider/RadiosondeProvider.

CWOPProvider, in providers/cwop.py, retrieves observations (wind, temperature, humidity, pressure, precipitation) from privately operated weather stations in the Citizen Weather Observer Program (CWOP, MADIS data provider APRSWXNET), via the NOAA MADIS public archive (no account needed, 2001 onward). Each hourly archive file (~33 MB gzip'd, worldwide) is decompressed in memory and cut to a bounding box (default: Japan) before caching. Coverage in Japan is sparse and uneven (call list_stations() for a period's roster), anemometer height is not recorded, and wind speed is quantized to 1 mph.

MADIS data redistribution. CWOPProvider only downloads data for your own use; do not publish its cache as-is. CWOP is MADIS distribution category 3 ("Public - Full distribution"), but per the MADIS restrictions, "the mesonet data providers request that users not redistribute or place the data on web pages without notifying NOAA" -- contact the MADIS team before publishing the data itself. Data from providers in categories other than full distribution (possible with data_providers=None) must not be redistributed, and redistribution of real-time data, including web displays, is prohibited. Derived figures and statistics should cite MADIS/CWOP as the source.

NIESAirMonitorProvider, in providers/nies_air.py, retrieves hourly values from Japan's ~1,800 air-quality monitoring stations (大気汚染常時監視局) via the NIES download service (https://tenbou.nies.go.jp/download/). Most stations also measure wind direction/speed, temperature and humidity, and, unlike AMeDAS, the station attribute file records each instrument's height above ground (anemometers are usually on rooftops), reported as station_wind_height_m. Confirmed values only, fiscal year 2009 onward, published with a lag of about 1.5 years. Prefecture-year zip files are cached as downloaded.

TBEICProvider, in providers/tbeic.py, retrieves 15-minute over-water wind (mean/maximum speed, direction) and air temperature from four MLIT monitoring posts in inner Tokyo Bay (検見川沖, 川崎人工島, 浦安沖, 千葉港口第一号灯標; anemometers about 8-10 m above mean sea level) via the Tokyo Bay Environmental Information Center (https://www.tbeic.go.jp/MonitoringPost/Top), 2010-04 to near real time. These are preliminary (速報値) values; requests are split into 7-day chunks because the server slows sharply for longer ranges.

Terms of use for these sources. NIES: downloaded data may be used freely; credit the source when publishing results. TBEIC (site terms, section 1-4): when using the data for citation or research, (1) state TBEIC and the specific data source in the results, (2) report the purpose and content of use by e-mail to the TBEIC administrator (pa.ktr-kankyo@mlit.go.jp), sending a copy of the results if possible, and (3) do not use the data commercially.

eo_grid.py provides regrid_to_grid() (regrid one regular lat/lon grid onto another, e.g. a coarser NWP product's native grid onto a finer analysis grid) and gaussian_weighted_grid_interpolation() (scattered station observations, scalar or multi-component, onto a regular grid via Gaussian-distance weighting). Split out of eo_utils.py, which is intentionally numpy-only, since both need scipy.interpolate.

It also provides nan_aware_gaussian_filter() and nan_aware_gaussian_derivative(), NaN-aware counterparts of scipy.ndimage.gaussian_filter built on normalized convolution: a gap in the input (e.g. a land-only product over a domain that includes sea) only affects pixels with no valid data within the kernel, rather than spreading NaN across the whole kernel footprint. Both reduce exactly to the plain scipy.ndimage call when the input has no NaN.

HimawariCloudClient / HimawariCloudProvider, in providers/himawari.py, download and decode Himawari-8/9 AHI gridded cloud-top brightness temperature from CEReS (Chiba University), requiring the himawari extra (satpy). This module's underlying data has its own license, separate from openeolib's Apache-2.0 code license: non-commercial/research use only, no redistribution, and required attribution when used or published. Both classes emit a UserWarning with the attribution text on instantiation; see the module docstring for the full terms before using it beyond research. CEReS distributes this data via FTP only (no HTTP mirror), which outbound firewalls block on many corporate/institutional networks; pass HimawariCloudClient(..., allow_ftp=False) in that case to use only files already cached under raw_dir (pre-populated from an environment where FTP does work) -- a cache miss then raises FileNotFoundError immediately instead of waiting out timeout seconds per missing step. HimawariCloudClient is the only provider in this package that depends on FTP; every other provider uses HTTP(S) or an OpenEO/CDS API client.

HimawariAWSClient / HimawariAWSProvider, in providers/himawari_aws.py, are an HTTPS-only alternative to the above for environments where FTP itself is unavailable rather than merely slow -- allow_ftp=False still needs some pre-populated cache, whereas this module can download from scratch over plain HTTPS. They read the same underlying AHI instrument's data from a different distributor: raw Himawari-8/9 AHI Level-1b Standard Data (HSD), produced by JMA and redistributed by NOAA via the public, unauthenticated noaa-himawari8/noaa-himawari9 AWS Open Data buckets, decoded with satpy's ahi_hsd reader (also requires the himawari extra, but no additional dependency beyond it). One Full Disk time step is split across AWS_SEGMENTS (10) files that must all be downloaded and passed to HimawariAWSProvider.open() together, so HimawariAWSClient.download_range()/download_full_disk() deal in groups of file paths (one list of 10 per time step) rather than one path per step, unlike HimawariCloudClient. The AWS data's license is considerably more permissive than CEReS's (no non-commercial/ research-only restriction; NOAA/JMA only request attribution for unaltered data) -- see the module docstring for the exact terms. Both classes auto-select the noaa-himawari8/noaa-himawari9 bucket by JMA's 2022-12-13 Himawari-9 operational cutover date unless satellite= is passed explicitly.

For ERA5, CDS credentials and the current CDS API/client configuration are required for live retrieval.

Animation

GridAnimationEngine in openeolib.animation is the generic animation component. Domain-specific wrappers, such as animate_rain() in providers.rain, configure rainfall-specific variables and color scales before delegating rendering to the generic engine.

from openeolib.animation import GridAnimationEngine

engine = GridAnimationEngine()
engine.animate_scalar_field(
    grids=grids,
    lats=lats,
    lons=lons,
    timestamps=timestamps,
    output_path="animation.gif",
)

GIF frames share one 256-colour palette, so a colour means the same value in every frame. While writing, the frames are staged as PNG files in a temporary directory (a few hundred KB each) and then appended to the GIF one at a time, so memory use does not grow with the number of frames. For long time series, an .mp4 output path (needs ffmpeg) gives smaller files.

Testing

The test suite is maintained for internal development only. tests/ is intentionally excluded from both the published package (see MANIFEST.in) and this repository's git history (see .gitignore), so it is not available to clone or install. This has no bearing on installing or using the library.

License

OpenEO-LIB is distributed under the Apache License 2.0. See LICENSE for the full license text.

Metadata

Release files for openeolib 0.3.10

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for openeolib 0.3.10
File Size Uploaded Provenance Actions
openeolib-0.3.10.tar.gz 262.5 kB With provenance Details for openeolib-0.3.10.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for openeolib 0.3.10
File Interpreter ABI Platform Actions
openeolib-0.3.10-py3-none-any.whl Python 3 none any Details for openeolib-0.3.10-py3-none-any.whl

Total release size: 529.4 kB

Release files / openeolib-0.3.10.tar.gz

Download URL openeolib-0.3.10.tar.gz
Size 262.5 kB
Tags Source
SHA-256 checksum
How to use checksums
a682aa51ca04dddd5c640d985fd899ea073b7d18dc12a0b04f471822fd81fba1
BLAKE2b-256 checksum
How to use checksums
10a85cccfa77bc67ac236e7443cfab5f35d93ccdb42da5f5a3ca525edca93234
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.

Transparency log

Release files / openeolib-0.3.10-py3-none-any.whl

Download URL openeolib-0.3.10-py3-none-any.whl
Size 266.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a84bac4a8a151070a9e5b46591c89d8c251d9338b6baa0958eadd9a1eda5fe75
BLAKE2b-256 checksum
How to use checksums
e2b830d8574a5b1b7614172a5d0a6294485ee1b089baa5d7f48d7f9c3f482c75
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.

Transparency log

Security for openeolib 0.3.10

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for openeolib-0.3.10.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page