pyozwald

Cached OzWALD time series for Australia — fetch once per grid point, never twice. OzWALD is ANU's Water and Landscape Dynamics dataset: modelled daily meteorology (~5 km) and 8-day biophysical variables (~500 m, MODIS-derived) served as one OPeNDAP NetCDF per variable per year. Every observation this machine ever samples lands in one SQLite store, so repeat requests, nearby coordinates in the same cell, and extended date ranges all reuse the same rows. Part of the Borevitz Lab ecosystem.

How it works

{data_root}/ozwald_store/
└── ozwald.db
    ├── observations(point, cadence, variable, date, value)
    └── coverage(point, cadence, variable, year, through)

• Coordinates snap to a dedup grid matching each product's native resolution — 0.05° for daily meteorology, 0.005° for the 8-day variables — so nearby requests share one stored series per cadence.
• OzWALD's unit of delivery is one NetCDF per (variable, year), so that's the unit of the coverage ledger. Store.get_df(...) diffs the requested years × variables against it and samples only the missing cells; a whole year is stored even when a sub-range was requested, since the marginal cost is nil and it maximises reuse.
• through records the last date a year's file actually contained — an in-progress year keeps being re-fetched until complete, then never again.
• Writes are transactional (SQLite/WAL): a crash mid-fetch leaves the cell unrecorded, and the next run re-fetches it.

Usage

The core API is troi-agnostic — a coordinate, dates, and a cadence:

from datetime import date
from pyozwald.store import Store

store = Store()

met = store.get_df(-33.516, 148.373, date(2023, 1, 1), date(2023, 12, 31))
#     daily meteorology: time, Pg, Tmax, Tmin, Uavg, Ueff, VPeff, ...

veg = store.get_df(-33.516, 148.373, date(2023, 1, 1), date(2023, 12, 31),
                   cadence='8day', variables=['NDVI', 'LAI', 'GPP'])
#     8-day biophysical series on the ~500 m grid

store.fill(-33.516, 148.373, date(2023, 1, 1), date(2023, 12, 31))  # → 0: already local

Pipelines that speak the shared troi.troi.Troi use the adapters (evaluated at the bbox centre):

df = store.get_df_troi(troi, cadence='daily')

download_ozwald_daily(troi) and download_ozwald_8day(troi) remain as thin wrappers.

Performance

Live measurements against NCI THREDDS — one grid point:

(One cell = one variable × one year at one grid point.) Store footprint: ~100 KB for the five cells above. Absolute times vary with network and THREDDS load; the zeros are the point — they are ledger lookups, no network involved.

Install

pip

pip install git+https://github.com/thestochasticman/pyozwald.git

Dependencies (the troi core included, pulled from GitHub) are declared in pyproject.toml and installed automatically.

From source

git clone https://github.com/thestochasticman/pyozwald.git
cd pyozwald
pip install -e .

Package design (shared across the lab's packages — no inheritance, composition only):

• Troi (from troi) — identity: what region, what dates.
• OzWALD (pyozwald.ozwald) — config: endpoint, variable catalogs per cadence, dedup grid steps.
• Paths (pyozwald.paths) — derived location of the store for a given Config.
• grid — the dedup grids (pure, offline-testable math).
• Store (pyozwald.store) — ties them together.

Test

# offline (pure math + synthetic store):
python pyozwald/grid.py     # True
python pyozwald/paths.py    # True
python pyozwald/store.py    # True

# live (small real samples from NCI THREDDS, incl. dedup assertions):
python pyozwald/download_ozwald.py  # True