silo

Cached SILO daily climate for Australia — fetch once per grid point, never twice. Every daily observation this machine ever fetches lands in one SQLite store keyed by SILO's native 0.05° (~5 km) grid, so repeat requests, nearby farms in the same cell, and extended date ranges all reuse the same rows. Part of the Borevitz Lab ecosystem.

How it works

{data_root}/silo_store/
└── silo.db
    ├── observations(point, date, variable, value)   # every value ever fetched
    └── coverage(point, start, end)                  # which date spans are populated

• Any coordinate snaps deterministically to its nearest SILO grid point (~5 km cells — the resolution SILO interpolates at anyway).
• Store.get_df(lat, lon, start, end) diffs the requested range against the coverage ledger and fetches only the missing spans from the DataDrill endpoint, then reads the range.
• Coverage records only what SILO actually returned — if the record lags behind a requested recent date, the tail stays uncovered and is re-requested next time.
• Writes are transactional (SQLite/WAL): a crash mid-fetch leaves the span unrecorded, and the next run re-fetches it.

Usage

The core API is troi-agnostic — a coordinate and dates:

from datetime import date
from pysilo.store import Store

store = Store()   # email from ~/.config/Troi.json, or pass email=...

df = store.get_df(-33.516, 148.373, date(2023, 1, 1), date(2023, 12, 31))
#    one row per day: date, daily_rain, max_temp, min_temp, radiation,
#    vp, et_short_crop, ... (18 variables)

store.fill(-33.516, 148.373, date(2023, 1, 1), date(2023, 12, 31))  # → 0: already local

Pipelines that speak the shared troi.troi.Troi use the adapters (evaluated at the bbox centre):

df = store.get_df_troi(troi)

download_silo(troi) remains as a thin wrapper returning the classic YYYY-MM-DD-columned frame.

SILO requires a registration email (sent as the API username) — set email in ~/.config/Troi.json, TROI_EMAIL, or pass email= per call.

Performance

Live measurements against SILO — one grid point, all 18 variables:

Store footprint: ~0.5 MB per point-year across all variables. Absolute times vary with network and SILO load; the zeros are the point — they are ledger lookups, no network involved.

Install

pip

pip install git+https://github.com/thestochasticman/pysilo.git

Dependencies (the troi core included, pulled from GitHub) are declared in pyproject.toml and installed automatically.

From source

git clone https://github.com/thestochasticman/pysilo.git
cd pysilo
pip install -e .

Package design (shared across the lab's packages — no inheritance, composition only):

• Troi (from troi) — identity: what region, what dates.
• SILO (pysilo.silo) — config: endpoint, comment codes, variables.
• Paths (pysilo.paths) — derived location of the store for a given Config.
• grid — the fixed 0.05° grid (pure, offline-testable math).
• Store (pysilo.store) — ties them together.

Test

# offline (pure math + synthetic store):
python pysilo/grid.py     # True
python pysilo/paths.py    # True
python pysilo/store.py    # True

# live (small real fetches from SILO, incl. dedup assertions):
python pysilo/download_silo.py  # True