pysentinel2

A local Sentinel-2 datacube that fills itself on demand. Every pixel this machine ever downloads lands in one sparse, pixel-indexed store — so nothing is ever downloaded twice: overlapping areas, extended date ranges and repeat runs all reuse the same chunks. Part of the Borevitz Lab ecosystem; the default source is Digital Earth Australia's ARD collections (ga_s2am_ard_3 / ga_s2bm_ard_3) via STAC.

Full documentation — architecture, grid geometry, storage, cleaning, indices, robustness — is in docs/, with flowcharts and figures generated from a real store.

Contents of the store for a 2 × 2 km example window. Clear, cloudy and off-swath days are all stored raw and classified at read time.

How it works

{data_root}/sentinel2_cube/
├── index.db      # SQLite: coverage rects · seen scenes · past searches
└── cube.zarr/
    ├── 2024-01-03/   # one group per solar day
    │   ├── nbart_red # arrays on a fixed EPSG:6933 10 m global grid
    │   └── ...       # sparse: only written 256×256-px chunks exist on disk
    └── 2024-01-08/ ...

• Any bbox maps deterministically to a pixel window on the fixed grid. Cube.get_ds(bbox, start, end) subtracts each day's recorded coverage rectangles from that window and downloads only the missing pixels — coverage accounting is pixel-exact, so small farms pay no chunk padding (256×256-px chunks remain the storage unit inside the Zarr arrays).
• STAC results are cached (full item JSON) in the index, so re-reads and re-fills of known regions work without re-searching. Cloud-cover filtering happens at read time from the index — relaxing the threshold later needs no re-search.
• Only raw bands (incl. fmask) are stored. get_ds(..., clean=True) applies cloud masking on read — there is no second "clean" copy on disk, roughly halving storage versus a raw+clean layout. See Cleaning & masking for exactly what the mask does.
• Spectral indices — NDVI, CFI, NIRv, NDTI, CAI — are on-read derivatives too: get_ds(..., indices=('NDVI', 'NIRv')) computes them from cloud-masked reflectance and stores nothing.
• Writes are whole-chunk and the index is transactional (SQLite/WAL): a crash mid-fill just leaves cells unmarked, and the next run resumes.

Usage

The core API is troi-agnostic — just a bbox and dates, no setup:

from datetime import date
from pysentinel2.cube import Cube

cube = Cube()
bbox = [148.36265, -33.52606, 148.38265, -33.50606]  # [W, S, E, N]

ds_raw = cube.get_ds(bbox, date(2024, 1, 1), date(2024, 12, 31))
ds     = cube.get_ds(bbox, date(2024, 1, 1), date(2024, 12, 31), clean=True)
ds     = cube.get_ds(bbox, date(2024, 1, 1), date(2024, 12, 31),
                     indices=('NDVI', 'CFI', 'NIRv', 'NDTI', 'CAI'))

cube.fill(bbox, date(2024, 1, 1), date(2024, 12, 31))  # → 0: already local

Pipelines that speak the shared troi.troi.Troi (the reproducibility layer — stubs, registry) use the adapters:

ds = cube.get_ds_troi(troi)            # = cube.get_ds(troi.bbox, troi.start, troi.end)

download_sentinel2(troi) and clean_sentinel2(troi) remain as thin wrappers over Cube.get_ds_troi for pipeline compatibility.

Package design (shared across the lab's packages — no inheritance, composition only):

• Troi (from troi) — identity: what region, what dates.
• Sentinel2 (pysentinel2.sentinel2) — config: STAC URL, collections, bands, CRS, cloud threshold, fmask codes.
• Paths (pysentinel2.paths) — derived locations of the store for a given Config.
• grid — the fixed global grid (pure, offline-testable math).
• Index (pysentinel2.index) — the SQLite ledger.
• Cube (pysentinel2.cube) — ties them together.

Cleaning & masking

clean=True (and any indices= request, which implies it) runs the window through pysentinel2.cube.clean_dataset. The design principle: invalid and contaminated are different things.

Contaminated pixels are dilated before masking, frames are gated on the two fractions independently, and every read is annotated with the statistics and thresholds that produced it. Nothing is persisted — different thresholds on the same window are just different reads of the same raw store. Full pipeline, tunables and figures: docs/cleaning.md.

Performance

Live measurements against DEA — a ~2 × 2 km AOI, 11-band ARD at 10 m (one cell = one 256 × 256-px chunk on one solar day):

Store footprint: 13.6 MB for 11 solar days — raw + fmask only, since the clean cube is a 0.1 s on-read transform rather than a second copy.

Absolute times vary with network and DEA load. The zero rows are the significant ones: those requests are resolved by index lookups alone, with no network access.

Multi-year fills are batched, not per-day: all 11 bands for up to 64 missing days come down in one bulk load per batch (see the fill algorithm), keeping the I/O threads saturated across day boundaries — a two-month cold fill measured 20-26 s where a per-day loop measured 33 s on a healthy DEA and 270 s on a degraded one, and the gap widens with the length of the range. An earlier fmask-first screening pass was removed after measurement: it skipped 8.8% of days' reflectance while paying an extra request round on every day.

Install

pip

pip install git+https://github.com/thestochasticman/pysentinel2.git

Dependencies (the troi core included, pulled from GitHub) are declared in pyproject.toml and installed automatically.

From source

git clone https://github.com/thestochasticman/pysentinel2.git
cd pysentinel2
pip install -e .

The wheels for rasterio/rioxarray/opencv bundle their native libraries on common platforms; in a conda environment the conda-forge equivalents are used instead if already installed.

Robustness notes

Hardening for DEA's public S3 + STAC quirks (cold-start 504s, stalled reads, corrupt tiles) is built in — see docs/robustness.md and, for the underlying investigations, diagnostics.md.

Test

# offline (pure math + synthetic store):
python pysentinel2/grid.py    # True
python pysentinel2/index.py   # True
python pysentinel2/paths.py   # True
python pysentinel2/cube.py    # True

# live (small real downloads from DEA, incl. dedup assertions):
python pysentinel2/download_sentinel2.py  # True
python pysentinel2/clean_sentinel2.py     # True