tethys-archiver

Faithful netCDF4 export of tethys datasets to local disk, with all metadata preserved.

Tethys is being decommissioned in favour of envlib. This tool is the step that makes retirement safe: it takes a full, checkable, local copy of a dataset before anything is switched off — and it doubles as the extraction half of the envlib migration.

Why it exists

tethysts pins pandas<2, which forces the numpy 1.x C ABI. cfdb>=0.9.4 (under envlib) requires numpy>2 and asserts that floor at runtime. uv reports the pair as unsatisfiable — so nothing can read tethys and write envlib in one process.

This tool lives on the legacy side of that wall. It writes plain netCDF4, which the modern stack reads with no legacy dependency at all. The file is the interchange format between two environments that cannot coexist.

Two further reasons it earns its place:

• The ECan quality-controlled record is single-copy. It was built from CSV exports of an internal database that no longer exists; the tethys objects are the only surviving copy.
• The extraction work was going to be written anyway — there are ~30 tethys-extraction-* sources. Better once, here, than once per source.

Install

uv sync

Python 3.11 only, deliberately — pandas 1.5.x has no cp312 wheel, and the pin set (numpy 1.26 / pandas 1.5.3 / h5py 3.16 / h5netcdf 1.8.1 / tethysts 4.5.16 / hdf5tools 0.2.4) is verified working end to end.

Use

Dry run is the default; a real run needs --yes.

# what would happen
uv run tethys-export --out ~/data/tethys-archive

# do it, with the verification gates
uv run tethys-export --out ~/data/tethys-archive --yes --verify

# re-check an existing archive against its manifest
uv run tethys-export --out ~/data/tethys-archive --check-manifest

Default scope is the six frozen ECan quality-controlled hourly datasets. --dataset (repeatable) selects specific ones; --all-regular takes every regular-cadence qc/raw time-series dataset in the bucket.

Output layout

<out>/<bucket>/
    datasets.json                          # bucket catalogue, verbatim
    manifest.json                          # per-file sha256 + counts
    <dataset_id>/
        dataset.json                       # verbatim
        versions.json                      # verbatim
        <version_date>.nc                  # all stations, dense (station, time)
        <version_date>.stations.json       # verbatim
        <version_date>.results_chunks.json # verbatim — per-chunk hashes

The JSON descriptors are written byte-for-byte as tethys published them, never re-serialised. results_chunks.json is the only place chunk_hash / chunk_id / chunk_day survive — get_results concatenates chunks and structurally cannot carry them through.

Reading the output

Nothing legacy required:

import xarray as xr
ds = xr.open_dataset('20220401T000000Z.nc', engine='h5netcdf')
ds['streamflow'][0, :]          # decoded to physical units automatically

Variables keep tethys's own names (ref, name, altitude, station_id, lon, lat). Renaming to envlib's station_ref / station_name / station_altitude is the consumer's job — see Faithful, not normalised below.

Design

Faithful, not normalised. No variable renames, no resampling, no vocabulary mapping, no unit conversion. Every transformation belongs downstream. The one reshaping this tool does perform is the scatter onto a dense axis at the dataset's declared cadence — with the geometry → station dimension rename that comes with it — and that is recorded in the file and is reversible.

Dense (station, time), not ragged. With shuffle + gzip the NaN runs cost almost nothing, and xarray opens the result as a 2-D array directly. CF ragged arrays would save a little space and cost real ergonomics, since xarray does not decode them natively.

Packing is explicit. h5netcdf does not apply CF scale_factor on write, so the archiver packs integers itself. That is the point: the rounding is testable, and the integers written are verified against the source objects rather than trusted to a library.

shuffle=True everywhere. Measured on the real lake dataset, whole file: 3.16 MB without it (1.50× the source) against 1.27 MB with it (0.61×). The time axis alone goes 1.80 MB → 0.16 MB.

Nothing dense is materialised. Rows are written one station at a time into pre-created chunked variables, so peak memory is one station's row. groundwater_depth is 376 M cells; the dense planes would otherwise be ~5 GB.

Stations are fetched one at a time, not in batches. Batching is faster, but a batched get_results returns the union of the batch's timestamps with NaN padding, which makes a station's genuinely-stored steps indistinguishable from concat padding — and the count gate needs exactly that distinction. Chunk downloads are already threaded within a single station's call, so most of the parallelism survives.

Resumability is the chunk cache, not checkpoint files. A crash costs the rebuild of one .nc; the re-fetch is nearly free. Note tethysts.clear_cache never deletes anything (it globs *.nc while the writer writes *.h5) — manage the cache directory by hand if you run the whole estate.

Verification

--verify runs three gates:

Plus, on every write: a packed-range audit (logged even when it passes — see the gage_height warning in PROVENANCE.md), a uniqueness assertion on the scattered timestamps, and a refusal to store a finite value that packs onto the fill.

The uniqueness assertion belongs at the write side rather than in a gate, and that is not a style preference: the scatter vals[pos] = values is last-wins on duplicate indices while the count gate reports pos.size, which counts a duplicate with multiplicity. A repeated timestamp therefore destroys an observation while every count still reconciles. Both review arms found this independently; one constructed it end-to-end through unmodified gate code.

The round-trip gate compares against the source objects rather than against tethysts output on purpose. An earlier prototype compared client-decoded to archive-decoded — both sides of the same decode path — and so could not have caught an error introduced by the client. There was one: see modified_date in PROVENANCE.md.

Scope

v1 covers regular-cadence, single-height, time_series datasets, written locally.

Not covered: irregular datasets (frequency_interval is the literal string 'None' — 30 of ECan's 36 QC datasets), which have no axis to build and are blocked on a cfdb representation; grid result types; and any push to remote storage. See OPEN_WORK.md.

Read-only, always

The archiver has no write path to the object store and takes no credentials. This matters more than it looks: the tethys prefix and live envlib member data share the ecan-env-monitoring bucket, and the tethys copy of the quality-controlled record is the only one that exists. remote.check_read_only refuses a remote carrying connection_config.