marinerg-data 0.1.0

Facility-side MARINERG-i data client: inspect, convert, validate, publish, and fetch marine test-facility datasets.

marinerg-data

Facility-side data client for the MARINERG-i e-infrastructure: inspect → convert → validate → publish, plus VRE consumption (search, resolve, fetch, lazy streaming). Django-free and pip-installable, so it runs both at a test facility and inside a Blue-Cloud / D4Science VRE Jupyter image.

The invariant that shapes everything: bulk data never transits the e-infrastructure. Facilities publish to their own Zenodo; this tool registers the record in CKAN; the server only re-validates metadata.

Install

pip install marinerg-data                 # core client (inspect/validate/publish)
pip install "marinerg-data[kerchunk]"     # + kerchunk reference sidecars
pip install "marinerg-data[mock]"         # + synthetic scenario generator
pip install marinerg-wave-tank            # optional wave-tank converter family

Commands

Command	Purpose
inspect <dir>	Sniff capture formats, report converter families; --draft-manifest writes a skeleton
convert <run-dir>	Run a converter family (--family wave-tank, else auto-detect); writes open-format outputs + <run>.manifest.json, then validates. With [kerchunk], also emits a <file>.nc.kerchunk.json reference sidecar per NetCDF
validate <manifest-or-dir>	JSON Schema (run-manifest v0.1) + referential integrity + file existence; --verify-checksums, --no-files
publish <bundle-dir>	Facility Zenodo record + CKAN registration; --sandbox, --new-version, --dry-run
mock --scenario <name>	Synthetic NetCDF + manifest for fixtures/seeding/demos (--scenario list)

Environment: ZENODO_TOKEN (a sandbox token with --sandbox), CKAN_URL, CKAN_TOKEN.

Cloud-optimised access (VRE)

With the [kerchunk] extra, convert emits a kerchunk reference sidecar per NetCDF. The sidecar lets xarray lazily stream individual chunks straight from Zenodo over HTTP range requests — open a 10 GB campaign, plot five minutes of one gauge, transfer megabytes not gigabytes. The reference URL is templated ({{u}}) so the sidecar is portable; it is retargeted to the Zenodo file URL at publish.

Develop

uv sync --group dev
uv run pytest          # Django-free, no network (HTTP is faked)
uv run ruff check src tests

Python 3.14, uv-native toolchain. See CLAUDE.md for architecture pointers and the token-free release flow.

Licence

Copyright ICHEC. GNU AGPL v3 or later. Exemptions available for MARINERG-i project partners.