marinerg-data

Facility-side data client for the MARINERG-i e-infrastructure: inspect → convert → validate → publish, plus VRE consumption (search, resolve, fetch, lazy streaming). Django-free and pip-installable, so it runs both at a test facility and inside a Blue-Cloud / D4Science VRE Jupyter image.

The invariant that shapes everything: bulk data never transits the e-infrastructure. Facilities publish to their own Zenodo; this tool registers the record in CKAN; the server only re-validates metadata.

Install

pip install marinerg-data                 # core client (inspect/validate/publish)
pip install "marinerg-data[kerchunk]"     # + kerchunk reference sidecars
pip install "marinerg-data[mock]"         # + synthetic scenario generator
pip install "marinerg-data[vre]"          # + complete Blue-Cloud / Jupyter runtime
pip install marinerg-wave-tank            # optional wave-tank converter family

The lightweight core client supports Python 3.11 or newer and has no notebook, NetCDF, or container dependency. kerchunk, mock, and vre are explicit opt-ins. Windows is a supported installation target for the core client; the release gate includes a Windows install-and-validate smoke test.

Commands

Environment: ZENODO_TOKEN (a sandbox token with --sandbox), CKAN_URL, CKAN_TOKEN.

Onboarding a facility

Equipment templates describe how a vendor writes its export — header style, delimiter, the metadata keys it invents, what its columns mean — so a new format is a YAML file rather than a converter. Facility templates hold what a facility restates on every campaign: identity, ROR, site, default rights. Campaign templates say which instruments a test report spans.

marinerg-data template list
marinerg-data template check qualisys-tsv ./run01.tsv

Sample raw handovers are in demo/raw/ — small enough to read, and realistic: every instrument a different shape, the sea state only in the operator's spreadsheet, and the clocks disagreeing.

Publishing is resumable

A Zenodo record cannot be withdrawn, so publish records each step before the one it cannot undo. The deposition goes into <bundle>/.marinerg-publish-pending.json before it is published, and the minted DOI into <bundle>/.marinerg-publish.json before CKAN is called. Rerun publish on the same bundle after any failure — it resumes from wherever it stopped and never mints a second record for the same data, including when a publish request times out after Zenodo has committed. Both files are local bookkeeping and are never uploaded.

Cloud-optimised access (VRE)

With the [kerchunk] extra, convert emits a kerchunk reference sidecar per NetCDF. The sidecar lets xarray lazily stream individual chunks straight from Zenodo over HTTP range requests — open a 10 GB campaign, plot five minutes of one gauge, transfer megabytes not gigabytes. The reference URL is templated ({{u}}) so the sidecar is portable; it is retargeted to the Zenodo file URL at publish.

Examples

Runnable VRE notebooks in examples/: lazy kerchunk streaming and intake-based discovery, each self-contained against synthetic data. See examples/README.md.

Blue-Cloud / D4Science handover

The operator-facing install profile, CKAN/DCAT endpoint contract, and live-demo release checklist are in docs/blue-cloud-handover.md. The VRE needs Python 3.11 or newer, outbound HTTPS to the MARINERG-i catalogue, Zenodo, and PyPI, and persistent shared workspace storage. No MARINERG-i data hosting, inbound connectivity, or identity federation is required for the public-data workflow.

Develop

uv sync --group dev
uv run pytest          # Django-free, no internet (HTTP is faked; the Zenodo
                       # contract test spawns a local zenodo-mock)
uv run ruff check src tests

Development and CI use Python 3.14 with the uv-native toolchain; the published client supports Python 3.11 or newer. See CLAUDE.md for architecture pointers and the token-free release flow.

Licence

Copyright ICHEC. MIT — see LICENSE.