marinerg-data 0.2.0

Facility-side MARINERG-i data client: inspect, convert, validate, publish, and fetch marine test-facility datasets.

marinerg-data

Facility-side data client for the MARINERG-i e-infrastructure: inspect → convert → validate → publish, plus VRE consumption (search, resolve, fetch, lazy streaming). Django-free and pip-installable, so it runs both at a test facility and inside a Blue-Cloud / D4Science VRE Jupyter image.

The invariant that shapes everything: bulk data never transits the e-infrastructure. Facilities publish to their own Zenodo; this tool registers the record in CKAN; the server only re-validates metadata.

Install

pip install marinerg-data                 # core client (inspect/validate/publish)
pip install "marinerg-data[kerchunk]"     # + kerchunk reference sidecars
pip install "marinerg-data[mock]"         # + synthetic scenario generator
pip install "marinerg-data[vre]"          # + complete Blue-Cloud / Jupyter runtime
pip install marinerg-wave-tank            # optional wave-tank converter family

The lightweight core client supports Python 3.11 or newer and has no notebook, NetCDF, or container dependency. kerchunk, mock, and vre are explicit opt-ins. Windows is a supported installation target for the core client; the release gate includes a Windows install-and-validate smoke test.

Commands

Command	Purpose
inspect <dir>	Sniff capture formats, report converter families; --draft-manifest writes a skeleton
convert <run-dir>	Run a converter family (--family wave-tank, else auto-detect); writes open-format outputs + <run>.manifest.json, then validates. With [kerchunk], also emits a <file>.nc.kerchunk.json reference sidecar per NetCDF
validate <manifest-or-dir>	JSON Schema (run-manifest v0.1) + referential integrity + file existence; --verify-checksums, --no-files
publish <bundle-dir>	Facility Zenodo record + CKAN registration; --sandbox, --new-version, --dry-run
fetch <doi> --cache <dir>	Download a published dataset into a content-addressed shared cache (keyed by DOI + optional manifest checksum); first lab member pays the download, the rest hit the local copy. --from-ckan <name> --ckan-url <url> resolves the DOI via CKAN; --sandbox
catalog --ckan-url <url>	Generate an intake catalogue from CKAN so MARINERG-i datasets open natively in a VRE (intake.open_catalog(...) → .to_dask()); prefers kerchunk sidecars (lazy zarr) over direct NetCDF
mock --scenario <name>	Synthetic NetCDF + manifest for fixtures/seeding/demos (--scenario list)

Environment: ZENODO_TOKEN (a sandbox token with --sandbox), CKAN_URL, CKAN_TOKEN.

Cloud-optimised access (VRE)

With the [kerchunk] extra, convert emits a kerchunk reference sidecar per NetCDF. The sidecar lets xarray lazily stream individual chunks straight from Zenodo over HTTP range requests — open a 10 GB campaign, plot five minutes of one gauge, transfer megabytes not gigabytes. The reference URL is templated ({{u}}) so the sidecar is portable; it is retargeted to the Zenodo file URL at publish.

Examples

Runnable VRE notebooks in examples/: lazy kerchunk streaming and intake-based discovery, each self-contained against synthetic data. See examples/README.md.

Blue-Cloud / D4Science handover

The operator-facing install profile, CKAN/DCAT endpoint contract, and live-demo release checklist are in docs/blue-cloud-handover.md. The VRE needs Python 3.11 or newer, outbound HTTPS to the MARINERG-i catalogue, Zenodo, and PyPI, and persistent shared workspace storage. No MARINERG-i data hosting, inbound connectivity, or identity federation is required for the public-data workflow.

Develop

uv sync --group dev
uv run pytest          # Django-free, no network (HTTP is faked)
uv run ruff check src tests

Development and CI use Python 3.14 with the uv-native toolchain; the published client supports Python 3.11 or newer. See CLAUDE.md for architecture pointers and the token-free release flow.

Licence

Copyright ICHEC. GNU AGPL v3 or later. Exemptions available for MARINERG-i project partners.