marinerg-data

marinerg-data is the command-line client for the MARINERG-i data catalogue. A test facility uses it to turn instrument exports into standard NetCDF, check them, and publish them. A researcher uses it to find and download what has been published.

How it fits together

Data files never pass through MARINERG-i. A facility deposits its files in its own Zenodo account, which gives the dataset a DOI. This tool then registers the dataset in the MARINERG-i catalogue, and fills in the record from the files. The catalogue's own docs explain how the catalogue works.

Facilities differ in how organised their data is. The tool works at every level:

Install

pip install "marinerg-data[convert,kerchunk]"   # to convert and publish
pip install "marinerg-data[vre]"                # to use data in a VRE

[convert] reads exports and writes NetCDF. [kerchunk] adds a small index beside each NetCDF, so a VRE can stream part of a file from Zenodo. Without either, the client still validates and publishes files that are already converted.

Publish your data

Publishing takes four commands. Each one says what to run next.

1. Inspect. marinerg-data inspect <folder> says which equipment template matches each export. A template describes how one instrument writes its files: the delimiter, the decimal separator, the units, and which column is time. If nothing matches, write a template first, as Templates explains.

2. Convert. marinerg-data convert <folder> reads each export through its template and writes a NetCDF file and a run manifest, which describes the run. It stops, naming the column and row, at anything the template does not settle, such as a unit the file and the template disagree on.

   For a test campaign with several instruments, keep a folder per instrument and convert them together with --campaign. Each run then gets one manifest over every instrument, with each clock's offset applied. It also drafts dataset.yaml, the metadata for the catalogue record. Read it and edit it: the tool never overwrites it.

3. Validate. marinerg-data validate <folder> checks the manifests and dataset.yaml against the vocabularies the catalogue uses, so a mistake is caught before anything is uploaded.

4. Publish. Set ZENODO_TOKEN to your Zenodo token, CKAN_URL to the catalogue's address and CKAN_TOKEN to your catalogue API token. Then run marinerg-data publish <folder> --sandbox to try it on sandbox.zenodo.org, and drop --sandbox when it looks right. --new-version publishes a correction as a new version of the same record.

If a publish fails part-way, run the same command again. It carries on from where it stopped, and never mints a second DOI for the same data.

Try it without data

demo/ walks the publish path with generated data, so you can see every step before using your own files.

Use published data

marinerg-data fetch <doi-or-name> --cache <folder> downloads a dataset into a shared cache, so a lab downloads each file once. A name, rather than a DOI, is looked up in the catalogue at CKAN_URL.

marinerg-data catalog writes an intake catalogue of the published datasets, for opening them in Python. examples/ shows both in Jupyter and in a Virtual Research Environment.

Templates

The package ships equipment templates for common instruments; marinerg-data template list shows them. For an instrument without one:

1. marinerg-data template draft <export> guesses a template from a sample file and writes it beside the file. The guesses are only a start: check every value, and replace each TODO. The template will not load until you have. Each channel names what it measures with a parameter code; marinerg-data template parameters lists them.
2. marinerg-data template check <template-file> <export> reads the export through it and reports what it mapped and what it could not.
3. Keep your templates in a folder, and pass it to every command with --templates <folder>. Templates in your user config folder are loaded too (~/.config/marinerg-data/templates on Linux).

A campaign also needs a facility template, with the facility's name, ROR and default licence, and a campaign template, listing the instruments and naming the operator's log. Keep them in facilities/ and campaigns/ inside your templates folder. The fixtures in tests/fixtures/templates are worked examples of all three kinds.

Give each run's start time in the operator's log, so the NetCDF counts time from it, as CF requires. Most instruments record only elapsed time. The start is the moment the reference clock, the one the clock offsets are measured against, read zero. Take it from the operator's notes or the wavemaker's log. Name the column in the campaign template's log, with start_time, and give timezone unless each time carries its own offset. Without a start time, convert warns, and time stays elapsed seconds.

Develop

uv sync --group dev
uv run tox

uv run tox runs the format check, lint, type check and tests, as CI does.

marinerg-data dev generates test and demo data. It is left out of --help; marinerg-data dev --help lists what it does.

Release

Open the newest pipeline for main and press the play button on release patch, minor or major. The release pipeline publishes to PyPI, then tags the commit and creates the GitLab release.

Licence

Copyright Irish Centre for High End Computing, 2026. MIT. See LICENSE.