fair-metadata-integration

Integrate and package all metadata into RO-Crate, linking to other metadata files.

Given a Bilayers workflow specification and the image data an analysis consumed and produced, this package writes a Workflow RO-Crate: an ro-crate-metadata.json describing the run, alongside copies of the files it describes.

Installation

pip install fair-metadata-integration

Requires Python 3.12 or newer. rocrate is installed automatically.

Usage

from fair_metadata_integration import create_ro_crate

create_ro_crate(
    dest_path="crate",
    workflow_schema_filename="config.yaml",
    input_path="input.ome.zarr",
    output_paths=["output.ome.zarr"],
)

• dest_path — directory the crate is written to, created if it does not exist.
• workflow_schema_filename — the Bilayers config.yaml describing the workflow.
• input_path — the input image (an OME-Zarr directory or a TIFF file).
• output_paths — the images the workflow produced. Each gets its own parameter.
• gen_preview — write an ro-crate-preview.html alongside the metadata (default True).

The source files are copied into the crate, so dest_path ends up self-contained:

crate/
├── config.yaml
├── input.ome.zarr/
├── output.ome.zarr/
├── ro-crate-metadata.json
└── ro-crate-preview.html

ro-crate-preview.html is the human-readable RO-Crate Website for the crate, a static HTML page listing the crate and its data entities, generated by rocrate. It is described in the metadata as a CreativeWork about the root dataset, and is not part of the root dataset's hasPart.

What the crate contains

The workflow specification is the crate's mainEntity, typed as a ComputationalWorkflow. The metadata descriptor declares RO-Crate 1.3, and the root dataset declares the Workflow RO-Crate 1.1 profile:

{
  "@id": "ro-crate-metadata.json",
  "@type": "CreativeWork",
  "about": { "@id": "./" },
  "conformsTo": { "@id": "https://w3id.org/ro/crate/1.3" }
},
{
  "@id": "./",
  "@type": "Dataset",
  "conformsTo": [
    { "@id": "https://w3id.org/workflowhub/workflow-ro-crate/1.1" }
  ],
  "mainEntity": { "@id": "config.yaml" }
}

The workflow's programmingLanguage is a ComputerLanguage entity for the Bilayers specification, identified by its Zenodo concept DOI 10.5281/zenodo.17652333, which resolves to the latest release of the spec.

Inputs and outputs are modelled as FormalParameter entities carrying an EDAM format IRI, from the file name: format_3915 for Zarr, format_3727 for OME-TIFF, format_3591 for TIFF, and PNG, JPEG, HDF5, CSV, TSV, xlsx, JSON, YAML, XML and text. Folders (e.g. OME-Zarr) are Dataset entities, files File entities, copied with their content. Each data entity points back at the parameter it realises via exampleOfWork:

{
  "@id": "input.ome.zarr/",
  "@type": "Dataset",
  "additionalType": "http://edamontology.org/format_3915",
  "exampleOfWork": { "@id": "#input" }
}

The crate's JSON-LD context is extended with terms for describing biological imaging metadata — biosample, specimen, organism_classification, preparation_method, channel and others — so those properties can be added to the crate's entities.

Development

git clone https://github.com/NL-BioImaging/fair-metadata-integration.git
cd fair-metadata-integration
pip install -e ".[test]"
pytest

Releasing

The package version comes from the git tag, via setuptools-scm; there is no version to bump in pyproject.toml.

1. Optionally, do a dry run: run the Publish to PyPI workflow manually from the Actions tab. It builds, tests and uploads a development version (e.g. 0.2.4.dev2) to TestPyPI.
2. Create a GitHub release with a new tag, e.g. v0.2.4, on the commit to release. Publishing the release builds, tests and uploads that version to PyPI.

PyPI never accepts the same version twice. If a release fails after uploading, release a new version rather than reusing the tag.

License

MIT