data-annotations 2.8.1

Annotate data artifacts with provenance and descriptions

data-annotations

data-annotations is a Python package for attaching provenance and structured descriptions to the files and directories your workflows produce.

It writes plain JSON annotation sidecars that are easy to inspect, archive, and publish with research outputs:

• files use artifact.ext.annotation.json
• directories use data-annotations.json at their root

Optional Markdown README sidecars can be generated for human-readable summaries.

Documentation

The full documentation is organized as a Diátaxis site: https://ceda-unibas.gitlab.io/tools/data-annotations/

• Source: https://gitlab.com/ceda-unibas/tools/data-annotations
• Changelog: https://gitlab.com/ceda-unibas/tools/data-annotations/-/blob/main/CHANGELOG.md
• Work items: https://gitlab.com/ceda-unibas/tools/data-annotations/-/work_items

Installation

Install the core library from PyPI:

pip install data-annotations

Or add it to a project with uv:

uv add data-annotations

Install CLI support when you want the data-annotations command:

pip install "data-annotations[cli]"
uv add "data-annotations[cli]"

Quick start

Decorate a function that writes an artifact. When the function runs, data-annotations records provenance and writes the JSON sidecar.

from pathlib import Path

from data_annotations.annotations import record_file_annotation
from data_annotations.description import FieldDefinition


@record_file_annotation(
    title="Participant Cohort",
    summary="Participant-level cohort assignments.",
    fields=[
        FieldDefinition(
            name="participant_id",
            data_type="string",
            summary="Stable participant identifier.",
            required=True,
            nullable=False,
        ),
    ],
    primary_key=["participant_id"],
    artifact_kind="dataset",
    write_readme=True,
)
def write_participants(artifact_path: Path, input_path: Path) -> Path:
    participant_ids = [
        line.strip()
        for line in input_path.read_text(encoding="utf-8").splitlines()[1:]
        if line.strip()
    ]
    artifact_path.parent.mkdir(parents=True, exist_ok=True)
    artifact_path.write_text(
        "participant_id\n" + "\n".join(participant_ids) + "\n",
        encoding="utf-8",
    )
    return artifact_path


artifact_path = Path("outputs") / "participants.csv"
write_participants(
    artifact_path=artifact_path,
    input_path=Path("data/raw/participants.csv"),
)

This writes:

outputs/participants.csv
outputs/participants.csv.annotation.json
outputs/participants.csv.README.md

CLI

The CLI supports retrospective annotation, provenance inspection, source recovery, and sanitized publish bundles.

data-annotations annotate file path/to/participants.csv --write-readme
data-annotations annotate directory path/to/run-001 --recursive
data-annotations provenance match path/to/participants.csv
data-annotations provenance chain path/to/participants.csv
data-annotations provenance checkout path/to/participants.csv
data-annotations publish path/to/run-001 path/to/publish-bundle

Development

From a source checkout (assuming you have Task installed):

task install
task lint
task type-check
task test

Build or preview the documentation site:

task docs-build
task docs-serve