CELINE Utils

A collection of shared utilities, libraries, and command-line tools that form the technical backbone of the CELINE data platform. Provides reusable building blocks for data pipelines, governance, lineage, metadata management, and platform integrations.

Not an end-user application — a platform utility layer embedded into CELINE applications and executed within orchestrated environments using Meltano, dbt, Prefect, and OpenLineage.

Scope and goals

  • Centralise cross-cutting platform logic used by multiple CELINE projects
  • Provide opinionated but extensible tooling for data pipelines
  • Enforce consistent governance and lineage semantics
  • Reduce duplication across pipeline applications
  • Act as a stable foundation for CELINE-compatible services and workflows

Key capabilities

Governance framework

A declarative governance.yaml specification defines the metadata, access control, and dataspace exposure rules for each dataset.

The GovernanceRule model covers:

  • Dataset ownership (owner, attribution)
  • License and access level (open, internal, restricted, secret)
  • Data classification (pii, green, yellow, red) and retention
  • Tags, documentation links, and source system
  • row_filters — list of filter specs ([{handler, args}]) for per-subject consent-based row filtering
  • expose: true — controls whether the dataset appears in the DCAT catalogue and is registered as an EDC asset

Extended blocks for DCAT-AP 3.0 and dataspace integration:

dcat: block — propagated to the DCAT-AP catalogue by dataset-api:

  • publisher_uri — overrides the API-level fallback publisher
  • themes — EU Publications Office data-theme URIs
  • language_uris — dct:language URIs
  • spatial_uris — dct:spatial URIs
  • accrual_periodicity — dct:accrualPeriodicity URI
  • conforms_to — dct:conformsTo URI
  • temporal.start / temporal.end — dct:temporal coverage

dataspace: block — consumed by export_governance.py when registering datasets in EDC:

  • contract_required — enables ds:contractRequired ODRL constraint
  • consent_required — enables ds:consentStatus ODRL constraint and consent-based row filtering
  • odrl_action — default ODRL action (default use)
  • purpose — ODRL purpose values
  • medallion — data quality level (gold / silver / bronze)

Governance rules are resolved with pattern matching via GovernanceResolver — defaults cascade from the defaults: block into each source entry, with per-source values taking precedence. The expose and dcat/dataspace fields use an OR-merge for booleans and override-merge for objects.

Both celine-utils (pipeline side) and dataset-api/cli/export_governance.py (catalogue side) parse the same governance.yaml format. EDC-specific sub-objects in the dataspace: block are silently ignored by celine-utils via model_config = ConfigDict(extra="ignore").

Pipeline orchestration

Structured execution layer for:

  • Meltano ingestion pipelines
  • dbt transformations, tests, and seeds (dbt_seed wrapper)
  • Prefect-based Python flows (Prefect 3.x)

The PipelineRunner coordinates execution, logging, error handling, and lineage emission consistently across tools. Pipeline run environment variables can be injected via pipeline_run_envs for local runtime configuration.

See the pipeline tutorial.

OpenLineage integration

  • Automatic emission of START, COMPLETE, FAIL, and ABORT events
  • Dataset-level schema facets
  • Data quality assertions from dbt tests
  • Custom CELINE governance facets (including row_filters, medallion, classification)

Dataset tooling

The DatasetClient enables:

  • Schema and table introspection
  • Column metadata inspection
  • Safe query construction
  • Export to Pandas

Platform integrations

  • Keycloak for identity and access management
  • Apache Superset for analytics platform integration
  • MQTT for lightweight messaging

CLI

celine-utils governance generate   # generate governance.yaml template
celine-utils pipeline init         # scaffold a new pipeline
celine-utils pipeline run          # run a pipeline

Repository structure

celine/
  admin/
  cli/
  common/
  datasets/
  pipelines/
schemas/
tests/

Configuration

Environment-driven via pydantic-settings:

  • Environment variables first
  • Optional .env files
  • Typed validation with container-friendly defaults

Documentation

  • Pipeline Tutorial — end-to-end pipeline setup guide
  • Governance — governance.yaml format, access levels, pattern matching, dcat/dataspace blocks
  • Schemas — JSON Schema definitions including governance.schema.json
  • CLI — full CLI reference

Installation

uv add celine-utils

Intended audience

  • Data engineers
  • Platform engineers
  • CELINE application developers

License

Copyright © 2025 Spindox Labs

Licensed under the Apache License, Version 2.0.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (2.5.0)

Owner

PyPI data
Data sourced directly from PyPI's database.

Requires

Python >=3.12

Provides Extra

pipelines openlineage admin all

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

celine_utils-2.0.1.tar.gz (79.5 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

celine_utils-2.0.1-py3-none-any.whl (88.7 kB view details)

Uploaded Python 3

File details

Details for the file celine_utils-2.0.1.tar.gz.

File metadata

  • Download URL: celine_utils-2.0.1.tar.gz
  • Upload date:
  • Size: 79.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for celine_utils-2.0.1.tar.gz
Algorithm Hash digest
SHA256 9d44b7b66baa175579f397087fa3fc8f5024b8f3477af129fcd437f841075270
MD5 36f2961596db3f973981ca11a194ccf2
BLAKE2b-256 0bd11d0cbf78e4591b5ea50c848371c75c830e516d27de653fb012e4314671c3

See more details on using hashes here.

Provenance

The following attestation bundles were made for celine_utils-2.0.1.tar.gz:

Publisher: release.yaml on celine-eu/celine-utils

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file celine_utils-2.0.1-py3-none-any.whl.

File metadata

  • Download URL: celine_utils-2.0.1-py3-none-any.whl
  • Upload date:
  • Size: 88.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for celine_utils-2.0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 4ee15d51f2619143ba7b880c072090c9bb7cfd0582056329473d8dbf692a5c8d
MD5 294bc4beca28c732dd368643380e3b7d
BLAKE2b-256 18a85bf0b85e63ac94c6c2c1a8504d24e6957db8bf9467b21c94db01fb50a065

See more details on using hashes here.

Provenance

The following attestation bundles were made for celine_utils-2.0.1-py3-none-any.whl:

Publisher: release.yaml on celine-eu/celine-utils

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

2.5.0

2 files

2.4.0

2 files

2.3.0

2 files

2.2.0

2 files

2.1.0

2 files

This release

2.0.1 This release

2 files

2.0.0

2 files

1.18.1

2 files

1.18.0

2 files

1.17.0

2 files

1.16.0

2 files

1.15.0

2 files

1.14.3

2 files

1.14.2

2 files

1.14.1

2 files

1.14.0

2 files

1.13.0

2 files

1.12.0

2 files

1.11.0

2 files

1.10.2

2 files

1.10.1

2 files

1.10.0

2 files

1.9.4

2 files

1.9.3

2 files

1.9.2

2 files

1.9.1

2 files

1.9.0

2 files

1.8.5

2 files

1.8.4

2 files

1.8.3

2 files

1.8.2

2 files

1.8.1

2 files

1.8.0

2 files

1.7.1

2 files

1.7.0

2 files

1.6.3

2 files

1.6.2

2 files

1.6.1

2 files

1.6.0

2 files

1.5.0

2 files

1.4.4

2 files

1.4.3

2 files

1.4.2

2 files

1.4.1

2 files

1.4.0

2 files

1.3.0

2 files

1.2.0

2 files

1.1.1

2 files

1.1.0

2 files

1.0.9

2 files

1.0.8

2 files

1.0.7

2 files

1.0.6

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page