CELINE Utils

A collection of shared utilities, libraries, and command-line tools that form the technical backbone of the CELINE data platform. Provides reusable building blocks for data pipelines, governance, lineage, metadata management, and platform integrations.

Not an end-user application — a platform utility layer embedded into CELINE applications and executed within orchestrated environments using Meltano, dbt, Prefect, and OpenLineage.

Scope and goals

  • Centralise cross-cutting platform logic used by multiple CELINE projects
  • Provide opinionated but extensible tooling for data pipelines
  • Enforce consistent governance and lineage semantics
  • Reduce duplication across pipeline applications
  • Act as a stable foundation for CELINE-compatible services and workflows

Key capabilities

Governance framework

A declarative governance.yaml specification defines the metadata, access control, and dataspace exposure rules for each dataset.

The GovernanceRule model covers:

  • Dataset ownership (owner, attribution)
  • License and access level (open, internal, restricted, secret)
  • Data classification (pii, green, yellow, red) and retention
  • Tags, documentation links, and source system
  • row_filters — list of filter specs ([{handler, args}]) for per-subject consent-based row filtering
  • expose: true — controls whether the dataset appears in the DCAT catalogue and is registered as an EDC asset

Extended blocks for DCAT-AP 3.0 and dataspace integration:

dcat: block — propagated to the DCAT-AP catalogue by dataset-api:

  • publisher_uri — overrides the API-level fallback publisher
  • themes — EU Publications Office data-theme URIs
  • language_uris — dct:language URIs
  • spatial_uris — dct:spatial URIs
  • accrual_periodicity — dct:accrualPeriodicity URI
  • conforms_to — dct:conformsTo URI
  • temporal.start / temporal.end — dct:temporal coverage

dataspace: block — consumed by export_governance.py when registering datasets in EDC:

  • contract_required — enables ds:contractRequired ODRL constraint
  • consent_required — enables ds:consentStatus ODRL constraint and consent-based row filtering
  • odrl_action — default ODRL action (default use)
  • purpose — ODRL purpose values
  • medallion — data quality level (gold / silver / bronze)

Governance rules are resolved with pattern matching via GovernanceResolver — defaults cascade from the defaults: block into each source entry, with per-source values taking precedence. The expose and dcat/dataspace fields use an OR-merge for booleans and override-merge for objects.

Both celine-utils (pipeline side) and dataset-api/cli/export_governance.py (catalogue side) parse the same governance.yaml format. EDC-specific sub-objects in the dataspace: block are silently ignored by celine-utils via model_config = ConfigDict(extra="ignore").

Pipeline orchestration

Structured execution layer for:

  • Meltano ingestion pipelines
  • dbt transformations, tests, and seeds (dbt_seed wrapper)
  • Prefect-based Python flows (Prefect 3.x)

The PipelineRunner coordinates execution, logging, error handling, and lineage emission consistently across tools. Pipeline run environment variables can be injected via pipeline_run_envs for local runtime configuration.

See the pipeline tutorial.

OpenLineage integration

  • Automatic emission of START, COMPLETE, FAIL, and ABORT events
  • Dataset-level schema facets
  • Data quality assertions from dbt tests
  • Custom CELINE governance facets (including row_filters, medallion, classification)

Dataset tooling

The DatasetClient enables:

  • Schema and table introspection
  • Column metadata inspection
  • Safe query construction
  • Export to Pandas

Platform integrations

  • Keycloak for identity and access management
  • Apache Superset for analytics platform integration
  • MQTT for lightweight messaging

CLI

celine-utils governance generate   # generate governance.yaml template
celine-utils pipeline init         # scaffold a new pipeline
celine-utils pipeline run          # run a pipeline

Repository structure

celine/
  admin/
  cli/
  common/
  datasets/
  pipelines/
schemas/
tests/

Configuration

Environment-driven via pydantic-settings:

  • Environment variables first
  • Optional .env files
  • Typed validation with container-friendly defaults

Documentation

  • Pipeline Tutorial — end-to-end pipeline setup guide
  • Governance — governance.yaml format, access levels, pattern matching, dcat/dataspace blocks
  • Schemas — JSON Schema definitions including governance.schema.json
  • CLI — full CLI reference

Installation

uv add celine-utils

Intended audience

  • Data engineers
  • Platform engineers
  • CELINE application developers

License

Copyright © 2025 Spindox Labs

Licensed under the Apache License, Version 2.0.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (2.5.0)

Owner

PyPI data
Data sourced directly from PyPI's database.

Requires

Python >=3.12

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

celine_utils-1.18.1.tar.gz (55.0 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

celine_utils-1.18.1-py3-none-any.whl (69.4 kB view details)

Uploaded Python 3

File details

Details for the file celine_utils-1.18.1.tar.gz.

File metadata

  • Download URL: celine_utils-1.18.1.tar.gz
  • Upload date:
  • Size: 55.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for celine_utils-1.18.1.tar.gz
Algorithm Hash digest
SHA256 4649a09bcb4371c70eb779f58a152ce6b989ff34d0514ffc369d983dfc80c328
MD5 2f1c5caf967e18ed81e20b090f233592
BLAKE2b-256 54a410c356a46106de69ccd19682c62b929e0b1634263112eec5b9c48342416e

See more details on using hashes here.

Provenance

The following attestation bundles were made for celine_utils-1.18.1.tar.gz:

Publisher: release.yaml on celine-eu/celine-utils

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file celine_utils-1.18.1-py3-none-any.whl.

File metadata

  • Download URL: celine_utils-1.18.1-py3-none-any.whl
  • Upload date:
  • Size: 69.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for celine_utils-1.18.1-py3-none-any.whl
Algorithm Hash digest
SHA256 ef1a3279a488b52dcf35a57ec72419e2b25e971cad7e986e668f0c4738c4956a
MD5 4022856475f556769f3ec6bd414fc4fe
BLAKE2b-256 9d275fc0ea709ff4d4661857cae2bf543bfd568687dab92bc1ecc58b8f1e790b

See more details on using hashes here.

Provenance

The following attestation bundles were made for celine_utils-1.18.1-py3-none-any.whl:

Publisher: release.yaml on celine-eu/celine-utils

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

2.5.0

2 files

2.4.0

2 files

2.3.0

2 files

2.2.0

2 files

2.1.0

2 files

2.0.1

2 files

2.0.0

2 files

This release

1.18.1 This release

2 files

1.18.0

2 files

1.17.0

2 files

1.16.0

2 files

1.15.0

2 files

1.14.3

2 files

1.14.2

2 files

1.14.1

2 files

1.14.0

2 files

1.13.0

2 files

1.12.0

2 files

1.11.0

2 files

1.10.2

2 files

1.10.1

2 files

1.10.0

2 files

1.9.4

2 files

1.9.3

2 files

1.9.2

2 files

1.9.1

2 files

1.9.0

2 files

1.8.5

2 files

1.8.4

2 files

1.8.3

2 files

1.8.2

2 files

1.8.1

2 files

1.8.0

2 files

1.7.1

2 files

1.7.0

2 files

1.6.3

2 files

1.6.2

2 files

1.6.1

2 files

1.6.0

2 files

1.5.0

2 files

1.4.4

2 files

1.4.3

2 files

1.4.2

2 files

1.4.1

2 files

1.4.0

2 files

1.3.0

2 files

1.2.0

2 files

1.1.1

2 files

1.1.0

2 files

1.0.9

2 files

1.0.8

2 files

1.0.7

2 files

1.0.6

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page