CELINE Utils
A collection of shared utilities, libraries, and command-line tools that form the technical backbone of the CELINE data platform. Provides reusable building blocks for data pipelines, governance, lineage, metadata management, and platform integrations.
Not an end-user application — a platform utility layer embedded into CELINE applications and executed within orchestrated environments using Meltano, dbt, Prefect, and OpenLineage.
Scope and goals
- Centralise cross-cutting platform logic used by multiple CELINE projects
- Provide opinionated but extensible tooling for data pipelines
- Enforce consistent governance and lineage semantics
- Reduce duplication across pipeline applications
- Act as a stable foundation for CELINE-compatible services and workflows
Key capabilities
Governance framework
A declarative governance.yaml specification defines the metadata, access control, and dataspace exposure rules for each dataset.
The GovernanceRule model covers:
- Dataset ownership (
ownership,attribution), resolved through an owner registry - License and access level (
open,internal,restricted—secretis accepted for compatibility and normalised torestricted) - Data classification (
pii,green,yellow,red) and retention - Tags, documentation links, and source system
row_filters— list of filter specs ([{handler, args}]) for per-subject consent-based row filtering- Two exposure gates, ANDed:
exposecontrols whether the dataset is listed in the DCAT catalogue and served by the API, whiledataspace.exposecontrols whether it is offered into the dataspace as an EDC asset.exposeis tri-state — unset falls back todataspace.exposeso pre-split files keep their behaviour
Extended blocks for DCAT-AP 3.0 and dataspace integration:
dcat: block — propagated to the DCAT-AP catalogue by dataset-api:
publisher_uri— overrides the API-level fallback publisherthemes— EU Publications Office data-theme URIslanguage_uris— dct:language URIsspatial_uris— dct:spatial URIsaccrual_periodicity— dct:accrualPeriodicity URIconforms_to— dct:conformsTo URItemporal.start/temporal.end— dct:temporal coverage
dataspace: block — consumed when registering datasets in EDC:
expose— offer the dataset into the dataspacecontract_required— enablesds:contractRequiredODRL constraintconsent_required— enablesds:consentStatusODRL constraint and consent-based row filteringodrl_action— default ODRL action (defaultuse)purpose— ODRL purpose valuesmedallion— data quality level (gold / silver / bronze)
ontology: block — which mapping spec says what the columns mean: spec (a shared mapping published in celine-ontologies) or spec_file (a path relative to the governance file). Exactly one, enforced by the schema.
Governance rules are resolved with pattern matching via GovernanceResolver: exact key first, then the longest matching glob, then defaults alone. The chosen rule is overlaid on the defaults using the fields the file explicitly set — not truthiness — so expose: false withdraws a dataset instead of being silently dropped. tags and dataspace.purpose union, consent_required and contract_required OR, ownership / row_filters / ontology replace wholesale. See the format reference for the full table.
celine.governance is the single parser: dataset-api, ds and celine-superset import it rather than reimplementing it. EDC-specific sub-objects in the dataspace: block belong to ds and are ignored here via model_config = ConfigDict(extra="ignore") rather than rejected.
Pipeline orchestration
Structured execution layer for:
- Meltano ingestion pipelines
- dbt transformations, tests, and seeds (
dbt_seedwrapper) - Prefect-based Python flows (Prefect 3.x)
The PipelineRunner coordinates execution, logging, error handling, and lineage emission consistently across tools. Pipeline run environment variables can be injected via pipeline_run_envs for local runtime configuration.
See the pipeline tutorial.
OpenLineage integration
- Automatic emission of START, COMPLETE, FAIL, and ABORT events
- Dataset-level schema facets
- Data quality assertions from dbt tests
- Custom CELINE governance facets (including
row_filters,medallion,classification)
Dataset tooling
The DatasetClient enables:
- Schema and table introspection
- Column metadata inspection
- Safe query construction
- Export to Pandas
Platform integrations
- MQTT pipeline run events, published through
celine-sdk - Keycloak client-credentials tokens, for authenticating to a protected Marquez
Keycloak and Superset administration was removed in 2.3.0 along with the
celine-utils admin command tree — it was a replicable-setup tool with no remaining
callers. Provisioning lives in celine-policies.
CLI
celine-utils governance generate marquez --app <app> # scaffold governance.yaml from Marquez
celine-utils pipeline init app <name> # scaffold a new pipeline app
celine-utils pipeline run (meltano | dbt | prefect) # run a pipeline stage
Full reference: CLI.
Repository structure
src/celine/
governance/ the thin core — three dependencies, imported by other repositories
utils/
cli/
common/
datasets/
pipelines/
schema/ JSON Schemas — published, and symlinked into the package
docs/
tests/
integration-tests/
Configuration
Environment-driven via pydantic-settings:
- Environment variables first
- Optional
.envfiles - Typed validation with container-friendly defaults
Documentation
Two tracks — see the documentation index.
Governance as a library
- The governance library —
celine.governanceAPI, the three-dependency contract, merging, validation governance.yamlformat — the grammar, resolution, the two exposure gatesowners.yamlregistry — owner aliases to canonical identities- Schemas — the three published JSON Schemas
Building and running pipelines
- Pipeline Tutorial — end-to-end pipeline setup guide
- CLI — full CLI reference
- Environment — every variable and its default
Why choices were made — Decisions.
Installation
uv add celine-utils # governance only — three dependencies
uv add "celine-utils[pipelines]" # dbt / Meltano / Prefect / lineage
uv add "celine-utils[all]" # pipelines + the typed OpenLineage facet
Parsing governance.yaml needs no extras. That is deliberate and enforced by CI —
see ADR-0001.
Intended audience
- Data engineers
- Platform engineers
- CELINE application developers
License
Copyright © 2025 Spindox Labs
Licensed under the Apache License, Version 2.0.
Metadata
Release files for celine-utils 2.6.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| celine_utils-2.6.2.tar.gz | 123.3 kB | With provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| celine_utils-2.6.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 230.6 kB
Release files / celine_utils-2.6.2.tar.gz
| Download URL | celine_utils-2.6.2.tar.gz |
|---|---|
| Size | 123.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
555361277fd899d0fb48a3674c5998504d6fa8508428833e0dea7520f801954e
|
|
BLAKE2b-256 checksum How to use checksums |
35ab4ba808ca96385c088bc9e9c8b48acdfa8de12d6df049100704f4d09b7320
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 12, 2026.
Transparency logRelease files / celine_utils-2.6.2-py3-none-any.whl
| Download URL | celine_utils-2.6.2-py3-none-any.whl |
|---|---|
| Size | 107.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ac73f6ce43a86c7c8a86ae9f5b2f4459c4e2c8ac081905a2b197ecf5043bea46
|
|
BLAKE2b-256 checksum How to use checksums |
fe736c8e5b83c7fcb56d8ec9b63e4fffc86653358a9d84903dc5646c220a4f78
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 12, 2026.
Transparency logSecurity for celine-utils 2.6.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 2 files in this release have been uploaded with provenance metadata.
Attestation for celine_utils-2.6.2.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 12, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
