celine-utils 1.6.3

CELINE utils

CELINE Utils

CELINE Utils is a collection of shared utilities, libraries, and command-line tools that form the technical backbone of the CELINE data platform.

It provides reusable building blocks for data pipelines, governance, lineage, metadata management, and platform integrations. The repository is designed to be embedded into CELINE applications and executed within orchestrated environments using Meltano, dbt, Prefect, and OpenLineage.

---

Scope and Goals

The goals of this repository are to:

• Centralize cross-cutting platform logic used by multiple CELINE projects
• Provide opinionated but extensible tooling for data pipelines
• Enforce consistent governance and lineage semantics
• Reduce duplication across pipeline applications
• Act as a stable foundation for CELINE-compatible services and workflows

This is not an end-user application; it is a platform utility layer.

---

Key Capabilities

Command Line Interface (CLI)

A unified CLI built with Typer exposes administrative, governance, and pipeline utilities:

celine-utils
 ├── governance
 │    └── generate
 └── pipeline
      ├── init
      └── run

---

Pipeline Orchestration

CELINE Utils provides a structured execution layer for:

• Meltano ingestion pipelines
• dbt transformations and tests
• Prefect-based Python flows

The PipelineRunner coordinates execution, logging, error handling, and lineage emission in a consistent way across tools.

See the pipeline tutorial to discover how to setup and deploy a new pipeline.

---

OpenLineage Integration

First-class OpenLineage support includes:

• Automatic emission of START, COMPLETE, FAIL, and ABORT events
• Dataset-level schema facets
• Data quality assertions from dbt tests
• Custom CELINE governance facets

---

Governance Framework

A declarative governance.yaml specification allows you to define:

• Dataset ownership
• License and access level
• Classification and retention
• Tags and documentation links

Governance rules are resolved using pattern matching and injected into lineage events.

---

Dataset Tooling

The DatasetClient enables:

• Schema and table introspection
• Column metadata inspection
• Safe query construction
• Export to Pandas

---

Platform Integrations

Built-in integrations include:

• Keycloak for identity and access management
• Apache Superset for analytics platform integration
• MQTT for lightweight messaging

---

Repository Structure

celine/
  admin/
  cli/
  common/
  datasets/
  pipelines/
schemas/
tests/

---

Configuration

Configuration is environment-driven using pydantic-settings:

• Environment variables first
• Optional .env files
• Typed validation
• Container-friendly defaults

---

Installation

pip install celine-utils

---

Intended Audience

CELINE Utils is intended for:

• Data engineers
• Platform engineers
• CELINE application developers

It is not a general-purpose data tooling library.

---

License

Copyright © 2025
Spindox Labs

Licensed under the Apache License, Version 2.0.