dpone-airflow-pack 0.68.16

Lightweight Airflow scheduler-side pack provider for dpone GitOps workloads

dpone-airflow-pack

dpone-airflow-pack is the lightweight Airflow scheduler/webserver provider for dpone GitOps packs.

It only reads a static airflow-pack.json and builds visible Airflow/Kubernetes tasks. It does not import the full dpone runtime and intentionally contains no source/sink/native transfer dependencies such as ClickHouse, MSSQL, pyodbc, pandas, polars, or ConnectorX.

Recommended Airflow DAG import:

from dpone_airflow_pack import build_dpone_gitops_task_group_from_pack

For remote immutable packs, keep DAG parsing cache-only:

dpone-airflow-pack-sync \
  --once \
  --index-uri s3://bucket/dpone-artifacts/prod/repo/latest/pack-index.json \
  --reader-connection-id s3_dpone_artifacts_reader \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow

dpone-airflow-pack-cache-status \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow \
  --json

Use load_dpone_airflow_pack_with_provenance("cached://<workload_id>") when a DAG wrapper needs to expose which cache generation and pack checksum were used during parse. A KubernetesExecutor runtime worker is not required to mount the scheduler or DAG processor cache.

For DAGs that should run a whole GitOps domain or a named workflow group, keep the grouping in the domain catalog and resolve it through the lightweight provider:

workflow_groups:
  daily:
    workload_ids:
      - dim_customer
      - fact_order

from dpone_airflow_pack import workload_ids_from_gitops_domain

workload_ids = workload_ids_from_gitops_domain(
    repo_root,
    "sales",
    group="daily",
)

The resolver only reads YAML files and validates that every group member exists in the same domain catalog.

The full dpone[full,accel] package belongs in the KPO runtime image, not in the scheduler image.