dpone-airflow-pack 0.66.4

Lightweight Airflow scheduler-side pack provider for dpone GitOps workloads

dpone-airflow-pack

dpone-airflow-pack is the lightweight Airflow scheduler/webserver provider for dpone GitOps packs.

It only reads a static airflow-pack.json and builds visible Airflow/Kubernetes tasks. It does not import the full dpone runtime and intentionally contains no source/sink/native transfer dependencies such as ClickHouse, MSSQL, pyodbc, pandas, polars, or ConnectorX.

Recommended Airflow DAG import:

from dpone_airflow_pack import build_dpone_gitops_task_group_from_pack

The full dpone[full,accel] package belongs in the KPO runtime image, not in the scheduler image.

Remote pack cache

Remote airflow-pack.json artifacts can be published to object storage and mirrored into a bounded local scheduler cache.

Bootstrap one generation:

dpone-airflow-pack-sync \
  --once \
  --index-uri s3://do-dwh/dpone-artifacts/prod/airflow-dags-dev/latest/pack-index.json \
  --reader-connection-id s3_dpone_artifacts_reader \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow

Refresh without restarting Airflow pods:

dpone-airflow-pack-sync \
  --watch \
  --interval-seconds 60 \
  --jitter-seconds 15 \
  --index-uri s3://do-dwh/dpone-artifacts/prod/airflow-dags-dev/latest/pack-index.json \
  --reader-connection-id s3_dpone_artifacts_reader \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow

The sync loop is fail-open. On S3, connection, index, or hash errors it writes a warning status and preserves the previous current generation. DAG parse stays local and cache-only.

Connection resolution is scheduler-side safe for Airflow 2.x and 3.x: environment connection URI first, Airflow hook second, and metadata database fallback third. The metadata fallback matters for Airflow 3 sidecar/init containers, where provider hooks may not resolve DB-backed connections outside task execution context even though the metadata row exists.

Inspect the current cache status:

dpone-airflow-pack-cache-status \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow

Airflow execution policy

airflow-pack.json can carry a scheduler-side execution policy:

{
  "airflow": {
    "execution": {
      "task_executor": "KubernetesExecutor",
      "deferrable": true,
      "on_finish_action": "delete_succeeded_pod"
    }
  }
}

The lightweight provider maps it to KubernetesPodOperator kwargs (executor, deferrable, on_finish_action) and treats those fields as pack-owned. DAG-local overrides may still set DAG-owned values such as retries, pool, get_logs, or in_cluster.