dpone-airflow-pack 0.66.5

Lightweight Airflow scheduler-side pack provider for dpone GitOps workloads

dpone-airflow-pack

dpone-airflow-pack is the lightweight Airflow scheduler/webserver provider for dpone GitOps packs.

It only reads a static airflow-pack.json and builds visible Airflow/Kubernetes tasks. It does not import the full dpone runtime and intentionally contains no source/sink/native transfer dependencies such as ClickHouse, MSSQL, pyodbc, pandas, polars, or ConnectorX.

Recommended Airflow DAG import:

from dpone_airflow_pack import build_dpone_gitops_task_group_from_pack

The full dpone[full,accel] package belongs in the KPO runtime image, not in the scheduler image.

Remote pack cache

Remote airflow-pack.json artifacts can be published to object storage and mirrored into a bounded local scheduler cache.

Bootstrap one generation:

dpone-airflow-pack-sync \
  --once \
  --index-uri s3://do-dwh/dpone-artifacts/prod/airflow-dags-dev/latest/pack-index.json \
  --reader-connection-id s3_dpone_artifacts_reader \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow \
  --status-path /opt/airflow/dags/.dpone-cache/airflow/status/last-sync-status.json \
  --airflow-variable-key dpone_airflow_pack_cache_status

Refresh without restarting Airflow pods:

dpone-airflow-pack-sync \
  --watch \
  --interval-seconds 60 \
  --jitter-seconds 15 \
  --index-uri s3://do-dwh/dpone-artifacts/prod/airflow-dags-dev/latest/pack-index.json \
  --reader-connection-id s3_dpone_artifacts_reader \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow \
  --status-path /opt/airflow/dags/.dpone-cache/airflow/status/last-sync-status.json \
  --airflow-variable-key dpone_airflow_pack_cache_status

The sync loop is fail-open. On S3, connection, index, or hash errors it writes a warning status and preserves the previous current generation. DAG parse stays local and cache-only.

When --airflow-variable-key is set, the sync process also publishes the same redacted status payload into Airflow metadata. This is the recommended diagnostic channel for KubernetesExecutor deployments because runtime worker pods do not need to mount the scheduler or dagProcessor cache volume. Variable publish failures are warnings only and never fail the Airflow pod.

Connection resolution is scheduler-side safe for Airflow 2.x and 3.x: environment connection URI first, Airflow hook second, and metadata database fallback third. The metadata fallback matters for Airflow 3 sidecar/init containers, where provider hooks may not resolve DB-backed connections outside task execution context even though the metadata row exists.

Inspect the current cache status:

dpone-airflow-pack-cache-status \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow

Airflow execution policy

airflow-pack.json can carry a scheduler-side execution policy:

{
  "airflow": {
    "execution": {
      "task_executor": "KubernetesExecutor",
      "deferrable": true,
      "on_finish_action": "delete_succeeded_pod"
    }
  }
}

The lightweight provider maps it to KubernetesPodOperator kwargs (executor, deferrable, on_finish_action) and treats those fields as pack-owned. DAG-local overrides may still set DAG-owned values such as retries, pool, get_logs, or in_cluster.