apache-airflow-providers-dpone 0.73.21

Formal Apache Airflow provider facade for dpone self-service DAGs

apache-airflow-providers-dpone

The formal Apache Airflow provider facade for dpone self-service DAGs.

Install

Use a supported Airflow matrix cell. This example uses Python 3.12, Airflow 3.2.0, and the CNCF Kubernetes provider 10.14.0. Use Apache Airflow's official constraints for the Airflow install only, then pin Airflow again while adding this provider:

AIRFLOW_VERSION=3.2.0
PYTHON_VERSION=3.12
AIRFLOW_CONSTRAINTS_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

python -m pip install \
  "apache-airflow[cncf.kubernetes]==${AIRFLOW_VERSION}" \
  --constraint "${AIRFLOW_CONSTRAINTS_URL}"
python -m pip install \
  "apache-airflow==${AIRFLOW_VERSION}" \
  "apache-airflow-providers-cncf-kubernetes==10.14.0" \
  "apache-airflow-providers-dpone==0.73.21"
python -m pip check
python -c "from airflow.providers.dpone import load_dpone_dags; print(load_dpone_dags.__module__)"

The provider installs dpone-airflow-pack==0.73.21; it does not require the full dpone runtime or connector drivers in the scheduler image.

Load DAGs

from airflow.providers.dpone import load_dpone_dags, write_dpone_loader_ack

index_path = "/opt/airflow/dags/.dpone-cache/current/airflow-index.json"
load_report = load_dpone_dags(
    globals(),
    index_path=index_path,
)
write_dpone_loader_ack(
    load_report,
    index_path=index_path,
    ack_path="/opt/airflow/dags/.dpone-cache/status/loader-ack.json",
)
if load_report.fatal:
    error_code = (
        load_report.errors[0].get("code", "DPONE_AIRFLOW_INDEX_INVALID")
        if load_report.errors
        else "DPONE_AIRFLOW_INDEX_INVALID"
    )
    raise RuntimeError(
        f"{error_code}: dpone Airflow deployment index could not be loaded"
    )

write_dpone_loader_ack(...) performs bounded local filesystem I/O only. It atomically binds the completed parse report to the exact deployment and index digest, allowing an external controller to prove that visible DAG metadata is fresh. Keep the acknowledgement under the cache status/ directory.

The distribution owns Airflow provider discovery and the typed canonical namespace. The dependency-light dpone-airflow-pack package owns static pack reading and DAG construction. Importing the provider performs no remote cache, Airflow metadata, Connection, Variable, Vault, Kubernetes, or database I/O.

The deployment index read is bounded to 8 MiB and each listed artifact to 64 MiB by default. An index-level error returns fatal=True; do not represent that as a successful empty parse. After a trusted index is loaded, DAG spec and workload-pack size/SHA-256 failures are isolated to the affected DAG under the default skip_and_report policy. Recover by rematerializing the immutable release/deployment from trusted storage, never by editing cache artifacts in place.

DponeDag.from_spec(...) and DponeTaskGroup.from_pack(...) are typed escape hatches for hybrid DAGs. The recommended path is load_dpone_dags(...).