apache-airflow-providers-dpone

The formal Apache Airflow provider facade for dpone self-service DAGs.

Install

Use a supported Airflow matrix cell. This example uses Python 3.12, Airflow 3.2.0, and the CNCF Kubernetes provider 10.14.0. Use Apache Airflow's official constraints for the Airflow install only, then pin Airflow again while adding this provider:

AIRFLOW_VERSION=3.2.0
PYTHON_VERSION=3.12
DPONE_VERSION=X.Y.Z
AIRFLOW_CONSTRAINTS_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

python3 -m pip install \
  "apache-airflow[cncf.kubernetes]==${AIRFLOW_VERSION}" \
  --constraint "${AIRFLOW_CONSTRAINTS_URL}"
python3 -m pip install \
  "apache-airflow==${AIRFLOW_VERSION}" \
  "apache-airflow-providers-cncf-kubernetes==10.14.0" \
  "apache-airflow-providers-dpone==${DPONE_VERSION}"
python3 -m pip check
python3 -c "from airflow.providers.dpone import load_dpone_dags; print(load_dpone_dags.__module__)"

The provider installs the same-version dpone-airflow-pack; it does not require the full dpone runtime or connector drivers in the scheduler image.

Use only a version already published on PyPI. For a release candidate, install the exact provider and pack wheels built from the same reviewed commit instead of using a not-yet-existing tag or floating branch.

Load DAGs

from airflow.providers.dpone import load_and_acknowledge_dpone_dags

index_path = "/opt/airflow/.dpone-cache/current/airflow-index.json"
loaded = load_and_acknowledge_dpone_dags(
    globals(),
    index_path=index_path,
    ack_path="/opt/airflow/.dpone-ack/loader-ack.json",
    ack_root="/opt/airflow/.dpone-ack",
)
if loaded.report.fatal:
    error_code = (
        loaded.report.errors[0].get("code", "DPONE_AIRFLOW_INDEX_INVALID")
        if loaded.report.errors
        else "DPONE_AIRFLOW_INDEX_INVALID"
    )
    raise RuntimeError(
        f"{error_code}: dpone Airflow deployment index could not be loaded"
    )

load_and_acknowledge_dpone_dags(...) holds one shared cache lease across the parse and acknowledgement. It performs bounded local filesystem I/O only and atomically binds the completed report to the exact deployment and index digest. Mount /opt/airflow/.dpone-cache read-only in the parser container and provide the separate /opt/airflow/.dpone-ack volume as its only writable evidence channel. The older cache-local status/ destination remains supported when ack_root is omitted, but it requires a writable cache mount and is not the recommended Kubernetes layout.

The distribution owns Airflow provider discovery and the typed canonical namespace. The dependency-light dpone-airflow-pack package owns static pack reading and DAG construction. Importing the provider performs no remote cache, Airflow metadata, Connection, Variable, Vault, Kubernetes, or database I/O.

The deployment index read is bounded to 8 MiB and each listed artifact to 64 MiB by default. An index-level error returns fatal=True; do not represent that as a successful empty parse. After a trusted index is loaded, DAG spec and workload-pack size/SHA-256 failures are isolated to the affected DAG under the default skip_and_report policy. Recover by rematerializing the immutable release/deployment from trusted storage, never by editing cache artifacts in place.

DponeDag.from_spec(...) and DponeTaskGroup.from_pack(...) are typed escape hatches for hybrid DAGs. The recommended production path is load_and_acknowledge_dpone_dags(...); load_dpone_dags(...) remains the parse-only lower-level API.

Operations

Strict runtime tasks set get_logs=true and on_finish_action=delete_succeeded_pod so Airflow streams base container stdout after pod start. Structured XCom/runtime evidence remains the task outcome authority; a cluster log collector is still required for post-delete forensics. The full dpone distribution ships the runtime Pod retention control; the parse-safe provider facade does not execute it during DAG import. Before production, deploy and certify its label-scoped plan/apply CronJob through the runtime pod retention runbook, and deploy the exact cache through the Kubernetes cache runbook. The stock JSONL publisher reports process_ordered, not crash durability; production apply requires a certified publisher/sink that reports durable_acknowledged and retains the operation, intent, outcome and completion chain.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.80.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for paulkov007 from gravatar.com paulkov007

Credits

Author/Maintainer: PaulKov

License expression

Apache-2.0
View SPDX License List

Requires

Python <3.13, >=3.11

Tags

airflow provider dpone gitops data-engineering

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

apache_airflow_providers_dpone-0.74.33.tar.gz (4.0 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

apache_airflow_providers_dpone-0.74.33-py3-none-any.whl (5.8 kB view details)

Uploaded Python 3

File details

Details for the file apache_airflow_providers_dpone-0.74.33.tar.gz.

File metadata

File hashes

Hashes for apache_airflow_providers_dpone-0.74.33.tar.gz
Algorithm Hash digest
SHA256 9b2107522ea9a65b4a700d754b87d8e4a3d4bcdedd42b2b82b4ef1b6bda89455
MD5 17b1c12b32128d1ab498ddb2ac362126
BLAKE2b-256 f7a99cfeca37e05268300940e3c4581064096ea93659a910225880fcc986be87

See more details on using hashes here.

Provenance

The following attestation bundles were made for apache_airflow_providers_dpone-0.74.33.tar.gz:

Publisher: pypi-release.yml on PaulKov/dpone-release-controller

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file apache_airflow_providers_dpone-0.74.33-py3-none-any.whl.

File metadata

File hashes

Hashes for apache_airflow_providers_dpone-0.74.33-py3-none-any.whl
Algorithm Hash digest
SHA256 18e1880ae3bd0d943d1a9d86f66f97fe246c012b8fa8b4280216c8a7a4796a45
MD5 2b02c6665b64d67ade4b10376ffe5022
BLAKE2b-256 633c17de07727d712d6f96c73210589c2f5f599603e3e22e2d427eeaf46874b1

See more details on using hashes here.

Provenance

The following attestation bundles were made for apache_airflow_providers_dpone-0.74.33-py3-none-any.whl:

Publisher: pypi-release.yml on PaulKov/dpone-release-controller

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.80.0

2 files

0.79.2

2 files

0.79.1

2 files

0.79.0

2 files

0.78.0

2 files

0.77.1

2 files

0.77.0

2 files

0.76.0

2 files

0.75.0

2 files

0.74.36

2 files

0.74.35

2 files

0.74.34

2 files

This release

0.74.33 This release

2 files

0.74.32

2 files

0.74.31

2 files

0.74.30

2 files

0.74.29

2 files

0.74.28

2 files

0.74.27

2 files

0.74.26

2 files

0.74.25

2 files

0.74.24

2 files

0.74.23

2 files

0.74.22

2 files

0.74.20

2 files

0.74.19

2 files

0.74.17

2 files

0.74.16

2 files

0.74.15

2 files

0.74.14

2 files

0.74.13

2 files

0.74.12

2 files

0.74.11

2 files

0.74.10

2 files

0.74.9

2 files

0.74.8

2 files

0.74.7

2 files

0.74.6

2 files

0.74.5

2 files

0.74.4

2 files

0.74.3

2 files

0.74.2

2 files

0.74.0

2 files

0.73.42

2 files

0.73.38

2 files

0.73.36

2 files

0.73.34

2 files

0.73.31

2 files

0.73.30

2 files

0.73.29

2 files

0.73.28

2 files

0.73.25

2 files

0.73.24

2 files

0.73.23

2 files

0.73.22

2 files

0.73.21

2 files

0.73.20

2 files

0.73.19

2 files

0.73.17

2 files

0.73.16

2 files

0.73.15

2 files

0.73.14

2 files

0.73.13

2 files

0.73.10

2 files

0.73.9

2 files

0.73.6

2 files

0.73.2

2 files

0.73.1

2 files

0.73.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page