dpone-airflow-pack

dpone-airflow-pack is the lightweight, Airflow-independent reader and DAG construction library for dpone GitOps packs. The formal apache-airflow-providers-dpone distribution owns provider discovery and the canonical airflow.providers.dpone namespace.

It only reads a static airflow-pack.json and builds visible Airflow/Kubernetes tasks. It does not import the full dpone runtime and intentionally contains no source/sink/native transfer dependencies such as ClickHouse, MSSQL, pyodbc, pandas, polars, or ConnectorX.

Recommended zero-boilerplate Airflow DAG import:

from airflow.providers.dpone import load_dpone_dags

load_report = load_dpone_dags(
    globals(),
    index_path="/opt/airflow/dags/.dpone-cache/current/airflow-index.json",
)
if load_report.fatal:
    error_code = (
        load_report.errors[0].get("code", "DPONE_AIRFLOW_INDEX_INVALID")
        if load_report.errors
        else "DPONE_AIRFLOW_INDEX_INVALID"
    )
    raise RuntimeError(
        f"{error_code}: dpone Airflow deployment index could not be loaded"
    )

The formal provider's canonical namespace is PEP 561 typed. DponeDag.from_spec(...) and DponeTaskGroup.from_pack(...) remain typed escape hatches for hybrid DAGs. The old dpone_airflow_pack imports remain compatibility shims and emit one deprecation warning per process for provider-facade objects.

For the complete import, generated-loader, and cache-layout cutover, follow the provider and cache migration guide.

Legacy installations that still consume a mutable remote latest/pack-index.json may keep the compatibility sync temporarily:

dpone-airflow-pack-sync \
  --once \
  --index-uri s3://bucket/dpone-artifacts/prod/repo/latest/pack-index.json \
  --reader-connection-id s3_dpone_artifacts_reader \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow

dpone-airflow-pack-cache-status \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow \
  --json

The loader reads only the local composite deployment index and its listed checksum-verified DAG specs/packs. It does not scan authoring sources, refresh a remote cache, or read Airflow Variables, Connections, Vault, Kubernetes, or a metadata database during parse.

For DAGs that should run a whole GitOps domain or a named workflow group, keep the grouping in the domain catalog and resolve it through the lightweight provider:

workflow_groups:
  daily:
    workload_ids:
      - dim_customer
      - fact_order

from dpone_airflow_pack import workload_ids_from_gitops_domain

workload_ids = workload_ids_from_gitops_domain(
    repo_root,
    "sales",
    group="daily",
)

The resolver only reads YAML files and validates that every group member exists in the same domain catalog.

The full dpone[full,accel] package belongs in the KPO runtime image, not in the scheduler image.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.74.33)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for paulkov007 from gravatar.com paulkov007

Credits

Author/Maintainer: PaulKov

License expression

Apache-2.0
View SPDX License List

Requires

Python <3.13, >=3.11

Tags

airflow gitops data-engineering scheduler kubernetes

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dpone_airflow_pack-0.73.13.tar.gz (93.9 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

dpone_airflow_pack-0.73.13-py3-none-any.whl (129.1 kB view details)

Uploaded Python 3

File details

Details for the file dpone_airflow_pack-0.73.13.tar.gz.

File metadata

  • Download URL: dpone_airflow_pack-0.73.13.tar.gz
  • Upload date:
  • Size: 93.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for dpone_airflow_pack-0.73.13.tar.gz
Algorithm Hash digest
SHA256 ce6e4ea4c77a6310937f55b4799f272352cadfac1cd4e1b1137a7662b111de74
MD5 18e6beadea7b34da511aa2da84d62f58
BLAKE2b-256 c018c234f0e6731981a82e16aad031da395271b8d6287224b72d66d2162b9e70

See more details on using hashes here.

File details

Details for the file dpone_airflow_pack-0.73.13-py3-none-any.whl.

File metadata

File hashes

Hashes for dpone_airflow_pack-0.73.13-py3-none-any.whl
Algorithm Hash digest
SHA256 205550b6da52398053594d0b362ad77ffeb21c89f605cb9112603c6d42f04844
MD5 966883dc43229cb39a71bdced0212606
BLAKE2b-256 31f75ad68c66969f57f86069b487d56956ab997dca1e5caef075faa307df07fc

See more details on using hashes here.

Release history Release notifications | RSS feed

0.74.33

2 files

0.74.32

2 files

0.74.31

2 files

0.74.30

2 files

0.74.29

2 files

0.74.28

2 files

0.74.27

2 files

0.74.26

2 files

0.74.25

2 files

0.74.24

2 files

0.74.23

2 files

0.74.22

2 files

0.74.20

2 files

0.74.19

2 files

0.74.17

2 files

0.74.16

2 files

0.74.15

2 files

0.74.14

2 files

0.74.13

2 files

0.74.12

2 files

0.74.11

2 files

0.74.10

2 files

0.74.9

2 files

0.74.8

2 files

0.74.7

2 files

0.74.6

2 files

0.74.5

2 files

0.74.4

2 files

0.74.3

2 files

0.74.2

2 files

0.74.0

2 files

0.73.42

2 files

0.73.38

2 files

0.73.36

2 files

0.73.34

2 files

0.73.31

2 files

0.73.30

2 files

0.73.29

2 files

0.73.28

2 files

0.73.25

2 files

0.73.24

2 files

0.73.23

2 files

0.73.22

2 files

0.73.21

2 files

0.73.20

2 files

0.73.19

2 files

0.73.17

2 files

0.73.16

2 files

0.73.15

2 files

0.73.14

2 files

This release

0.73.13 This release

2 files

0.73.10

2 files

0.73.9

2 files

0.73.6

2 files

0.73.2

2 files

0.73.1

2 files

0.73.0

2 files

0.72.5

2 files

0.72.4

2 files

0.72.3

2 files

0.72.2

2 files

0.72.1

2 files

0.72.0

2 files

0.70.0

2 files

0.69.0

2 files

0.68.22

2 files

0.68.21

2 files

0.68.20

2 files

0.68.19

2 files

0.68.18

2 files

0.68.17

2 files

0.68.16

2 files

0.68.15

2 files

0.68.14

2 files

0.68.12

2 files

0.68.11

2 files

0.68.10

2 files

0.68.8

2 files

0.68.7

2 files

0.68.6

2 files

0.68.5

2 files

0.68.4

2 files

0.68.2

2 files

0.68.1

2 files

0.68.0

2 files

0.67.0

2 files

0.66.17

2 files

0.66.16

2 files

0.66.15

2 files

0.66.14

2 files

0.66.13

2 files

0.66.12

2 files

0.66.11

2 files

0.66.10

2 files

0.66.9

2 files

0.66.8

2 files

0.66.7

2 files

0.66.6

2 files

0.66.5

2 files

0.66.4

2 files

0.66.3

2 files

0.66.2

2 files

0.66.1

2 files

0.66.0

2 files

0.65.2

2 files

0.65.1

2 files

0.65.0

2 files

0.64.1

2 files

0.64.0

2 files

0.63.0

2 files

0.62.1

2 files

0.62.0

2 files

0.61.3

2 files

0.61.2

2 files

0.61.1

2 files

0.61.0

2 files

0.60.2

2 files

0.60.1

2 files

0.60.0

2 files

0.59.0

2 files

0.58.0

2 files

0.57.0

2 files

0.56.0

2 files

0.55.0

2 files

0.54.0

2 files

0.53.0

2 files

0.52.0

2 files

0.51.0

2 files

0.50.0

2 files

0.49.0

2 files

0.48.1

2 files

0.48.0

2 files

0.47.0

2 files

0.46.0

2 files

0.45.0

2 files

0.44.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page