dpone-airflow-pack

dpone-airflow-pack is the lightweight, Airflow-independent reader and DAG construction library for dpone GitOps packs. The formal apache-airflow-providers-dpone distribution owns provider discovery and the canonical airflow.providers.dpone namespace.

It only reads a static airflow-pack.json and builds visible Airflow/Kubernetes tasks. It does not import the full dpone runtime and intentionally contains no source/sink/native transfer dependencies such as ClickHouse, MSSQL, pyodbc, pandas, polars, or ConnectorX.

Recommended zero-boilerplate Airflow DAG import:

from airflow.providers.dpone import load_dpone_dags

load_report = load_dpone_dags(
    globals(),
    index_path="/opt/airflow/dags/.dpone-cache/current/airflow-index.json",
)
if load_report.fatal:
    error_code = (
        load_report.errors[0].get("code", "DPONE_AIRFLOW_INDEX_INVALID")
        if load_report.errors
        else "DPONE_AIRFLOW_INDEX_INVALID"
    )
    raise RuntimeError(
        f"{error_code}: dpone Airflow deployment index could not be loaded"
    )

The formal provider's canonical namespace is PEP 561 typed. DponeDag.from_spec(...) and DponeTaskGroup.from_pack(...) remain typed escape hatches for hybrid DAGs. The old dpone_airflow_pack imports remain compatibility shims and emit one deprecation warning per process for provider-facade objects.

For the complete import, generated-loader, and cache-layout cutover, follow the provider and cache migration guide.

Legacy installations that still consume a mutable remote latest/pack-index.json may keep the compatibility sync temporarily:

dpone-airflow-pack-sync \
  --once \
  --index-uri s3://bucket/dpone-artifacts/prod/repo/latest/pack-index.json \
  --reader-connection-id s3_dpone_artifacts_reader \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow

dpone-airflow-pack-cache-status \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow \
  --json

The loader reads only the local composite deployment index and its listed checksum-verified DAG specs/packs. It does not scan authoring sources, refresh a remote cache, or read Airflow Variables, Connections, Vault, Kubernetes, or a metadata database during parse.

For DAGs that should run a whole GitOps domain or a named workflow group, keep the grouping in the domain catalog and resolve it through the lightweight provider:

workflow_groups:
  daily:
    workload_ids:
      - dim_customer
      - fact_order

from dpone_airflow_pack import workload_ids_from_gitops_domain

workload_ids = workload_ids_from_gitops_domain(
    repo_root,
    "sales",
    group="daily",
)

The resolver only reads YAML files and validates that every group member exists in the same domain catalog.

The full dpone[full,accel] package belongs in the KPO runtime image, not in the scheduler image.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.74.33)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for paulkov007 from gravatar.com paulkov007

Credits

Author/Maintainer: PaulKov

License expression

Apache-2.0
View SPDX License List

Requires

Python <3.13, >=3.11

Tags

airflow gitops data-engineering scheduler kubernetes

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dpone_airflow_pack-0.73.30.tar.gz (116.8 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

dpone_airflow_pack-0.73.30-py3-none-any.whl (162.6 kB view details)

Uploaded Python 3

File details

Details for the file dpone_airflow_pack-0.73.30.tar.gz.

File metadata

  • Download URL: dpone_airflow_pack-0.73.30.tar.gz
  • Upload date:
  • Size: 116.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for dpone_airflow_pack-0.73.30.tar.gz
Algorithm Hash digest
SHA256 cf080cec85580dacfd768981400981517ed5dc1a05c0001899a806c3347415de
MD5 2f8c09184fc2cf32d127cc013bacfbff
BLAKE2b-256 12b2cc746acfe86ba910524261ca32fbaaef2dccc0c4fd2715cad0c4fd4fa3a3

See more details on using hashes here.

File details

Details for the file dpone_airflow_pack-0.73.30-py3-none-any.whl.

File metadata

File hashes

Hashes for dpone_airflow_pack-0.73.30-py3-none-any.whl
Algorithm Hash digest
SHA256 1565e29dfd382a4de4d2500baa71a80365cabbf99078a5611863876268de4a70
MD5 b9120638f4fe12f1496e1efde656ebb9
BLAKE2b-256 cce2c01a1fcb96e0e46543e24a026c9e3d985409f3e57e04dc6acab153c41836

See more details on using hashes here.

Release history Release notifications | RSS feed

0.74.33

2 files

0.74.32

2 files

0.74.31

2 files

0.74.30

2 files

0.74.29

2 files

0.74.28

2 files

0.74.27

2 files

0.74.26

2 files

0.74.25

2 files

0.74.24

2 files

0.74.23

2 files

0.74.22

2 files

0.74.20

2 files

0.74.19

2 files

0.74.17

2 files

0.74.16

2 files

0.74.15

2 files

0.74.14

2 files

0.74.13

2 files

0.74.12

2 files

0.74.11

2 files

0.74.10

2 files

0.74.9

2 files

0.74.8

2 files

0.74.7

2 files

0.74.6

2 files

0.74.5

2 files

0.74.4

2 files

0.74.3

2 files

0.74.2

2 files

0.74.0

2 files

0.73.42

2 files

0.73.38

2 files

0.73.36

2 files

0.73.34

2 files

0.73.31

2 files

This release

0.73.30 This release

2 files

0.73.29

2 files

0.73.28

2 files

0.73.25

2 files

0.73.24

2 files

0.73.23

2 files

0.73.22

2 files

0.73.21

2 files

0.73.20

2 files

0.73.19

2 files

0.73.17

2 files

0.73.16

2 files

0.73.15

2 files

0.73.14

2 files

0.73.13

2 files

0.73.10

2 files

0.73.9

2 files

0.73.6

2 files

0.73.2

2 files

0.73.1

2 files

0.73.0

2 files

0.72.5

2 files

0.72.4

2 files

0.72.3

2 files

0.72.2

2 files

0.72.1

2 files

0.72.0

2 files

0.70.0

2 files

0.69.0

2 files

0.68.22

2 files

0.68.21

2 files

0.68.20

2 files

0.68.19

2 files

0.68.18

2 files

0.68.17

2 files

0.68.16

2 files

0.68.15

2 files

0.68.14

2 files

0.68.12

2 files

0.68.11

2 files

0.68.10

2 files

0.68.8

2 files

0.68.7

2 files

0.68.6

2 files

0.68.5

2 files

0.68.4

2 files

0.68.2

2 files

0.68.1

2 files

0.68.0

2 files

0.67.0

2 files

0.66.17

2 files

0.66.16

2 files

0.66.15

2 files

0.66.14

2 files

0.66.13

2 files

0.66.12

2 files

0.66.11

2 files

0.66.10

2 files

0.66.9

2 files

0.66.8

2 files

0.66.7

2 files

0.66.6

2 files

0.66.5

2 files

0.66.4

2 files

0.66.3

2 files

0.66.2

2 files

0.66.1

2 files

0.66.0

2 files

0.65.2

2 files

0.65.1

2 files

0.65.0

2 files

0.64.1

2 files

0.64.0

2 files

0.63.0

2 files

0.62.1

2 files

0.62.0

2 files

0.61.3

2 files

0.61.2

2 files

0.61.1

2 files

0.61.0

2 files

0.60.2

2 files

0.60.1

2 files

0.60.0

2 files

0.59.0

2 files

0.58.0

2 files

0.57.0

2 files

0.56.0

2 files

0.55.0

2 files

0.54.0

2 files

0.53.0

2 files

0.52.0

2 files

0.51.0

2 files

0.50.0

2 files

0.49.0

2 files

0.48.1

2 files

0.48.0

2 files

0.47.0

2 files

0.46.0

2 files

0.45.0

2 files

0.44.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page