dpone-airflow-pack

dpone-airflow-pack is the lightweight Airflow scheduler/webserver provider for dpone GitOps packs.

It only reads a static airflow-pack.json and builds visible Airflow/Kubernetes tasks. It does not import the full dpone runtime and intentionally contains no source/sink/native transfer dependencies such as ClickHouse, MSSQL, pyodbc, pandas, polars, or ConnectorX.

Recommended Airflow DAG import:

from dpone_airflow_pack import build_dpone_gitops_task_group_from_pack

For remote immutable packs, keep DAG parsing cache-only:

dpone-airflow-pack-sync \
  --once \
  --index-uri s3://bucket/dpone-artifacts/prod/repo/latest/pack-index.json \
  --reader-connection-id s3_dpone_artifacts_reader \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow

dpone-airflow-pack-cache-status \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow \
  --json

Use load_dpone_airflow_pack_with_provenance("cached://<workload_id>") when a DAG wrapper needs to expose which cache generation and pack checksum were used during parse. A KubernetesExecutor runtime worker is not required to mount the scheduler or DAG processor cache.

For DAGs that should run a whole GitOps domain or a named workflow group, keep the grouping in the domain catalog and resolve it through the lightweight provider:

workflow_groups:
  daily:
    workload_ids:
      - dim_customer
      - fact_order

from dpone_airflow_pack import workload_ids_from_gitops_domain

workload_ids = workload_ids_from_gitops_domain(
    repo_root,
    "sales",
    group="daily",
)

The resolver only reads YAML files and validates that every group member exists in the same domain catalog.

The full dpone[full,accel] package belongs in the KPO runtime image, not in the scheduler image.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.74.33)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for paulkov007 from gravatar.com paulkov007

Credits

Author/Maintainer: PaulKov

License expression

Apache-2.0
View SPDX License List

Requires

Python <3.13, >=3.11

Tags

airflow gitops data-engineering scheduler kubernetes

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dpone_airflow_pack-0.68.17.tar.gz (26.2 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

dpone_airflow_pack-0.68.17-py3-none-any.whl (38.9 kB view details)

Uploaded Python 3

File details

Details for the file dpone_airflow_pack-0.68.17.tar.gz.

File metadata

  • Download URL: dpone_airflow_pack-0.68.17.tar.gz
  • Upload date:
  • Size: 26.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dpone_airflow_pack-0.68.17.tar.gz
Algorithm Hash digest
SHA256 f34f80855c35deb94af6bdf473ece6209848f43a70a603a891ae23e8bdf4b05c
MD5 2680ad606d18c6341872b24644c37f9c
BLAKE2b-256 80ca72bf14e1dce79846f9d0444a6204fb7f034a52b61c75273ef1dd54df4c0c

See more details on using hashes here.

File details

Details for the file dpone_airflow_pack-0.68.17-py3-none-any.whl.

File metadata

File hashes

Hashes for dpone_airflow_pack-0.68.17-py3-none-any.whl
Algorithm Hash digest
SHA256 3c88cef7f5ed3362249116ef9417b4000ee10c19ac973ea27d8a321b639eba91
MD5 12807c48cff25d2c295b63249b8778a4
BLAKE2b-256 4a5248ed5c9119038b8c20fd4eeb53876387f1f7a03f42a3f4eb87666651d313

See more details on using hashes here.

Release history Release notifications | RSS feed

0.74.33

2 files

0.74.32

2 files

0.74.31

2 files

0.74.30

2 files

0.74.29

2 files

0.74.28

2 files

0.74.27

2 files

0.74.26

2 files

0.74.25

2 files

0.74.24

2 files

0.74.23

2 files

0.74.22

2 files

0.74.20

2 files

0.74.19

2 files

0.74.17

2 files

0.74.16

2 files

0.74.15

2 files

0.74.14

2 files

0.74.13

2 files

0.74.12

2 files

0.74.11

2 files

0.74.10

2 files

0.74.9

2 files

0.74.8

2 files

0.74.7

2 files

0.74.6

2 files

0.74.5

2 files

0.74.4

2 files

0.74.3

2 files

0.74.2

2 files

0.74.0

2 files

0.73.42

2 files

0.73.38

2 files

0.73.36

2 files

0.73.34

2 files

0.73.31

2 files

0.73.30

2 files

0.73.29

2 files

0.73.28

2 files

0.73.25

2 files

0.73.24

2 files

0.73.23

2 files

0.73.22

2 files

0.73.21

2 files

0.73.20

2 files

0.73.19

2 files

0.73.17

2 files

0.73.16

2 files

0.73.15

2 files

0.73.14

2 files

0.73.13

2 files

0.73.10

2 files

0.73.9

2 files

0.73.6

2 files

0.73.2

2 files

0.73.1

2 files

0.73.0

2 files

0.72.5

2 files

0.72.4

2 files

0.72.3

2 files

0.72.2

2 files

0.72.1

2 files

0.72.0

2 files

0.70.0

2 files

0.69.0

2 files

0.68.22

2 files

0.68.21

2 files

0.68.20

2 files

0.68.19

2 files

0.68.18

2 files

This release

0.68.17 This release

2 files

0.68.16

2 files

0.68.15

2 files

0.68.14

2 files

0.68.12

2 files

0.68.11

2 files

0.68.10

2 files

0.68.8

2 files

0.68.7

2 files

0.68.6

2 files

0.68.5

2 files

0.68.4

2 files

0.68.2

2 files

0.68.1

2 files

0.68.0

2 files

0.67.0

2 files

0.66.17

2 files

0.66.16

2 files

0.66.15

2 files

0.66.14

2 files

0.66.13

2 files

0.66.12

2 files

0.66.11

2 files

0.66.10

2 files

0.66.9

2 files

0.66.8

2 files

0.66.7

2 files

0.66.6

2 files

0.66.5

2 files

0.66.4

2 files

0.66.3

2 files

0.66.2

2 files

0.66.1

2 files

0.66.0

2 files

0.65.2

2 files

0.65.1

2 files

0.65.0

2 files

0.64.1

2 files

0.64.0

2 files

0.63.0

2 files

0.62.1

2 files

0.62.0

2 files

0.61.3

2 files

0.61.2

2 files

0.61.1

2 files

0.61.0

2 files

0.60.2

2 files

0.60.1

2 files

0.60.0

2 files

0.59.0

2 files

0.58.0

2 files

0.57.0

2 files

0.56.0

2 files

0.55.0

2 files

0.54.0

2 files

0.53.0

2 files

0.52.0

2 files

0.51.0

2 files

0.50.0

2 files

0.49.0

2 files

0.48.1

2 files

0.48.0

2 files

0.47.0

2 files

0.46.0

2 files

0.45.0

2 files

0.44.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page