Skip to main content

Fast & Fake Backfill Airflow DAGs Status

Project description

Airflow Fakefill Marker

Due to migrating to Kubernetes-host Airflow and using different backend, we need to find out a way to fill out all the history since its starting date for thousands of dags. To make this process going faster and easier, in the meantime, I didn't find this kind of tool on Github, so I implement this simple tool to help with marking dags as success. Hope it can also help others.

Installations

Method 1

$ pip install fakefill

Method 2

$ pip install git+https://git@github.com/benbenbang/airflow_fastfill.git

Method 3

$ git clone git@github.com:benbenbang/airflow_fastfill.git
$ cd airflow_fastfill
$ pip install .

Usages

$ fakefill

It takes 1 of 2 required argument, and 6 optional arguments. You can also define them in a yaml file and pass to the cli.

  • Options

    • Required [1 / 2]:

      • dag_id [-d][reqired]: can be a real dag id or "all" to fill all the dags
      • config_path [-cp][choose one]: path to the config yaml
    • Optional:

      • start_date [-sd]: starting date, default will be counted from 365 days ago
      • maximum_day [-md]: maximum fill date per dag, rangint: [1, 180]
      • maximum_unit [-mu]: maxium fill unit per dag, rangint: [1, 43200]
      • ignore [-i]: still procceed auto fill even the dag ran recently
      • pause_only [-p]: pass true to fill dags which are pause
      • confirm [-y]: pass true to bypass the prompt if dag_id is all
      • traceback [-v]: pass print our Airflow Database error

Examples

Fill all the dags for the past 30 days without prompt, and only fill if all the dags which have status == pause

$ fakefill -d all -p -md 30 -y

Run fastfill for dag id == dag_a by counting default fakefill days == 365

$ fakefill -d dag_a

Run fastfill with config yaml

$ fakefill -cp config.yml

The yaml file needs to be defined with two dictonary types: dags and settings. For dags section, it needs to be a list, while the settingssection is dict

Sample:

dags:
  - dag_a
  - dag_b
  - dag_c

settings:
  start_date: 2019-01-01
  maximum: "365"
  traceback: false
  confirm: true
  pause_only: true

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fakefill-1.0.1.tar.gz (13.3 kB view details)

Uploaded Source

Built Distribution

fakefill-1.0.1-py3-none-any.whl (15.1 kB view details)

Uploaded Python 3

File details

Details for the file fakefill-1.0.1.tar.gz.

File metadata

  • Download URL: fakefill-1.0.1.tar.gz
  • Upload date:
  • Size: 13.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.3.1 pkginfo/1.8.2 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12

File hashes

Hashes for fakefill-1.0.1.tar.gz
Algorithm Hash digest
SHA256 cf3d1f984d8d253df19d72000a4a254726b36d4fe74a7f33a635dbead014e930
MD5 346d170c5dbf32bc4fa6b8545d72b3ff
BLAKE2b-256 1a61365de288c8cd0b0067aca88b7374fdbaa9f74ab1fba1f6cd3c8fa79fb50a

See more details on using hashes here.

File details

Details for the file fakefill-1.0.1-py3-none-any.whl.

File metadata

  • Download URL: fakefill-1.0.1-py3-none-any.whl
  • Upload date:
  • Size: 15.1 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.3.1 pkginfo/1.8.2 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12

File hashes

Hashes for fakefill-1.0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 aea106e4c6d3a4b4394607d41e61ec32e6fbf67a396d9b3dd396d667b65c0647
MD5 7baa2f5b022135bf261fbe71e2255147
BLAKE2b-256 69a7852c3027463b6bf6f98b4cd4a7a224e1d67011e827ddfd3294ef50b8eae5

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page