Fast & Fake Backfill Airflow DAGs Status
Project description
Airflow Fakefill Marker
Due to migrating to Kubernetes-host Airflow and using different backend, we need to find out a way to fill out all the history since its starting date for thousands of dags. To make this process going faster and easier, in the meantime, I didn't find this kind of tool on Github, so I implement this simple tool to help with marking dags as success.
Hope it can also help others.
Installations
Method 1
$ pip install fakefill
Method 2
$ pip install git+https://git@github.com/benbenbang/airflow_fastfill.git
Method 3
$ git clone git@github.com:benbenbang/airflow_fastfill.git
$ cd airflow_fastfill
$ pip install .
Usages
$ fakefill
It takes 1 of 2 required argument, and 6 optional arguments. You can also define them in a yaml file and pass to the cli.
-
Options
-
Required [1 / 2]:
- dag_id [-d][reqired]: can be a real dag id or "all" to fill all the dags
- config_path [-cp][choose one]: path to the config yaml
-
Optional:
- start_date [-sd]: starting date, default will be counted from 365 days ago
- maximum_day [-md]: maximum fill date per dag, rangint: [1, 180]
- maximum_unit [-mu]: maxium fill unit per dag, rangint: [1, 43200]
- ignore [-i]: still procceed auto fill even the dag ran recently
- pause_only [-p]: pass true to fill dags which are pause
- confirm [-y]: pass true to bypass the prompt if dag_id is all
- traceback [-v]: pass print our Airflow Database error
-
Examples
Fill all the dags for the past 30 days without prompt, and only fill if all the dags which have status == pause
$ fakefill -d all -p -md 30 -y
Run fastfill for dag id == dag_a
by counting default fakefill days == 365
$ fakefill -d dag_a
Run fastfill with config yaml
$ fakefill -cp config.yml
The yaml file needs to be defined with two dictonary types: dags
and settings
. For dags
section, it needs to be a list
, while the settings
section is dict
Sample:
dags:
- dag_a
- dag_b
- dag_c
settings:
start_date: 2019-01-01
maximum: "365"
traceback: false
confirm: true
pause_only: true
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file fakefill-1.0.1.tar.gz
.
File metadata
- Download URL: fakefill-1.0.1.tar.gz
- Upload date:
- Size: 13.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/4.3.1 pkginfo/1.8.2 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf3d1f984d8d253df19d72000a4a254726b36d4fe74a7f33a635dbead014e930 |
|
MD5 | 346d170c5dbf32bc4fa6b8545d72b3ff |
|
BLAKE2b-256 | 1a61365de288c8cd0b0067aca88b7374fdbaa9f74ab1fba1f6cd3c8fa79fb50a |
File details
Details for the file fakefill-1.0.1-py3-none-any.whl
.
File metadata
- Download URL: fakefill-1.0.1-py3-none-any.whl
- Upload date:
- Size: 15.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/4.3.1 pkginfo/1.8.2 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | aea106e4c6d3a4b4394607d41e61ec32e6fbf67a396d9b3dd396d667b65c0647 |
|
MD5 | 7baa2f5b022135bf261fbe71e2255147 |
|
BLAKE2b-256 | 69a7852c3027463b6bf6f98b4cd4a7a224e1d67011e827ddfd3294ef50b8eae5 |