MultiDagRunPlugin for airflow
Project description
Multi dag run
This plugin contains operators for triggering a DAG run multiple times and you can dynamically specify how many DAG run instances create.
It can be useful when you have to handle a big data and you want to split it into chunks and run multiple instances of the same task in parallel.
When you see a lot launched target DAGs you can set up more workers. So this makes it pretty easy to scale.
Install
pip install airflow_multi_dagrun
Example
Code for scheduling dags
import datetime as dt
from airflow import DAG
from airflow_multi_dagrun.operators import TriggerMultiDagRunOperator
def generate_dag_run():
for i in range(100):
yield {'index': i}
default_args = {
'owner': 'airflow',
'start_date': dt.datetime(2015, 6, 1),
}
dag = DAG('reindex_scheduler', schedule_interval=None, default_args=default_args)
ran_dags = TriggerMultiDagRunOperator(
task_id='gen_target_dag_run',
dag=dag,
trigger_dag_id='example_target_dag',
python_callable=generate_dag_run,
)
This code will schedule dag with id example_target_dag
100 times and pass payload to it.
Example of triggered dag:
dag = DAG(
dag_id='example_target_dag',
schedule_interval=None,
default_args={'start_date': datetime.utcnow(), 'owner': 'airflow'},
)
def run_this_func(dag_run, **kwargs):
print("Chunk received: {}".format(dag_run.conf['index']))
chunk_handler = PythonOperator(
task_id='chunk_handler',
provide_context=True,
python_callable=run_this_func,
dag=dag
)
Run example
There is docker-compose config, so it requires docker to be installed: docker
, docker-compose
make init
- create dbmake add-admin
- createadmin
user (is asks a password)make web
- start docker containers, run airflow webservermake scheduler
- start docker containers, run airflow scheduler
make down
will stop and remove docker containers
Contributions
If you have found a bug or have some idea for improvement feel free to create an issue or pull request.
License
Apache 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for airflow_multi_dagrun-2.2.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 15bd2f8f5d0050662b7b9a4b59535a19e722073c95c7e2be8967f27ffc719686 |
|
MD5 | 7639bc0e3f790824d1b4d213bea4ea37 |
|
BLAKE2b-256 | 0cef546d11b75a0a845058d10082c062ce491bcf0fb1cf091d3304641d60defb |
Hashes for airflow_multi_dagrun-2.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 65f36754b1276f2e4779d4a8ed425fd86eaa2221e8000a5bbd49e688d115e623 |
|
MD5 | fe9b3f9bddcdb507ef2ae97eb79b263a |
|
BLAKE2b-256 | 5dfca05fe42e050cafdf774112909c023081a8d06bef5ddcb95c8b59c60bff66 |