airflow-bootstrap-utils 0.2.1

Exported Console Scripts

• airflow-generate-dag: Generate Airflow DAGs from control and configuration files.

• airflow-validate-control-file: Validate control files.

Installation

`shell pip install airflow-bootstrap-utils `

Usage

generate-airflow-dag-script --control_file /tmp/airflow-demo/simple_workflow.yaml --outdir /tmp/airflow-demo
--config_file was not specified and therefore was set to '/tmp/airflow-bootstrap-utils/venv/lib/python3.10/site-packages/airflow_bootstrap_utils/conf/config.yaml'
--template_dir was not specified and therefore was set to '/tmp/airflow-bootstrap-utils/venv/lib/python3.10/site-packages/airflow_bootstrap_utils/templates'
--logfile was not specified and therefore was set to '/tmp/airflow-demo/generate_dag.log'
Wrote Airflow DAG Python script list to file '/tmp/sundaram/airflow-bootstrap-utils/user_processing_L2/2025-02-15-101314/airflow_dag_scripts.txt'

Contents of airflow_dag_scripts.txt:

cat /tmp/sundaram/airflow-bootstrap-utils/user_processing_L2/2025-02-15-101314/airflow_dag_scripts.txt
## method-created: /tmp/airflow-bootstrap-utils/venv/lib/python3.10/site-packages/airflow_bootstrap_utils/manager.py
## date-created: 2025-02-15-101314
## created-by: sundaram
## control-file: /tmp/airflow-demo/simple_workflow.yaml
## logfile: /tmp/airflow-demo/generate_dag.log
/tmp/sundaram/airflow-bootstrap-utils/user_processing_L1/2025-02-15-101314/L1/user_processing_L1.airflow.dag.py
/tmp/sundaram/airflow-bootstrap-utils/user_processing_L2/2025-02-15-101314/L2/user_processing_L2.airflow.dag.py

Contents of L1 airflow script:

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime

#------------------------------
# Airflow DAG definition
#------------------------------
dag = DAG(
    'user_processing_L1',
    description='This is an example DAG in the airflow-bootstrap-utils.  Please provide a better description in your instance in your copy of the control file.',
    schedule_interval='“@weekly”',
    start_date=datetime(2024,3,22),
    catchup=False,
    )

#------------------------------
# Airflow task defintions
#------------------------------

"""
The do_something executable is going to do something.
It is going to use the lab number argument to do
something.
The lab number is a required parameter.
"""
do_something = BashOperator(
    task_id='do_something',
    bash_command='bash /opt/do_something.sh --lab_number L1',
    dag=dag
)


"""
The do_something_else executable is going to do something else.
It is going to use the lab number argument to do
something else.
The lab number is a required parameter.
"""
do_something_else = BashOperator(
    task_id='do_something_else',
    bash_command='bash /opt/do_something_else.sh --lab_number L1',
    dag=dag
)

#------------------------------
# Define the task dependencies
#------------------------------
do_something >> do_something_else

Contents of L2 airflow script:

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime

#------------------------------
# Airflow DAG definition
#------------------------------
dag = DAG(
    'user_processing_L2',
    description='This is an example DAG in the airflow-bootstrap-utils.  Please provide a better description in your instance in your copy of the control file.',
    schedule_interval='“@weekly”',
    start_date=datetime(2024,3,22),
    catchup=False,
    )

#------------------------------
# Airflow task defintions
#------------------------------

"""
The do_something executable is going to do something.
It is going to use the lab number argument to do
something.
The lab number is a required parameter.
"""
do_something = BashOperator(
    task_id='do_something',
    bash_command='bash /opt/do_something.sh --lab_number L1',
    dag=dag
)


"""
The do_something_else executable is going to do something else.
It is going to use the lab number argument to do
something else.
The lab number is a required parameter.
"""
do_something_else = BashOperator(
    task_id='do_something_else',
    bash_command='bash /opt/do_something_else.sh --lab_number L1',
    dag=dag
)

#------------------------------
# Define the task dependencies
#------------------------------
do_something >> do_something_else