airflow-bootstrap-utils 0.2.1

Collection of Python tools to generate Airflow DAGs from control and configuration files.

Airflow Bootstrap Utils

Collection of Python tools to generate Airflow DAGs from control and configuration files.

Exported Console Scripts

• airflow-generate-dag: Generate Airflow DAGs from control and configuration files.

• airflow-validate-control-file: Validate control files.

Installation

`shell pip install airflow-bootstrap-utils `

Usage

generate-airflow-dag-script --control_file /tmp/airflow-demo/simple_workflow.yaml --outdir /tmp/airflow-demo
--config_file was not specified and therefore was set to '/tmp/airflow-bootstrap-utils/venv/lib/python3.10/site-packages/airflow_bootstrap_utils/conf/config.yaml'
--template_dir was not specified and therefore was set to '/tmp/airflow-bootstrap-utils/venv/lib/python3.10/site-packages/airflow_bootstrap_utils/templates'
--logfile was not specified and therefore was set to '/tmp/airflow-demo/generate_dag.log'
Wrote Airflow DAG Python script list to file '/tmp/sundaram/airflow-bootstrap-utils/user_processing_L2/2025-02-15-101314/airflow_dag_scripts.txt'

Contents of airflow_dag_scripts.txt:

cat /tmp/sundaram/airflow-bootstrap-utils/user_processing_L2/2025-02-15-101314/airflow_dag_scripts.txt
## method-created: /tmp/airflow-bootstrap-utils/venv/lib/python3.10/site-packages/airflow_bootstrap_utils/manager.py
## date-created: 2025-02-15-101314
## created-by: sundaram
## control-file: /tmp/airflow-demo/simple_workflow.yaml
## logfile: /tmp/airflow-demo/generate_dag.log
/tmp/sundaram/airflow-bootstrap-utils/user_processing_L1/2025-02-15-101314/L1/user_processing_L1.airflow.dag.py
/tmp/sundaram/airflow-bootstrap-utils/user_processing_L2/2025-02-15-101314/L2/user_processing_L2.airflow.dag.py

Contents of L1 airflow script:

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime

#------------------------------
# Airflow DAG definition
#------------------------------
dag = DAG(
    'user_processing_L1',
    description='This is an example DAG in the airflow-bootstrap-utils.  Please provide a better description in your instance in your copy of the control file.',
    schedule_interval='“@weekly”',
    start_date=datetime(2024,3,22),
    catchup=False,
    )

#------------------------------
# Airflow task defintions
#------------------------------

"""
The do_something executable is going to do something.
It is going to use the lab number argument to do
something.
The lab number is a required parameter.
"""
do_something = BashOperator(
    task_id='do_something',
    bash_command='bash /opt/do_something.sh --lab_number L1',
    dag=dag
)


"""
The do_something_else executable is going to do something else.
It is going to use the lab number argument to do
something else.
The lab number is a required parameter.
"""
do_something_else = BashOperator(
    task_id='do_something_else',
    bash_command='bash /opt/do_something_else.sh --lab_number L1',
    dag=dag
)

#------------------------------
# Define the task dependencies
#------------------------------
do_something >> do_something_else

Contents of L2 airflow script:

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime

#------------------------------
# Airflow DAG definition
#------------------------------
dag = DAG(
    'user_processing_L2',
    description='This is an example DAG in the airflow-bootstrap-utils.  Please provide a better description in your instance in your copy of the control file.',
    schedule_interval='“@weekly”',
    start_date=datetime(2024,3,22),
    catchup=False,
    )

#------------------------------
# Airflow task defintions
#------------------------------

"""
The do_something executable is going to do something.
It is going to use the lab number argument to do
something.
The lab number is a required parameter.
"""
do_something = BashOperator(
    task_id='do_something',
    bash_command='bash /opt/do_something.sh --lab_number L1',
    dag=dag
)


"""
The do_something_else executable is going to do something else.
It is going to use the lab number argument to do
something else.
The lab number is a required parameter.
"""
do_something_else = BashOperator(
    task_id='do_something_else',
    bash_command='bash /opt/do_something_else.sh --lab_number L1',
    dag=dag
)

#------------------------------
# Define the task dependencies
#------------------------------
do_something >> do_something_else

History

0.1.0 (2024-03-20)

• First release on PyPI.