dag-factory 0.23.0a1

Dynamically build Apache Airflow DAGs from YAML files

dag-factory

Welcome to dag-factory! dag-factory is a library for Apache Airflow® to construct DAGs declaratively via configuration files.

The minimum requirements for dag-factory are:

• Python 3.8.0+
• Apache Airflow® 2.0+

For a gentle introduction, please take a look at our Quickstart Guide. For more examples, please see the examples folder.

• Quickstart
• Benefits
• Features
  • Dynamically Mapped Tasks
  • Multiple Configuration Files
  • Callbacks
  • Custom Operators
• Notes
  • HttpSensor (since 1.0.0)
• Contributing

Benefits

• Construct DAGs without knowing Python
• Construct DAGs without learning Airflow primitives
• Avoid duplicative code
• Everyone loves YAML! ;)

Features

Multiple Configuration Files

If you want to split your DAG configuration into multiple files, you can do so by leveraging a suffix in the configuration file name.

    from dagfactory import load_yaml_dags  # load relevant YAML files as airflow DAGs

    load_yaml_dags(globals_dict=globals(), suffix=['dag.yaml'])

Custom Operators

dag-factory supports using custom operators. To leverage, set the path to the custom operator within the operator key in the configuration file. You can add any additional parameters that the custom operator requires.

...
  tasks:
    begin:
      operator: airflow.operators.dummy_operator.DummyOperator
    make_bread_1:
      operator: customized.operators.breakfast_operators.MakeBreadOperator
      bread_type: 'Sourdough'

Notes

HttpSensor (since 1.0.0)

The package airflow.providers.http.sensors.http is available for Airflow 2.0+

The following example shows response_check logic in a python file:

task_2:
  operator: airflow.providers.http.sensors.http.HttpSensor
  http_conn_id: 'test-http'
  method: 'GET'
  response_check_name: check_sensor
  response_check_file: /path/to/example1/http_conn.py
  dependencies: [task_1]

The response_check logic can also be provided as a lambda:

task_2:
  operator: airflow.providers.http.sensors.http.HttpSensor
  http_conn_id: 'test-http'
  method: 'GET'
  response_check_lambda: 'lambda response: "ok" in response.text'
  dependencies: [task_1]

License

To learn more about the terms and conditions for use, reproduction and distribution, read the Apache License 2.0.

Privacy Notice

This project follows Astronomer's Privacy Policy.

For further information, read this

Security Policy

Check the project's Security Policy to learn how to report security vulnerabilities in DAG Factory and how security issues reported to the DAG Factory security team are handled.