Skip to main content

Dynamically build Airflow DAGs from YAML files

Project description


Github Actions Coverage PyPi Code Style Downloads

dag-factory is a library for dynamically generating Apache Airflow DAGs from YAML configuration files.


To install dag-factory run pip install dag-factory. It requires Python 3.6.0+ and Apache Airflow 1.10+.


After installing dag-factory in your Airflow environment, there are two steps to creating DAGs. First, we need to create a YAML configuration file. For example:

    owner: 'example_owner'
    start_date: 2018-01-01  # or '2 days'
    end_date: 2018-01-05
    retries: 1
    retry_delay_sec: 300
  schedule_interval: '0 3 * * *'
  concurrency: 1
  max_active_runs: 1
  dagrun_timeout_sec: 60
  default_view: 'tree'  # or 'graph', 'duration', 'gantt', 'landing_times'
  orientation: 'LR'  # or 'TB', 'RL', 'BT'
  description: 'this is an example dag!'
  on_success_callback_name: print_hello
  on_success_callback_file: /usr/local/airflow/dags/
  on_failure_callback_name: print_hello
  on_failure_callback_file: /usr/local/airflow/dags/
      operator: airflow.operators.bash_operator.BashOperator
      bash_command: 'echo 1'
      operator: airflow.operators.bash_operator.BashOperator
      bash_command: 'echo 2'
      dependencies: [task_1]
      operator: airflow.operators.bash_operator.BashOperator
      bash_command: 'echo 3'
      dependencies: [task_1]

Then in the DAGs folder in your Airflow environment you need to create a python file like this:

from airflow import DAG
import dagfactory

dag_factory = dagfactory.DagFactory("/path/to/dags/config_file.yml")


And this DAG will be generated and ready to run in Airflow!



  • Construct DAGs without knowing Python
  • Construct DAGs without learning Airflow primitives
  • Avoid duplicative code
  • Everyone loves YAML! ;)


Contributions are welcome! Just submit a Pull Request or Github Issue.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for dag-factory, version 0.7.1
Filename, size File type Python version Upload date Hashes
Filename, size dag_factory-0.7.1-py2.py3-none-any.whl (10.1 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size dag-factory-0.7.1.tar.gz (9.6 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page