An opinionated framework for ETL built on top of Airflow
Project description
gusty
Gusty is an opinionated framework for data ETL built on top of Airflow, where every task is represented by one YAML file, and each task creates a view in a database.
Structure
The .yml
approach to generating jobs within Airflow DAGs is not a new idea, but it is useful and there are a few built in benefits to it here.
-
Dependencies - Dependencies can quickly be set in
.yml
files through one of three means:- Using the
dependencies
specification, you can set dependencies between jobs in the same DAG. - Using the
external_dependencies
specification, you can set dependencies between jobs in different DAGs. - For the
MaterializedPostgresOperator
, dependencies in the same DAG that are a part of theviews
schema are automatically registered.
- Using the
-
Operator configuration - After you build an operator, you can pass parameters to it in each
.yml
job definition file. This means that, for example, if you have to call different API endpoints, you may only need to build one operator to ingest data from this API, and then can specify the endpoint to call in the.yml
job definition file. -
Support for popular notebook formats - There are currently two notebook operators,
RmdOperator
andJupyterOperator
, which enable you to simply write RMarkdown or Jupyter Notebook files and deploy them as jobs in your data pipeline. More importantly,RmdOperator
andJupyterOperator
are actually executed on separate dedicated docker containers, and interact with the Airflow container via SSH, which is useful if you want to deploy these services separately in the cloud!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.