Library to convert DBT manifest metadata to Airflow tasks
Project description
DBT Airflow Factory
Library to convert DBT manifest metadata to Airflow tasks
Documentation
Read the full documentation at https://dbt-airflow-factory.readthedocs.io/
Installation
Use the package manager pip to install the library:
pip install dbt-airflow-factory
Usage
The library is expected to be used inside an Airflow environment with a Kubernetes image referencing dbt.
dbt-airflow-factory's main task is to parse manifest.json
and create Airflow DAG out of it. It also reads config
files from config
directory and therefore is highly customizable (e.g., user can set path to manifest.json
).
To start, create a directory with a following structure, where manifest.json
is a file generated by dbt:
.
├── config
│ ├── base
│ │ ├── airflow.yml
│ │ ├── dbt.yml
│ │ └── k8s.yml
│ └── dev
│ └── dbt.yml
├── dag.py
└── manifest.json
Then, put the following code into dag.py
:
from dbt_airflow_factory.airflow_dag_factory import AirflowDagFactory
from os import path
dag = AirflowDagFactory(path.dirname(path.abspath(__file__)), "dev").create()
When uploaded to Airflow DAGs directory, it will get picked up by Airflow, parse manifest.json
and prepare a DAG to run.
Configuration files
It is best to look up the example configuration files in tests directory to get a glimpse of correct configs.
You can use Airflow template variables in your dbt.yml
and k8s.yml
files, as long as they are inside
quotation marks:
target: "{{ var.value.env }}"
some_other_field: "{{ ds_nodash }}"
Analogously, you can use "{{ var.value.VARIABLE_NAME }}"
in airflow.yml
, but only the Airflow variable getter.
Any other Airflow template variables will not work in airflow.yml
.
Creation of the directory with data-pipelines-cli
DBT Airflow Factory works best in tandem with data-pipelines-cli tool. dp not only prepares directory for the library to digest, but also automates Docker image building and pushes generated directory to the cloud storage of your choice.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file dbt-airflow-factory-0.35.0.tar.gz
.
File metadata
- Download URL: dbt-airflow-factory-0.35.0.tar.gz
- Upload date:
- Size: 22.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.18
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | bb2a4c5b2e83e20c65398cf0e5be22f2406ae91d653e8d976424ac403ad59746 |
|
MD5 | d94fedb34bbd01fe4cff367eaa2ab690 |
|
BLAKE2b-256 | b585bddc1cf85388101ebcdfbfdf30c9e07dbf008cb21803e5f6f5c1f72a98c8 |