Airflow DAGs done declaratively
Project description
Airflow declarative DAGs via YAML.
Compatibility:
- Python 2.7 / 3.5+
- Airflow 1.10.4+
Key Features
- Declarative DAGs in plain text YAML helps a lot to understand how DAG will looks like. Made for humans, not programmers.
- It makes extremely hard to turn your DAGs into code mess. Even if you make complicated YAMLs generator the result would be readable for humans.
- No more guilty about coupling business logic with task management system (Airflow). They now could coexists separated.
- Static analysis becomes a trivial task.
- It’s a good abstraction to create your own scheduler/worker compatible with original Airflow one.
Examples
Check tests/dags directory for example of DAGs which will works and which won’t. Use src/airflow_declarative/schema.py module for the reference about YAML file schema. It should be self descriptive.
Don’t be shy to experiment: trafaret-config will help you to understand what had gone wrong and why and where.
Usage
We provide support for two installation options:
- As a complementary side package for the upstream Airflow.
- As a built-in Airflow functionality using patches for Airflow.
Upstream Airflow
The idea is to put a Python script to the dags_folder which would load the declarative dags via airflow_declarative. More details: Installation using Upstream Airflow.
import os import airflow_declarative # Assuming that the yaml dags are located in the same directory # as this Python module: root = os.path.dirname(__file__) dags_list = [ airflow_declarative.from_path(os.path.join(root, item)) for item in os.listdir(root) if item.endswith((".yml", ".yaml")) ] globals().update({dag.dag_id: dag for dags in dags_list for dag in dags})
Patched Airflow
We provide ready to use patches in the patches directory. To use them you will need to apply a patch to a corresponding Airflow version and then build it yourself. More details: Installation using Patched Airflow.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for airflow_declarative-1.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ebdddaef9436155839732d3ac9635551a46b90ed20b43e3ec5d96499d658a625 |
|
MD5 | b184f5d2d317c99e0a72909cab994a9c |
|
BLAKE2-256 | 6c03c41df4e3df2ebfdbbebb02294b3ce636b64361a6fd540682fe77b0725068 |