Debussy Airflow

A set of Airflow Hooks, Operators and Sensors for Debussy Concert,

the core component of Debussy Framework

⚠️Not exactly a provider

This package provides operators, sensors, and hooks that enables running Debussy Concert on Apache Airflow. However, it's not a traditional Airflow provider package in the sense Debussy Concert is not a tool, but rather a framework for data pipeline development and code generation. Therefore, our hooks, operators and sensors acts as interfaces, extending other Airflow providers, and abstracting their usage.

Installation

You can install this package via

pip install debussy-airflow

Prerequisites

An environment running apache-airflow 2.2.0+ with the following requirements (already managed as dependencies):

- apache-airflow-providers-google == 8.0.0
- facebook-business == 13.0.0
- mysql-connector-python == 8.0.24
- paramiko == 2.8.1

Modules

Hooks

• db_api_hook
• facebook_ads_hook
• http_api_hook
• http_hook
• storage_hook

Operators

• basic_operator
• db_to_storage_operator
• facebook_ads_to_storage_operator
• http_operator
• rest_api_to_storage_operator
• storage_to_rdbms_operator
• storage_to_storage_operator

Sensors

• http_sensor

Examples

See the examples directory for an example DAG.

Issues

Please submit issues and pull requests in our official repo: https://github.com/DotzInc/debussy_airflow

We are happy to hear from you. Please email any feedback to the authors at dataengineer@dotz.com.

Project Contributors and Maintainers

This project is built with active contributions from:

• Lawrence Fernandes
• Nilton Duarte
• Maycon Francis

This project is formatted via black:

pip install black
black .

Changelog

• 0.0.1 - First release
• 0.0.2 - Fix naming conventions for hooks, operators and sensors, and updated example_dags
• 0.0.3 - Improved README description, and setup.py
• 0.0.4 - Moved config to setup.cfg, added requirements.txt and CHANGELOG.md
• 0.0.5 - Fix DbApiHookInterface class name on db_api_hook and import path on storage_to_rdbms_operator
• 0.0.6 - Improvements on db_api_hook and storage_to_storage_operator, including support to PostgreSQL, and updated requirements