No project description provided
Project description
Debussy Airflow
A set of Airflow Hooks, Operators and Sensors for Debussy Concert,
the core component of Debussy Framework
⚠️Not exactly a provider
This package provides operators, sensors, and hooks that enables running Debussy Concert on Apache Airflow. However, it's not a traditional Airflow provider package in the sense Debussy Concert is not a tool, but rather a framework for data pipeline development and code generation. Therefore, our hooks, operators and sensors acts as interfaces, extending other Airflow providers, and abstracting their usage.
Installation
You can install this package via
pip install debussy-airflow
Prerequisites
An environment running apache-airflow
2.2.0+ with the following requirements (already managed as dependencies):
- apache-airflow-providers-google == 8.0.0
- facebook-business == 13.0.0
- mysql-connector-python == 8.0.24
- paramiko == 2.8.1
Modules
Hooks
Operators
- basic_operator
- db_to_storage_operator
- facebook_ads_to_storage_operator
- http_operator
- rest_api_to_storage_operator
- storage_to_rdbms_operator
- storage_to_storage_operator
Sensors
Examples
See the examples directory for an example DAG.
Issues
Please submit issues and pull requests in our official repo: https://github.com/DotzInc/debussy_airflow
We are happy to hear from you. Please email any feedback to the authors at dataengineer@dotz.com.
Project Contributors and Maintainers
This project is built with active contributions from:
This project is formatted via black
:
pip install black
black .
Changelog
- 0.0.1 - First release
- 0.0.2 - Fix naming conventions for hooks, operators and sensors, and updated example_dags
- 0.0.3 - Improved README description, and setup.py
- 0.0.4 - Moved config to setup.cfg, added requirements.txt and CHANGELOG.md
- 0.0.5 - Fix DbApiHookInterface class name on db_api_hook and import path on storage_to_rdbms_operator
- 0.0.6 - Improvements on db_api_hook and storage_to_storage_operator, including support to PostgreSQL, and updated requirements
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.