Common functions for airflow
Project description
airflow-commons
A python package that contains common functionalities for airflow
Installation
Use the package manager pip to install airflow-commons.
pip install airflow-commons
Modules
- bigquery_operator: With this module you can manage your Google BigQuery operations.
- mysql_operator: Using this module, you can connect to your MySQL data source and manage your data operations.
- s3_operator: This operator connects to your s3 bucket and lets you manage your bucket.
- glossary: This module consists of constants used across project
- sql_resources: Template BigQuery and MySQL queries such as merge, delete, select etc. are located here.
- utils: Generic methods like connection, querying etc. are implemented in this module.
Usage
- Sample deduplication code works like:
from airflow_commons import bigquery_operator
bigquery_operator.deduplicate(
service_account_file="path_to_file",
start_date="01-01-2020 14:00:00",
end_date="01-01-2020 15:00:00",
project_id="bigquery_project_id",
source_dataset="source_dataset",
source_table="source_table",
target_dataset="target_dataset",
target_table="target_table",
oldest_allowable_target_partition="01-01-2015 00:00:00",
primary_keys=["primary_keys"],
time_columns=["time_columns"],
allow_partition_pruning=True,
)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
airflow-commons-0.0.63.tar.gz
(21.7 kB
view details)
Built Distribution
File details
Details for the file airflow-commons-0.0.63.tar.gz
.
File metadata
- Download URL: airflow-commons-0.0.63.tar.gz
- Upload date:
- Size: 21.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.9.15
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 84f8c291cd7a58a5bca21fea7a41734df874133c430085bd4f4c457de6968d33 |
|
MD5 | ad389c949e67f852d5320d0f005b76f1 |
|
BLAKE2b-256 | c686fedeb2e201b98ddb72b1031efb7ed1592996362351fb38165cbcb23a9578 |
File details
Details for the file airflow_commons-0.0.63-py3-none-any.whl
.
File metadata
- Download URL: airflow_commons-0.0.63-py3-none-any.whl
- Upload date:
- Size: 33.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.9.15
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8d7415aeb24f5c37ca692120ac8d101fafb1ca8070fa663df84436a0e63d2bb2 |
|
MD5 | 97aaf5c94473d539a9fe6a509c0ff5e0 |
|
BLAKE2b-256 | da9668ff0b2e7c1da2ffb81b703e4c626b2a1032fb72913e23ce56d3046fcf40 |