Skip to main content

Common functions for airflow

Project description

airflow-commons

A python package that contains common functionalities for airflow

Installation

Use the package manager pip to install airflow-commons.

pip install airflow-commons

Modules

  • bigquery_operator: With this module you can manage your Google BigQuery operations.
  • mysql_operator: Using this module, you can connect to your MySQL data source and manage your data operations.
  • s3_operator: This operator connects to your s3 bucket and lets you manage your bucket.
  • glossary: This module consists of constants used across project
  • sql_resources: Template BigQuery and MySQL queries such as merge, delete, select etc. are located here.
  • utils: Generic methods like connection, querying etc. are implemented in this module.

Usage

  • Sample deduplication code works like:
from airflow_commons import bigquery_operator

bigquery_operator.deduplicate(
        service_account_file="path_to_file",
        start_date="01-01-2020 14:00:00",
        end_date="01-01-2020 15:00:00",
        project_id="bigquery_project_id",
        source_dataset="source_dataset",
        source_table="source_table",
        target_dataset="target_dataset",
        target_table="target_table",
        oldest_allowable_target_partition="01-01-2015 00:00:00",
        primary_keys=["primary_keys"],
        time_columns=["time_columns"],
        allow_partition_pruning=True,
    )

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow-commons-0.0.63.tar.gz (21.7 kB view details)

Uploaded Source

Built Distribution

airflow_commons-0.0.63-py3-none-any.whl (33.3 kB view details)

Uploaded Python 3

File details

Details for the file airflow-commons-0.0.63.tar.gz.

File metadata

  • Download URL: airflow-commons-0.0.63.tar.gz
  • Upload date:
  • Size: 21.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.9.15

File hashes

Hashes for airflow-commons-0.0.63.tar.gz
Algorithm Hash digest
SHA256 84f8c291cd7a58a5bca21fea7a41734df874133c430085bd4f4c457de6968d33
MD5 ad389c949e67f852d5320d0f005b76f1
BLAKE2b-256 c686fedeb2e201b98ddb72b1031efb7ed1592996362351fb38165cbcb23a9578

See more details on using hashes here.

File details

Details for the file airflow_commons-0.0.63-py3-none-any.whl.

File metadata

File hashes

Hashes for airflow_commons-0.0.63-py3-none-any.whl
Algorithm Hash digest
SHA256 8d7415aeb24f5c37ca692120ac8d101fafb1ca8070fa663df84436a0e63d2bb2
MD5 97aaf5c94473d539a9fe6a509c0ff5e0
BLAKE2b-256 da9668ff0b2e7c1da2ffb81b703e4c626b2a1032fb72913e23ce56d3046fcf40

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page