Skip to main content

Common functions for airflow

Project description

airflow-commons

A python package that contains common functionalities for airflow

Installation

Use the package manager pip to install airflow-commons.

pip install airflow-commons

Modules

  • bigquery_operator: With this module you can manage your Google BigQuery operations.
  • mysql_operator: Using this module, you can connect to your MySQL data source and manage your data operations.
  • s3_operator: This operator connects to your s3 bucket and lets you manage your bucket.
  • glossary: This module consists of constants used across project
  • sql_resources: Template BigQuery and MySQL queries such as merge, delete, select etc. are located here.
  • utils: Generic methods like connection, querying etc. are implemented in this module.

Usage

  • Sample deduplication code works like:
from airflow_commons import bigquery_operator

bigquery_operator.deduplicate(
        service_account_file="path_to_file",
        start_date="01-01-2020 14:00:00",
        end_date="01-01-2020 15:00:00",
        project_id="bigquery_project_id",
        source_dataset="source_dataset",
        source_table="source_table",
        target_dataset="target_dataset",
        target_table="target_table",
        oldest_allowable_target_partition="01-01-2015 00:00:00",
        primary_keys=["primary_keys"],
        time_columns=["time_columns"],
        allow_partition_pruning=True,
    )

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow-commons-0.0.76.tar.gz (25.0 kB view details)

Uploaded Source

Built Distribution

airflow_commons-0.0.76-py3-none-any.whl (37.6 kB view details)

Uploaded Python 3

File details

Details for the file airflow-commons-0.0.76.tar.gz.

File metadata

  • Download URL: airflow-commons-0.0.76.tar.gz
  • Upload date:
  • Size: 25.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.18

File hashes

Hashes for airflow-commons-0.0.76.tar.gz
Algorithm Hash digest
SHA256 a25a9bed0b75dd24cf2f35a5a9a7702a58788edcee204b696724a0d312d95c1f
MD5 92d2c3b769cfca93320c88f1a5c31c1e
BLAKE2b-256 9a047c330a1906e264b2e20a7990c69370738289dc93e751e81811595e41ea0e

See more details on using hashes here.

File details

Details for the file airflow_commons-0.0.76-py3-none-any.whl.

File metadata

File hashes

Hashes for airflow_commons-0.0.76-py3-none-any.whl
Algorithm Hash digest
SHA256 162e6d3831b815a062bb79daff7d25fef2f48ff0bad91f02188de8e9e749cb29
MD5 3ddbc6faab3210f45a7de178defabf56
BLAKE2b-256 12a4179086ba0ffcc435c0e8a960247d483e286911b3fd1afb792beffccb9858

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page