Skip to main content

Common functions for airflow

Project description

airflow-commons

A python package that contains common functionalities for airflow

Installation

Use the package manager pip to install airflow-commons.

pip install airflow-commons

Modules

  • bigquery_operator: With this module you can manage your Google BigQuery operations.
  • mysql_operator: Using this module, you can connect to your MySQL data source and manage your data operations.
  • s3_operator: This operator connects to your s3 bucket and lets you manage your bucket.
  • glossary: This module consists of constants used across project
  • sql_resources: Template BigQuery and MySQL queries such as merge, delete, select etc. are located here.
  • utils: Generic methods like connection, querying etc. are implemented in this module.

Usage

  • Sample deduplication code works like:
from airflow_commons import bigquery_operator

bigquery_operator.deduplicate(
        service_account_file="path_to_file",
        start_date="01-01-2020 14:00:00",
        end_date="01-01-2020 15:00:00",
        project_id="bigquery_project_id",
        source_dataset="source_dataset",
        source_table="source_table",
        target_dataset="target_dataset",
        target_table="target_table",
        oldest_allowable_target_partition="01-01-2015 00:00:00",
        primary_keys=["primary_keys"],
        time_columns=["time_columns"],
        allow_partition_pruning=True,
    )

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow-commons-0.0.53.tar.gz (18.2 kB view details)

Uploaded Source

Built Distribution

airflow_commons-0.0.53-py3-none-any.whl (28.3 kB view details)

Uploaded Python 3

File details

Details for the file airflow-commons-0.0.53.tar.gz.

File metadata

  • Download URL: airflow-commons-0.0.53.tar.gz
  • Upload date:
  • Size: 18.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.9.12

File hashes

Hashes for airflow-commons-0.0.53.tar.gz
Algorithm Hash digest
SHA256 078c03761ab009ba487be89d46b1e50a534337a74ba306918bc2cba22b421ade
MD5 6d68115946569c082f52b5e0f3a553c4
BLAKE2b-256 aac7bc33f15d8b9b1e9f1c389beb7038e300d3755b4e5445c56bfdcb335f1564

See more details on using hashes here.

File details

Details for the file airflow_commons-0.0.53-py3-none-any.whl.

File metadata

File hashes

Hashes for airflow_commons-0.0.53-py3-none-any.whl
Algorithm Hash digest
SHA256 fe3a9f66e43432023a6e7ecb2cd32b5de9e59435d98dc2ba34f0dd842d1a9d0a
MD5 b34d213100d9a1540d4cef576a8331ba
BLAKE2b-256 ea1909c58fafcbe79cc585625ac65387ad5636fd05bafdf478918f3684931a5b

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page