Project description

BigQuery logger handler for Airflow

Installation

pip install airflow-bigquerylogger

Configuration

AIRFLOW__CORE__REMOTE_LOGGING='true'
AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER='gs://bucket/path'
AIRFLOW__CORE__REMOTE_LOG_CONN_ID='gcs_log'
AIRFLOW__CORE__LOGGING_CONFIG_CLASS='bigquerylogger.config.LOGGING_CLASS'
AIRFLOW__CORE__LOG_BIGQUERY_DATASET='dataset.table'
AIRFLOW__CORE__LOG_BIGQUERY_LIMIT=50

Google Cloud BigQuery

Rows that were written to a table recently via streaming (using the tabledata.insertall method) cannot be modified using UPDATE, DELETE, or MERGE statements. I recommend setting up a table retention!

Credits

Thanks to Bluecore engineering team for this usefull article.

Project details

These details have not been verified by PyPI

Release history Release notifications | RSS feed

This version

0.4.5

Apr 7, 2020

0.4.4

Apr 7, 2020

0.4.3

Apr 7, 2020

0.4.2

Apr 7, 2020

0.4.1

Apr 7, 2020

0.4.0

Apr 7, 2020

0.3.0

Apr 7, 2020

0.2.0

Apr 6, 2020

0.1.0

Apr 3, 2020

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow-bigquerylogger-0.4.5.tar.gz (3.7 kB view details)

Uploaded Apr 7, 2020 Source

Built Distribution

airflow_bigquerylogger-0.4.5-py3-none-any.whl (5.3 kB view details)

Uploaded Apr 7, 2020 Python 3

File details

Details for the file airflow-bigquerylogger-0.4.5.tar.gz.

File metadata

Download URL: airflow-bigquerylogger-0.4.5.tar.gz
Upload date: Apr 7, 2020
Size: 3.7 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.7.7

File hashes

Hashes for airflow-bigquerylogger-0.4.5.tar.gz
Algorithm	Hash digest
SHA256	`5bcd66d096045af05be1626fb81faf6df343d19d84883e64f34b289507498837`
MD5	`d750fd3fe9373131c0a2fecad63ae7e0`
BLAKE2b-256	`71e8448b3144d9f1955324c22f8d8a20fc52690f02805d64731549c588d608d2`

See more details on using hashes here.

File details

Details for the file airflow_bigquerylogger-0.4.5-py3-none-any.whl.

File metadata

Download URL: airflow_bigquerylogger-0.4.5-py3-none-any.whl
Upload date: Apr 7, 2020
Size: 5.3 kB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.7.7

File hashes

Hashes for airflow_bigquerylogger-0.4.5-py3-none-any.whl
Algorithm	Hash digest
SHA256	`8dda82b74e38dd46cfeeee6b08c4089cac1d0e64f433367cf1054991006f22d0`
MD5	`22b6274f367ece59d503384ff30556fb`
BLAKE2b-256	`6a409713d044875b14ea9831d4f2e314fd96d39ac7460f2017c41820502b0a8c`