Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

A collection of BigQuery utilities

Project description

bq-utils - A collection of BigQuery utilities

Currently the utilities covered in this library are limited to managing table and view descriptions. It's quite useful to have the field descriptions available on the schema tab and on the query editor pane on the BigQuery UI and this library helps you with uploading them and copying them around.

Install

pip install bq-utils

Python

Copy descriptions between tables and views

from google.cloud import bigquery
from bqutils.bigquery_description_manager import BigQueryDescriptionManager

bq_client = bigquery.Client()
description_manager = BigQueryDescriptionManager(bq_client)
source_table_id = 's_project.s_dataset.s_table'
target_table_id = 't_project.t_dataset.t_table'
description_manager.copy_field_descriptions(source_table_id, target_table_id)

Upload descriptions from csv

from google.cloud import bigquery
from bqutils.bigquery_description_manager import BigQueryDescriptionManager

bq_client = bigquery.Client()
description_manager = BigQueryDescriptionManager(bq_client)
descriptions_csv_path = 'table_descriptions.csv'
target_table_id = 't_project.t_dataset.t_table'
description_manager.upload_field_descriptions(descriptions_csv_path, target_table_id)

Command line

Usage

usage: __main__.py [-h] [--source SOURCE] --target TARGET
                   [--csv_path CSV_PATH] [--debug]
                   {desccopy,descupload}

Copy or upload field descriptions for BigQuery tables/views

positional arguments:
  {desccopy,descupload}

optional arguments:
  -h, --help            show this help message and exit
  --source SOURCE       fully-qualified source table ID
  --target TARGET       fully-qualified target table ID
  --csv_path CSV_PATH   path for the csv file
  --debug               set debug mode on, default is false

Copy descriptions between tables and views

python -m bqutils desccopy --source s_project.s_dataset.s_table --target t_project.t_dataset.t_table

Upload descriptions from csv

python -m bqutils descupload --csv_path table_descriptions.csv --target t_project.t_dataset.t_table

Project details


Release history Release notifications

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for bq-utils, version 0.1.0
Filename, size File type Python version Upload date Hashes
Filename, size bq_utils-0.1.0-py3-none-any.whl (5.3 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size bq-utils-0.1.0.tar.gz (3.6 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page