bigQueryExporterEnhanced 1.0.6

Package codes to execute queries on BQ and save to local machine, to BQ table or to GCS.

Credit

This repository is forked from https://github.com/IcarusSO/bigQueryExporter

Change Log (Compare with original version)

query_to_local()

• Create a temp table with a random hash on BQ as output table, so that simultaneous execution of the function will not overwrite each other.
• Remove the temp table after the execution. (May also set keep_temp_table=True if you wish to keep them).

bigQueryExporter

Export query data from google bigquery to local machine

Installation

pip install bigQueryExporterEnhanced

Prepare for the connection

from bigQueryExport import BigQueryExporter
bigQueryExporter = BigQueryExporter(project_name, dataset_name, bucket_name)   

Query To Table

bigQueryExporter.query_to_table(query, job_name, dataset_name)

Table To GS

bigQueryExporter.table_to_gs(destination_table, job_name)

GS To Local

bigQueryExporter.gs_to_local(bucket, job_name, data_dir_path)

Query To GS (Query to Table + Table to GS)

bigQueryExporter.query_to_gs(query, job_name)

Query To Local (Query to Table + Table to GS + GS to Local)

export_path = bigQueryExporter.query_to_local(query)

# or with the options
export_path = bigQueryExporter.query_to_local(query, 
                                              job_name='simple_query', 
                                              data_dir_path='out/',
                                              keep_temp_table=False,
                                              overwrite_output_folder=True)

Requirement

• Your server/ local machine should have the right to access the project

• Right should be granted following the insturction on Google SDK

• Execute the following command

  gcloud auth application-default login