Skip to main content

Package codes to execute queries on BQ and save to local machine, to BQ table or to GCS.

Project description

Credit

This repository is forked from https://github.com/IcarusSO/bigQueryExporter

Change Log (Compare with original version)

query_to_local()

  • Create a temp table with a random hash on BQ as output table, so that simultaneous execution of the function will not overwrite each other.
  • Remove the temp table after the execution. (May also set keep_temp_table=True if you wish to keep them).

bigQueryExporter

Export query data from google bigquery to local machine

Installation

pip install bigQueryExporterEnhanced

Prepare for the connection

from bigQueryExport import BigQueryExporter
bigQueryExporter = BigQueryExporter(project_name, dataset_name, bucket_name)   

Query To Table

bigQueryExporter.query_to_table(query, job_name, dataset_name)

Table To GS

bigQueryExporter.table_to_gs(destination_table, job_name)

GS To Local

bigQueryExporter.gs_to_local(bucket, job_name, data_dir_path)

Query To GS (Query to Table + Table to GS)

bigQueryExporter.query_to_gs(query, job_name)

Query To Local (Query to Table + Table to GS + GS to Local)

export_path = bigQueryExporter.query_to_local(query)

# or with the options
export_path = bigQueryExporter.query_to_local(query, 
                                              job_name='simple_query', 
                                              data_dir_path='out/',
                                              keep_temp_table=False,
                                              overwrite_output_folder=True)

Requirement

  • Your server/ local machine should have the right to access the project

  • Right should be granted following the insturction on Google SDK

  • Execute the following command

    gcloud auth application-default login

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for bigQueryExporterEnhanced, version 1.0.8
Filename, size File type Python version Upload date Hashes
Filename, size bigQueryExporterEnhanced-1.0.8-py3-none-any.whl (6.0 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size bigQueryExporterEnhanced-1.0.8.tar.gz (4.6 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page