Skip to main content

A local client for connecting to and working with Google BigQuery

Project description

shipyard-bigquery

The shipyard-bigquery Python package is a local client for connecting to BigQuery to perform common operations such as loading and unloading data.

Installation

python3 -m pip install shipyard-bigquery

Usage

To initialize the client, provide the service account (in JSON format) as a string:

import os
from shipyard_bigquery import BigQueryClient 
service_account = os.getenv('GOOGLE_APPLICATION_CREDENTIALS') # in this case, service account is stored as an environment variable
client = BigQueryClient(service_account)
client.connect()

Upload a file to a table in BigQuery

The upload method allows for the loading of a local CSV file to BigQuery and supports two different types of upload: overwrite and append.

client.upload(file = "my_file.csv", dataset = "my_dataset", table = "my_table", upload_type = 'overwrite')

Additionally, a the schema of the file to be loaded can be decalred ahead of time, otherwise they will be infered

schema = '[{"name": "string_col", "type": "string"}, {"name": "char_col", "type": "string"}, {"name": "int_col", "type": "Int64"}, {"name": "float_col", "type": "Float64"}, {"name": "bool_col", "type": "Bool"}, {"name": "date_col", "type": "Date"},{"name": "datetime_col", "type": "Datetime"}]'
client.upload(file = "my_file.csv", dataset = "my_dataset", table = "my_table", upload_type = 'overwrite',schema = schema)

Fetch Query Results

The fetch method allows for the downloading of a SQL query to a pandas dataframe

df = client.fetch('select * from my_table limit 10')

Execute Remote Queries

The execute method allows for you to execute queries without return values in BigQuery. Common operations would be dropping tables, deleting tables, or altering tables

client.execute_query('drop table if exists my_table')

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

shipyard_bigquery-0.1.4.tar.gz (7.1 kB view details)

Uploaded Source

Built Distribution

shipyard_bigquery-0.1.4-py3-none-any.whl (9.9 kB view details)

Uploaded Python 3

File details

Details for the file shipyard_bigquery-0.1.4.tar.gz.

File metadata

  • Download URL: shipyard_bigquery-0.1.4.tar.gz
  • Upload date:
  • Size: 7.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.4.2 CPython/3.9.16 Darwin/22.5.0

File hashes

Hashes for shipyard_bigquery-0.1.4.tar.gz
Algorithm Hash digest
SHA256 33e0c6a1b68699978bff5d3a1ac19714e69323811a5e8bfc28c7a8a450817669
MD5 91dbf3db788a1b88923c76ae325ab85a
BLAKE2b-256 6beb9ea41334b592af8aa3001debfdfb0ae61632838b9ad7a30aaafb63c12f44

See more details on using hashes here.

File details

Details for the file shipyard_bigquery-0.1.4-py3-none-any.whl.

File metadata

File hashes

Hashes for shipyard_bigquery-0.1.4-py3-none-any.whl
Algorithm Hash digest
SHA256 410fe071201dc20c6ad1735480bcd2ce1534886d25898ce88a95a5d020e91716
MD5 10916ac3726b425c1c2cf143dd362b31
BLAKE2b-256 e3fb84c9c9d963481fad86de38cee07ddbdaaabe25f971f383959835e628d469

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page