Skip to main content

Vakanties.nl pypi package

Project description

Vaknl-gcp

Package for working with dataclasses and Google cloud instances as Bigquery and Storage. For this to work the dataclasses need to contain only basic python variables as: str, int, dict, list etc.

Prerequisites

These modules dependant on the environment variable GOOGLE_CLOUD_PROJECT a.k.a. gcp project id.

Bigquery

execute_query(query):

Execute query and log errors

returns query_job

stream_to_bigquery(objects: list, table_ref):

Cast python objects to json and stream them to GBQ Note: this is more expensive compared to using buckets but also quicker

returns table ref

write_disposition_bucket(table_ref, blob_name, write_disposition):

Get data from bucket to GBQ using a write_disposition method

Requires: bucket with name storage_to_bigquery-[project-id]

  • WRITE_DISPOSITION_UNSPECIFIED Unknown.
  • WRITE_EMPTY This job should only be writing to empty tables.
  • WRITE_TRUNCATE This job will truncate table data and write from the beginning.
  • WRITE_APPEND This job will append to a table.
returns load_job.result 

Storage

storage_to_bigquery(objects: list, table_ref, write_disposition):

Function that stores data into multiple storage blobs. Afterwards these wil be composed into one storage blob The reason for this process is to downsize the sie of the data send to Google Cloud Storage.

Requires: bucket with name storage_to_bigquery-[project-id]

get_bucket(name_bucket: str):

returns name_bucket

get_blobs_from_bucket(bucket):

returns blobs:list

upload_from_string(blob, string):

returns blob.public_url

Secrets (beta)

get_secret(secret_id):

returns json

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for vaknl-gcp, version 1.1.7
Filename, size File type Python version Upload date Hashes
Filename, size vaknl_gcp-1.1.7-py3-none-any.whl (6.4 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size vaknl-gcp-1.1.7.tar.gz (4.0 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page