Skip to main content

A Big Query json utility package

Project description

This is bqtools a handy set of utils to help create big query tables from json objects and make them into tables (so get the schema) and to create json new line files. Plus a set of common functions to ease writing biq query python code

Goals

  • Simplify handling and move between big query stryuctured data and json data
  • Allow you to create big query table schemas from json structures (as resource or schema objects) uses reflection of types
  • Provides easy generation of views for day partitioned tabled
    • head - latest data
    • diff views for time snap shots of data i.e. each day partition has current view of data
  • Calculate valid json structures from representative json data that can be used as basis of big query schemas
  • Clean json data such that it can be loaded into big query
    • Replace bare lists with dictionaries
    • Replace field names with valid values that can be column names in big query (removes spaces, characters not allowed in field names using same algorithms big query uses when auto detecting schemas)
    • Encodes json output of dates, datetimes, times, timedeltas encoded in format acceptable for big query corresponding field types
  • Generate code for bq command line tool from bq table structures
  • Simplify common tasks of handling big query data
    • Basic tests on dataset or tables existing
    • Schema patching compare an existing schema to a template json object calculate if changed and generate a merged schema that can be used in a patch
    • Flattening views to avoid view depth limits
import bqtools

# if you load a json object say something like
foo = {
        "id":1,
        "description":""
        "aboolean":False
      }

# generate a schema
table = {
   "type":"TABLE",
   "location":os.environ["location"],
   "tableReference":{
       "projectId": os.environ["projectid"],
       "datasetId": os.environ["dataset"],
       "tableId": key
   },
   "timePartitioning":{
       "type": "DAY",
       "expirationMs": "94608000000"
   },
   "schema": {}
}

# use bqtools to create a schema structure
table["schema"]["fields"] = bqtools.get_bq_schema_from_json_repr(foo)



Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for bqtools-json, version 0.4.9
Filename, size & hash File type Python version Upload date
bqtools_json-0.4.9-py2-none-any.whl (45.4 kB) View hashes Wheel py2

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page