Skip to main content

A Big Query json utility package

Project description

This is bqtools a handy set of utils to help create big query tables from json objects and make them into tables (so get the schema) and to create json new line files. Plus a set of common functions to ease writing biq query python code

Goals

  • Simplify handling and move between big query stryuctured data and json data
  • Allow you to create big query table schemas from json structures (as resource or schema objects) uses reflection of types
  • Provides easy generation of views for day partitioned tabled
    • head - latest data
    • diff views for time snap shots of data i.e. each day partition has current view of data
  • Calculate valid json structures from representative json data that can be used as basis of big query schemas
  • Clean json data such that it can be loaded into big query
    • Replace bare lists with dictionaries
    • Replace field names with valid values that can be column names in big query (removes spaces, characters not allowed in field names using same algorithms big query uses when auto detecting schemas)
    • Encodes json output of dates, datetimes, times, timedeltas encoded in format acceptable for big query corresponding field types
  • Generate code for bq command line tool from bq table structures
  • Simplify common tasks of handling big query data
    • Basic tests on dataset or tables existing
    • Schema patching compare an existing schema to a template json object calculate if changed and generate a merged schema that can be used in a patch
    • Flattening views to avoid view depth limits
import bqtools

# if you load a json object say something like
foo = {
        "id":1,
        "description":""
        "aboolean":False
      }

# generate a schema
table = {
   "type":"TABLE",
   "location":os.environ["location"],
   "tableReference":{
       "projectId": os.environ["projectid"],
       "datasetId": os.environ["dataset"],
       "tableId": key
   },
   "timePartitioning":{
       "type": "DAY",
       "expirationMs": "94608000000"
   },
   "schema": {}
}

# use bqtools to create a schema structure
table["schema"]["fields"] = bqtools.get_bq_schema_from_json_repr(foo)

Demonstrates some of power of tools via bqsync that is installed if you install via pip.

pip install bqtools-json

Or you can find the source for this here

Also provides means to handle views across environments supports SQL views based upon Jinja templates. So you can "configure" views for different environments.

Provides support for calculating view dependencies such views can be applied in parallel tranches.

Provides support for calculating authorised views based on set of views "compiled"

Compilation flattens views to lower view depth but automatically does not flatten authorised views to keep access to data tight as feasible.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

bqtools-json-0.4.97.tar.gz (115.1 kB view details)

Uploaded Source

Built Distribution

bqtools_json-0.4.97-py2-none-any.whl (157.2 kB view details)

Uploaded Python 2

File details

Details for the file bqtools-json-0.4.97.tar.gz.

File metadata

  • Download URL: bqtools-json-0.4.97.tar.gz
  • Upload date:
  • Size: 115.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.4.3 requests-toolbelt/0.9.1 tqdm/4.32.1 CPython/2.7.13

File hashes

Hashes for bqtools-json-0.4.97.tar.gz
Algorithm Hash digest
SHA256 7171c92423543acc7b38041402af9e491a373a8e9e31eb7f37b78ee0349455ac
MD5 178effc3593dd68994d200624201b300
BLAKE2b-256 6685716f6ea189bc6ffa9208d73e47518b5253883e5b57e35dcea310a603043f

See more details on using hashes here.

File details

Details for the file bqtools_json-0.4.97-py2-none-any.whl.

File metadata

  • Download URL: bqtools_json-0.4.97-py2-none-any.whl
  • Upload date:
  • Size: 157.2 kB
  • Tags: Python 2
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.4.3 requests-toolbelt/0.9.1 tqdm/4.32.1 CPython/2.7.13

File hashes

Hashes for bqtools_json-0.4.97-py2-none-any.whl
Algorithm Hash digest
SHA256 48d6546757bbb7ff569d7155bd9e9498daf7a276e2b5b51fb31c4afb50ac4a59
MD5 455e699980ff51d2d8bb3b6a50c69802
BLAKE2b-256 b46a8ee9a62521c4eddb97a6f43fd0036316b274704f0d6541ae4258a072ae4c

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page