Library to convert Google BigQuery Table Schema into Json Schema
Project description
Google BigQuery Table Schema Converter
Python library to convert Google BigQuery table schema into draft-07 json schema and vice versa.
The library includes two main modules:
gbqschema_converter
├── gbqschema_to_jsonschema.py
└── jsonschema_to_gbqschema.py
Each of those modules has two main functions:
json_representation
: corresponds to json output (input forgbqschema_to_jsonschema
).sdk_representation
: corresponds to Google Python SDK format output (input forgbqschema_to_jsonschema
).
Installation
python3 -m venv env && source ${PWD}/env/bin/activate
(env) pip install --no-cache-dir gbqschema_converter
Usage: CLI
Convert json-schema to GBQ table schema
(env) json2gbq -h
usage: json2gbq [-h] (-i INPUT | -f FILE)
Google BigQuery Table Schema Converter
optional arguments:
-h, --help show this help message and exit
-i INPUT, --input INPUT
Input object as string.
-f FILE, --file FILE Input object as file path.
Example: stdin
Execution:
(env) json2gbq -i '{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "array",
"items": {
"$ref": "#/definitions/element"
},
"definitions": {
"element": {
"type": "object",
"properties": {
"att_01": {
"type": "integer",
"description": "Att 1"
},
"att_02": {
"type": "number",
"description": "Att 2"
},
"att_03": {
"type": "string"
},
"att_04": {
"type": "boolean"
},
"att_05": {
"type": "string",
"format": "date"
},
"att_06": {
"type": "string",
"format": "date-time"
},
"att_07": {
"type": "string",
"format": "time"
}
},
"required": [
"att_02"
]
}
}
}'
Output:
2020-04-08 21:42:51.700 [INFO ] [Google BigQuery Table Schema Converter] Output (5.52 ms elapsed):
[
{
"description": "Att 1",
"name": "att_01",
"type": "INTEGER",
"mode": "NULLABLE"
},
{
"description": "Att 2",
"name": "att_02",
"type": "NUMERIC",
"mode": "REQUIRED"
},
{
"name": "att_03",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "att_04",
"type": "BOOLEAN",
"mode": "NULLABLE"
},
{
"name": "att_05",
"type": "DATE",
"mode": "NULLABLE"
},
{
"name": "att_06",
"type": "TIMESTAMP",
"mode": "NULLABLE"
},
{
"name": "att_07",
"type": "STRING",
"mode": "NULLABLE"
}
]
Example: file
Execution:
(env) json2gbq -f ${PWD}/data/jsonschema.json
Output:
2020-04-08 21:57:25.516 [INFO ] [Google BigQuery Table Schema Converter] Output (6.39 ms elapsed):
[
{
"description": "Att 1",
"name": "att_01",
"type": "INTEGER",
"mode": "NULLABLE"
},
{
"description": "Att 2",
"name": "att_02",
"type": "NUMERIC",
"mode": "REQUIRED"
},
{
"name": "att_03",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "att_04",
"type": "BOOLEAN",
"mode": "NULLABLE"
},
{
"name": "att_05",
"type": "DATE",
"mode": "NULLABLE"
},
{
"name": "att_06",
"type": "TIMESTAMP",
"mode": "NULLABLE"
},
{
"name": "att_07",
"type": "STRING",
"mode": "NULLABLE"
}
]
Convert GBQ table schema to json-schema
(env) gbq2json -h
usage: gbq2json [-h] (-i INPUT | -f FILE)
Google BigQuery Table Schema Converter
optional arguments:
-h, --help show this help message and exit
-i INPUT, --input INPUT
Input object as string.
-f FILE, --file FILE Input object as file path.
Example: stdin
Execution:
(env) gbq2json -i '[
{
"description": "Att 1",
"name": "att_01",
"type": "INTEGER",
"mode": "NULLABLE"
},
{
"description": "Att 2",
"name": "att_02",
"type": "NUMERIC",
"mode": "REQUIRED"
},
{
"name": "att_03",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "att_04",
"type": "BOOLEAN",
"mode": "NULLABLE"
},
{
"name": "att_05",
"type": "DATE",
"mode": "NULLABLE"
},
{
"name": "att_06",
"type": "DATETIME",
"mode": "NULLABLE"
},
{
"name": "att_07",
"type": "TIMESTAMP",
"mode": "NULLABLE"
}
]'
Output:
2020-04-08 21:51:05.370 [INFO ] [Google BigQuery Table Schema Converter] Output (1.08 ms elapsed):
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "array",
"items": {
"$ref": "#/definitions/element"
},
"definitions": {
"element": {
"type": "object",
"properties": {
"att_01": {
"type": "integer",
"description": "Att 1"
},
"att_02": {
"type": "number",
"description": "Att 2"
},
"att_03": {
"type": "string"
},
"att_04": {
"type": "boolean"
},
"att_05": {
"type": "string",
"format": "date"
},
"att_06": {
"type": "string",
"pattern": "^[0-9]{4}-((|0)[1-9]|1[0-2])-((|[0-2])[1-9]|3[0-1])(|T)((|[0-1])[0-9]|2[0-3]):((|[0-5])[0-9]):((|[0-5])[0-9])(|.[0-9]{1,6})$"
},
"att_07": {
"type": "string",
"format": "date-time"
}
},
"additionalProperties": false,
"required": [
"att_02"
]
}
}
}
Example: file
Execution:
(env) gbq2json -f ${PWD}/data/gbqschema.json
Output:
2020-04-08 21:55:20.275 [INFO ] [Google BigQuery Table Schema Converter] Output (1.72 ms elapsed):
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "array",
"items": {
"$ref": "#/definitions/element"
},
"definitions": {
"element": {
"type": "object",
"properties": {
"att_01": {
"type": "integer",
"description": "Att 1"
},
"att_02": {
"type": "number",
"description": "Att 2"
},
"att_03": {
"type": "string"
},
"att_04": {
"type": "boolean"
},
"att_05": {
"type": "string",
"format": "date"
},
"att_06": {
"type": "string",
"pattern": "^[0-9]{4}-((|0)[1-9]|1[0-2])-((|[0-2])[1-9]|3[0-1])(|T)((|[0-1])[0-9]|2[0-3]):((|[0-5])[0-9]):((|[0-5])[0-9])(|.[0-9]{1,6})$"
},
"att_07": {
"type": "string",
"format": "date-time"
}
},
"additionalProperties": false,
"required": [
"att_02"
]
}
}
}
Usage: python program
Convert json-schema to GBQ table schema
Example: output as json
from gbqschema_converter.jsonschema_to_gbqschema import json_representation as converter
schema_in = {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "array",
"items": {
"$ref": "#/definitions/element",
},
"definitions": {
"element": {
"type": "object",
"properties": {
"att_01": {
"type": "integer",
"description": "Att 1"
},
"att_02": {
"type": "number",
},
}
"required": [
"att_02",
],
},
},
}
schema_out = converter(schema_in)
print(schema_out)
Output:
[{'description': 'Att 1', 'name': 'att_01', 'type': 'INTEGER', 'mode': 'NULLABLE'}, {'name': 'att_02', 'type': 'NUMERIC', 'mode': 'REQUIRED'}]
Example: output as list of SchemaField (SDK format)
from gbqschema_converter.jsonschema_to_gbqschema import sdk_representation as converter
schema_in = {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "array",
"items": {
"$ref": "#/definitions/element",
},
"definitions": {
"element": {
"type": "object",
"properties": {
"att_01": {
"type": "integer",
"description": "Att 1"
},
"att_02": {
"type": "number",
},
},
"required": [
"att_02",
],
},
},
}
schema_out = converter(schema_in)
print(schema_out)
Output:
[SchemaField('att_01', 'INTEGER', 'NULLABLE', 'Att 1', ()), SchemaField('att_02', 'NUMERIC', 'REQUIRED', None, ())]
Convert GBQ table schema to json-schema
Example: output as json
from gbqschema_converter.gbqschema_to_jsonschema import json_representation as converter
schema_in = [
{
'description': 'Att 1',
'name': 'att_01',
'type': 'INTEGER',
'mode': 'NULLABLE'
},
{
'name': 'att_02',
'type': 'NUMERIC',
'mode': 'REQUIRED'
}
]
schema_out = converter(schema_in)
print(schema_out)
Output:
{'$schema': 'http://json-schema.org/draft-07/schema#', 'type': 'array', 'items': {'$ref': '#/definitions/element'}, 'definitions': {'element': {'type': 'object', 'properties': {'att_01': {
'type': 'integer', 'description': 'Att 1'}, 'att_02': {'type': 'number'}}, 'additionalProperties': False, 'required': ['att_02']}}}
Example: output as list of SchemaField (SDK format)
from gbqschema_converter.gbqschema_to_jsonschema import sdk_representation as converter
from google.cloud.bigquery import SchemaField
schema_in = [
SchemaField('att_01', 'INTEGER', 'NULLABLE', 'Att 1', ()),
SchemaField('att_02', 'NUMERIC', 'REQUIRED', None, ()),
]
schema_out = converter(schema_in)
print(schema_out)
Output:
{'$schema': 'http://json-schema.org/draft-07/schema#', 'type': 'array', 'items': {'$ref': '#/definitions/element'}, 'definitions': {'element': {'type': 'object', 'properties': {'att_01': {
'type': 'integer', 'description': 'Att 1'}, 'att_02': {'type': 'number'}}, 'additionalProperties': False, 'required': ['att_02']}}}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for gbqschema_converter-1.2.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | ecabff4ef5ec53d8906470e75ad71e06f2702cdd26b88cd4e322e10f6b3bba6d |
|
MD5 | 80dde9a124af07099f03fad8ac91b1c0 |
|
BLAKE2b-256 | 66bfff87d048d082ede86c3ba7d554d9fa7510dda7a00c552b2f083a75f975ca |
Close
Hashes for gbqschema_converter-1.2.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bcbda794a8a9e39e9f69de220e326a0abe076b20ad15309bdf6b4581b68d12e5 |
|
MD5 | a01fcacc3314521861b516271d3af504 |
|
BLAKE2b-256 | 0115f3f184fa59f93a34d8a356b384bf1083e32296c49cff50fb0851f650e853 |