bigschema provide primitives for writing more maintainable/readable schemas for bigquery in yamland getting bigquery target output from the specification.
Project description
### BigSchema
This library is meant as a simple way to generate the big query boilerplate
given a simple yaml based schema.
#### CLI Tool
The library comes with a simple command line tool that takes a source file,
an optional destination directory default is ```./schema/``` and an output
format and transforms the given yaml specifications to BigQuery compliant
schema in the desired output format.
```bash
$ bigschema --help
Usage: bigschema [OPTIONS] SOURCE [DESTINATION]
Options:
-f, --output-format [java|json]
Which format to output the data in? Default
is JSON
-o, --overwrite BOOLEAN Should we overwrite the data in existing
file? Default is FALSE
--help Show this message and exit.
```
#### File Specification
```yaml
spec-mode: "compact" | "verbose"
---------------------------------------------------------------------------------
spec-mode: "verbose"
--------------------
table:
type: "table"
fields:
- name:
type: "integer" | "float" | "string" | "timestamp" | "boolean" | "bytes" | "record_name"
mode: "nullable" | "repeated" | "required"
record:
type: "record"
fields:
- name:
type: "integer" | "float" | "string" | "timestamp" | "boolean" | "bytes" | "record_name"
mode: "nullable" | "repeated" | "required"
----------------------------------------------------------------------------------
spec-mode: "compact"
--------------------
table:
type: table
fields:
- name_of_column, type_of_column, mode_of_column
record:
type: record
fields:
- name_of_column, type_of_column, mode_of_column
```
This library is meant as a simple way to generate the big query boilerplate
given a simple yaml based schema.
#### CLI Tool
The library comes with a simple command line tool that takes a source file,
an optional destination directory default is ```./schema/``` and an output
format and transforms the given yaml specifications to BigQuery compliant
schema in the desired output format.
```bash
$ bigschema --help
Usage: bigschema [OPTIONS] SOURCE [DESTINATION]
Options:
-f, --output-format [java|json]
Which format to output the data in? Default
is JSON
-o, --overwrite BOOLEAN Should we overwrite the data in existing
file? Default is FALSE
--help Show this message and exit.
```
#### File Specification
```yaml
spec-mode: "compact" | "verbose"
---------------------------------------------------------------------------------
spec-mode: "verbose"
--------------------
table:
type: "table"
fields:
- name:
type: "integer" | "float" | "string" | "timestamp" | "boolean" | "bytes" | "record_name"
mode: "nullable" | "repeated" | "required"
record:
type: "record"
fields:
- name:
type: "integer" | "float" | "string" | "timestamp" | "boolean" | "bytes" | "record_name"
mode: "nullable" | "repeated" | "required"
----------------------------------------------------------------------------------
spec-mode: "compact"
--------------------
table:
type: table
fields:
- name_of_column, type_of_column, mode_of_column
record:
type: record
fields:
- name_of_column, type_of_column, mode_of_column
```
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
bigschema-1.0.1.tar.gz
(6.3 kB
view hashes)
Built Distribution
bigschema-1.0.1-py3-none-any.whl
(11.0 kB
view hashes)
Close
Hashes for bigschema-1.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 14d2bdd2e80ecae48842db6bd51a49371b79d978086ce79103db0faa6533fe76 |
|
MD5 | 6c166479dc684b52cd77177377289265 |
|
BLAKE2b-256 | 0e668ea91ac28d95b7ae341488f4719d59a77094ef96d8317e6e72fc664974e7 |