bigschema provide primitives for writing more maintainable/readable schemas for bigquery in yamland getting bigquery target output from the specification.
Project description
### BigSchema
This library is meant as a simple way to generate the big query boilerplate
given a simple yaml based schema.
#### CLI Tool
The library comes with a simple command line tool that takes a source file,
an optional destination directory default is ```./schema/``` and an output
format and transforms the given yaml specifications to BigQuery compliant
schema in the desired output format.
```bash
$ bigschema --help
Usage: bigschema [OPTIONS] SOURCE [DESTINATION]
Options:
-f, --output-format [java|json]
Which format to output the data in? Default
is JSON
-o, --overwrite BOOLEAN Should we overwrite the data in existing
file? Default is FALSE
--help Show this message and exit.
```
#### File Specification
```yaml
spec-mode: "compact" | "verbose"
---------------------------------------------------------------------------------
spec-mode: "verbose"
--------------------
table:
type: "table"
fields:
- name:
type: "integer" | "float" | "string" | "timestamp" | "boolean" | "bytes" | "record_name"
mode: "nullable" | "repeated" | "required"
record:
type: "record"
fields:
- name:
type: "integer" | "float" | "string" | "timestamp" | "boolean" | "bytes" | "record_name"
mode: "nullable" | "repeated" | "required"
----------------------------------------------------------------------------------
spec-mode: "compact"
--------------------
table:
type: table
fields:
- name_of_column, type_of_column, mode_of_column
record:
type: record
fields:
- name_of_column, type_of_column, mode_of_column
```
This library is meant as a simple way to generate the big query boilerplate
given a simple yaml based schema.
#### CLI Tool
The library comes with a simple command line tool that takes a source file,
an optional destination directory default is ```./schema/``` and an output
format and transforms the given yaml specifications to BigQuery compliant
schema in the desired output format.
```bash
$ bigschema --help
Usage: bigschema [OPTIONS] SOURCE [DESTINATION]
Options:
-f, --output-format [java|json]
Which format to output the data in? Default
is JSON
-o, --overwrite BOOLEAN Should we overwrite the data in existing
file? Default is FALSE
--help Show this message and exit.
```
#### File Specification
```yaml
spec-mode: "compact" | "verbose"
---------------------------------------------------------------------------------
spec-mode: "verbose"
--------------------
table:
type: "table"
fields:
- name:
type: "integer" | "float" | "string" | "timestamp" | "boolean" | "bytes" | "record_name"
mode: "nullable" | "repeated" | "required"
record:
type: "record"
fields:
- name:
type: "integer" | "float" | "string" | "timestamp" | "boolean" | "bytes" | "record_name"
mode: "nullable" | "repeated" | "required"
----------------------------------------------------------------------------------
spec-mode: "compact"
--------------------
table:
type: table
fields:
- name_of_column, type_of_column, mode_of_column
record:
type: record
fields:
- name_of_column, type_of_column, mode_of_column
```
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
bigschema-1.0.3.tar.gz
(6.6 kB
view details)
File details
Details for the file bigschema-1.0.3.tar.gz
.
File metadata
- Download URL: bigschema-1.0.3.tar.gz
- Upload date:
- Size: 6.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3b364aeb647894c84e01bd8256d48550d302f9d3120bd59ff0473e2cecb711da |
|
MD5 | a859a0e23511a5d1a3aff91361f833c7 |
|
BLAKE2b-256 | fed4b4fbe8d8b8990a805170b7fd175b8f1913e811cacbaec1e508563c0c72e6 |