Skip to main content

bigschema provide primitives for writing more maintainable/readable schemas for bigquery in yamland getting bigquery target output from the specification.

Project description

### BigSchema

This library is meant as a simple way to generate the big query boilerplate
given a simple yaml based schema.


#### CLI Tool

The library comes with a simple command line tool that takes a source file,
an optional destination directory default is ```./schema/``` and an output
format and transforms the given yaml specifications to BigQuery compliant
schema in the desired output format.

```bash
$ bigschema --help

Usage: bigschema [OPTIONS] SOURCE [DESTINATION]
Options:
-f, --output-format [java|json]
Which format to output the data in? Default
is JSON
-o, --overwrite BOOLEAN Should we overwrite the data in existing
file? Default is FALSE
--help Show this message and exit.

```


#### File Specification

```yaml
spec-mode: "compact" | "verbose"

---------------------------------------------------------------------------------
spec-mode: "verbose"
--------------------

table:
type: "table"
fields:
- name:
type: "integer" | "float" | "string" | "timestamp" | "boolean" | "bytes" | "record_name"
mode: "nullable" | "repeated" | "required"

record:
type: "record"
fields:
- name:
type: "integer" | "float" | "string" | "timestamp" | "boolean" | "bytes" | "record_name"
mode: "nullable" | "repeated" | "required"


----------------------------------------------------------------------------------
spec-mode: "compact"
--------------------

table:
type: table
fields:
- name_of_column, type_of_column, mode_of_column

record:
type: record
fields:
- name_of_column, type_of_column, mode_of_column
```

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for bigschema, version 1.0.3
Filename, size File type Python version Upload date Hashes
Filename, size bigschema-1.0.3.tar.gz (6.6 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page