Skip to main content

bigschema provide primitives for writing more maintainable/readable schemas for bigquery in yamland getting bigquery target output from the specification.

Project description

### BigSchema

This library is meant as a simple way to generate the big query boilerplate
given a simple yaml based schema.


#### CLI Tool

The library comes with a simple command line tool that takes a source file,
an optional destination directory default is ```./schema/``` and an output
format and transforms the given yaml specifications to BigQuery compliant
schema in the desired output format.

```bash
$ bigschema --help

Usage: bigschema [OPTIONS] SOURCE [DESTINATION]
Options:
-f, --output-format [java|json]
Which format to output the data in? Default
is JSON
-o, --overwrite BOOLEAN Should we overwrite the data in existing
file? Default is FALSE
--help Show this message and exit.

```


#### File Specification

```yaml
spec-mode: "compact" | "verbose"

---------------------------------------------------------------------------------
spec-mode: "verbose"
--------------------

table:
type: "table"
fields:
- name:
type: "integer" | "float" | "string" | "timestamp" | "boolean" | "bytes" | "record_name"
mode: "nullable" | "repeated" | "required"

record:
type: "record"
fields:
- name:
type: "integer" | "float" | "string" | "timestamp" | "boolean" | "bytes" | "record_name"
mode: "nullable" | "repeated" | "required"


----------------------------------------------------------------------------------
spec-mode: "compact"
--------------------

table:
type: table
fields:
- name_of_column, type_of_column, mode_of_column

record:
type: record
fields:
- name_of_column, type_of_column, mode_of_column
```

Project details


Release history Release notifications

This version
History Node

1.0.3

History Node

1.0.2

History Node

1.0.1

History Node

1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
bigschema-1.0.3.tar.gz (6.6 kB) Copy SHA256 hash SHA256 Source None Apr 26, 2016

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page