Skip to main content

bigschema provide primitives for writing more maintainable/readable schemas for bigquery in yamland getting bigquery target output from the specification.

Project description

### BigSchema

This library is meant as a simple way to generate the big query boilerplate
given a simple yaml based schema.


#### CLI Tool

The library comes with a simple command line tool that takes a source file,
an optional destination directory default is ```./schema/``` and an output
format and transforms the given yaml specifications to BigQuery compliant
schema in the desired output format.

```bash
$ bigschema --help

Usage: bigschema [OPTIONS] SOURCE [DESTINATION]
Options:
-f, --output-format [java|json]
Which format to output the data in? Default
is JSON
-o, --overwrite BOOLEAN Should we overwrite the data in existing
file? Default is FALSE
--help Show this message and exit.

```


#### File Specification

```yaml
spec-mode: "compact" | "verbose"

---------------------------------------------------------------------------------
spec-mode: "verbose"
--------------------

table:
type: "table"
fields:
- name:
type: "integer" | "float" | "string" | "timestamp" | "boolean" | "bytes" | "record_name"
mode: "nullable" | "repeated" | "required"

record:
type: "record"
fields:
- name:
type: "integer" | "float" | "string" | "timestamp" | "boolean" | "bytes" | "record_name"
mode: "nullable" | "repeated" | "required"


----------------------------------------------------------------------------------
spec-mode: "compact"
--------------------

table:
type: table
fields:
- name_of_column, type_of_column, mode_of_column

record:
type: record
fields:
- name_of_column, type_of_column, mode_of_column
```

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

bigschema-1.0.3.tar.gz (6.6 kB view details)

Uploaded Source

File details

Details for the file bigschema-1.0.3.tar.gz.

File metadata

  • Download URL: bigschema-1.0.3.tar.gz
  • Upload date:
  • Size: 6.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for bigschema-1.0.3.tar.gz
Algorithm Hash digest
SHA256 3b364aeb647894c84e01bd8256d48550d302f9d3120bd59ff0473e2cecb711da
MD5 a859a0e23511a5d1a3aff91361f833c7
BLAKE2b-256 fed4b4fbe8d8b8990a805170b7fd175b8f1913e811cacbaec1e508563c0c72e6

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page