Target Snowflake

A Singer Snowflake target, for use with Singer streams generated by Singer taps.

Snowflake Connector

Docs

Install

pip install target-snowflake

Usage

1. Follow the Singer.io Best Practices for setting up separate tap and target virtualenvs to avoid version conflicts.

2. Create a config file at ~/singer.io/target_snowflake_config.json with Snowflake connection information and target Snowflake schema and warehouse.

   {
     "snowflake_account": "https://XXXXX.snowflakecomputing.com",
     "snowflake_username": "myuser",
     "snowflake_role": "myrole",
     "snowflake_password": "1234",
     "snowflake_database": "my_analytics",
     "snowflake_schema": "mytapname",
     "snowflake_warehouse": "dw"
   }
   

1. Run `target-snowfkajke` against a [Singer](https://singer.io) tap.

 ```bash
 ~/.virtualenvs/tap-something/bin/tap-something \
   | ~/.virtualenvs/target-snowflake/bin/target-snowflake \
     --config ~/singer.io/target_snowflake_config.json >> state.json

If you are running windows, the following is equivalent:

venvs\tap-exchangeratesapi\Scripts\tap-exchangeratesapi.exe | ^
venvs\target-snowflake\Scripts\target-snowlfake.exe ^
--config target_snowflake_config.json

Config.json

The fields available to be specified in the config file are specified here.

S3 Config.json

Limitations

• Snowflake SQL Identifiers:
  • Although Snowflake supports quoted identifiers to have non-alphanumeric values, target-snowflake limits identifiers to uppercase alphanumerics, and underscores
  • This is done to make querability/useability in Snowflake simpler, so as to not require users to have to use sometimes cumbersome quotes to query their data
• Requires a JSON Schema for every stream.
• Only string, string with date-time format, integer, number, boolean, object, and array types with or without null are supported. Arrays can have any of the other types listed, including objects as types within items.
  • Example of JSON Schema types that work
    • ['number']
    • ['string']
    • ['string', 'null']
    • ['string', 'integer']
    • ['integer', 'number']
  • Exmaple of JSON Schema types that DO NOT work
    • ['any']
    • ['null']
• JSON Schema combinations such as anyOf and allOf are not supported.
• JSON Schema $ref is partially supported:
  • NOTE: The following limitations are known to NOT fail gracefully
  • Presently you cannot have any circular or recursive $refs
  • $refs must be present within the schema:
    • URI's do not work
    • if the $ref is broken, the behaviour is considered unexpected
• Any values which are the string \\N will be streamed to Snowflake as the literal null

Sponsorship

Target Snowflake is sponsored by Data Mill (Data Mill Services, LLC) datamill.co.

Data Mill helps organizations utilize modern data infrastructure and data science to power analytics, products, and services.

Copyright Data Mill Services, LLC 2018