pipelinewise-target-postgres

Singer target that loads data into Snowflake following the Singer spec.

This is a PipelineWise compatible target connector.

How to use it

The recommended method of running this target is to use it from PipelineWise. When running it from PipelineWise you don't need to configure this tap with JSON files and most of things are automated. Please check the related documentation at Target Postgres

If you want to run this Singer Target independently please read further.

Install

First, make sure Python 3 is installed on your system or follow these installation instructions for Mac or Ubuntu.

It's recommended to use a virtualenv:

  python3 -m venv venv
  pip install pipelinewise-target-postgres

or

  python3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip
  pip install .

To run

Like any other target that's following the singer specificiation:

some-singer-tap | target-postgres --config [config.json]

It's reading incoming messages from STDIN and using the properites in config.json to upload data into Postgres.

Note: To avoid version conflicts run tap and targets in separate virtual environments.

Configuration settings

Running the the target connector requires a config.json file. An example with the minimal settings:

{
  "host": "localhost",
  "port": 5432,
  "user": "my_user",
  "password": "secret",
  "dbname": "my_db_name",
  "default_target_schema": "my_target_schema"
}

Full list of options in config.json:

To run tests:

1. Define environment variables that requires running the tests

  export TARGET_POSTGRES_HOST=<postgres-host>
  export TARGET_POSTGRES_PORT=<postgres-port>
  export TARGET_POSTGRES_USER=<postgres-password>
  export TARGET_POSTGRES_PASSWORD=<postgres-password>
  export TARGET_POSTGRES_DBNAME=<postgres-dbname>
  export TARGET_POSTGRES_SCHEMA=<postgres-schema>

2. Install python dependencies in a virtual env and run nose unit and integration tests

  python3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip
  pip install .
  pip install nose

3. To run unit tests:

  nosetests --where=tests/unit

4. To run integration tests:

  nosetests --where=tests/integration

To run pylint:

1. Install python dependencies and run python linter

  python3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip
  pip install .
  pip install pylint
  pylint target_postgres -d C,W,unexpected-keyword-arg,duplicate-code