Tap Core

Singer tap core provide safe tools to easily build new taps following the Singer spec convention and protocol.

How to use it

tap-core is a Singer Tap which intend to work with regular Singer Tap.

The Goal is to use this package as a foundation to build other taps focusing on the core tools, reducing the energy spent on maintaining the common parts.

Getting started

To make it easy for you to get started with GitLab, here's a list of recommended next steps.

Already a pro? Just edit this README.md and make it your own. Want to make it easy? Use the template at the bottom!

Install

First, make sure Python 3 is installed on your system or follow these installation instructions for Mac or Ubuntu.

It's recommended to use a venv:

Defaults

python -m venv .virtualenvs/tap-core
.virtualenvs/tap-core/bin/pip install tap-core

Alternative

python -m venv .virtualenvs/tap-core
source .virtualenvs/tap-core/bin/activate
pip install tap-core
deactivate

To run

Like any other tap that's following the singer specificiation:

tap-source | target-destination --config [config.json]

Setup the config.json file

{
    "selected": ["x", "y", "z"],
    "access_token": "TOKEN"
}

By providing a selected list of models, they will be enabled for sync by default.

Build the catalog

.virtualenvs/tap-core/bin/tap-core -c config_tap.json --discover > catalog.json

Select the streams to sync

Add or modify the "selected": true, in the metadata.

      "stream": "teams",
      "metadata": [
        {
          "breadcrumb": [],
          "metadata": {
            "selected": true,
            "table-key-properties": [
              "id"
            ],
            "forced-replication-method": "INCREMENTAL",
            "valid-replication-keys": [
              "updated_at"
            ]
          }
        },

echo "{}" > state.json
.virtualenvs/tap-core/bin/tap-core -c config_tap.json --catalog catalog.json -s state.json \
  | .virtualenvs/target-core/bin/target-core -c config_target.json

Example

python -m src.tap -c config_tap.json --discover > catalog.json

echo "{}" > state.json

python -m src.tap -c config_tap.json --catalog catalog.json -s state.json \
  | .virtualenvs/target-core/bin/target-core -c config_target.json > state.json
tail -1 state.json > state.json.tmp && mv state.json.tmp state.json