Target Core

Singer target-core provide safe tools to easily build new targets following the Singer spec convention and protocol.

How to use it

target-core is a Singer Target which intend to work with regular Singer Tap.

The Goal is to use this package as a foundation to build other taps focusing on the core tools, reducing the energy spent on maintaining the common parts.

Packages extending the target-core

• target-s3-jsonl

Install

First, make sure Python 3 is installed on your system or follow these installation instructions for Mac or Ubuntu.

It's recommended to use a venv:

Defaults

python -m venv venv
. venv/bin/activate
pip install --upgrade target-core

Head

python -m venv venv
. venv/bin/activate
pip install --upgrade https+git@gitlab.com:omegax/target-core.git

Isolated virtual environment

python -m venv ~/.virtualenvs/target-core
~/.virtualenvs/target-core/bin/pip install target-core

Alternative

python -m venv ~/.virtualenvs/target-core
source ~/.virtualenvs/target-core/bin/activate
pip install target-core
deactivate

To run

Like any other target that's following the singer specificiation:

some-singer-tap | target-core --config [config.json]

It's reading incoming messages from STDIN and using the properites in config.json to upload the data.

Note: To avoid version conflicts run tap and targets in separate virtual environments.

Configuration settings

Running the the target connector requires a config.json file. An example with the minimal settings:

{
    "path_template": "{stream}_{date_time:%Y%m%d_%H%M%S}_part_{part:0>3}.json"
}

Profile based authentication

Profile based authentication used by default using the default profile. To use another profile set aws_profile parameter in config.json or set the AWS_PROFILE environment variable.

Non-Profile based authentication

For non-profile based authentication set aws_access_key_id , aws_secret_access_key and optionally the aws_session_token parameter in the config.json. Alternatively you can define them out of config.json by setting AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables.

Full list of options in config.json:

Test

Install the tools

pip install .[test,lint]

Run pytest

pytest -p no:cacheprovider

Lint

flake8 --show-source --statistics --count --extend-exclude .virtualenvs

Release

1. Update the version number at the beginning of target-core/target_core/__init__.py
2. Merge the changes PR into main
3. Create a tag git tag -a 1.0.0 -m 'Release version 1.0.0'
4. Release the new version in github

License

Apache License Version 2.0