target-core 0.1.0

Singer.io Target Core features

Target Core

Singer target-core provide safe tools to easily build new targets following the Singer spec convention and protocol.

How to use it

target-core is a Singer Target which intend to work with regular Singer Tap.

The Goal is to use this package as a foundation to build other taps focusing on the core tools, reducing the energy spent on maintaining the common parts.

Packages extending the target-core

• target-s3-jsonl

Install

First, make sure Python 3 is installed on your system or follow these installation instructions for Mac or Ubuntu.

It's recommended to use a venv:

Defaults

python -m venv venv
. venv/bin/activate
pip install --upgrade target-core

Head

python -m venv venv
. venv/bin/activate
pip install --upgrade [e-mail]:omegax/target-core.git

Isolated virtual environment

python -m venv ~/.virtualenvs/target-core
~/.virtualenvs/target-core/bin/pip install target-core

Alternative

python -m venv ~/.virtualenvs/target-core
source ~/.virtualenvs/target-core/bin/activate
pip install target-core
deactivate

To run

Like any other target that's following the singer specificiation:

some-singer-tap | target-core --config [config.json]

It's reading incoming messages from STDIN and using the properites in config.json to upload the data.

Note: To avoid version conflicts run tap and targets in separate virtual environments.

Configuration settings

Running the the target connector requires a config.json file. An example with the minimal settings:

{
    "path_template": "{stream}_{date_time:%Y%m%d_%H%M%S}_part_{part:0>3}.json"
}

Profile based authentication

Profile based authentication used by default using the default profile. To use another profile set aws_profile parameter in config.json or set the AWS_PROFILE environment variable.

Non-Profile based authentication

For non-profile based authentication set aws_access_key_id , aws_secret_access_key and optionally the aws_session_token parameter in the config.json. Alternatively you can define them out of config.json by setting AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables.

Full list of options in config.json:

Property	Type	Mandatory?	Description
path_template	String		(Default: None) Custom naming convention of the s3 key. Replaces tokens stream, and date_time with the appropriate values. Supports datetime and other python advanced string formatting e.g. {stream}_{date_time:%FT%T.%f}.jsonl or {stream:_>8}/{date_time:%Y}/{date_time:%m}/{date_time:%d}/{date_time:%Y%m%d_%H%M%S_%f}.json. Supports "folders" in s3 keys e.g. my_folder/my_sub_folder/{stream}/export_date={date}/{date_time}.json.
timezone_offset	Integer		Offset value in hour. Use offset 0 hours is you want the path_template to use utc time zone. The null values is used by default.
memory_buffer	Integer		Memory buffer's size used before storing the data into the temporary file. 64Mb used by default if unspecified. This value is used to partition the files by size.
file_size	Integer		File size limit. File parts will be created. The path_template must contain a part section for the part number. Example "path_template": "{stream}_{date_time:%Y%m%d_%H%M%S}_part_{part:0>3}.json".
work_dir	String		(Default: platform-dependent) Directory of temporary JSONL files with RECORD messages.
compression	String		The type of compression to apply before uploading. Supported options are none (default), gzip, and lzma. For gzipped files, the file extension will automatically be changed to .json.gz for all files. For lzma compression, the file extension will automatically be changed to .json.xz for all files.

Test

Install the tools

pip install .[test,lint]

Run pytest

pytest -p no:cacheprovider

Lint

flake8 --show-source --statistics --count --extend-exclude .virtualenvs

Release

1. Update the version number at the beginning of target-core/target_core/__init__.py
2. Merge the changes PR into main
3. Create a tag git tag -a 1.0.0 -m 'Release version 1.0.0'
4. Release the new version in github

License

Apache License Version 2.0