Skip to main content

Singer.io Target Core features

Project description

Target Core

GitLab - License Python package builder Coverage PyPI - Python Version PyPI version PyPi project installs

Singer target-core provide safe tools to easily build new targets following the Singer spec convention and protocol.

How to use it

target-core is a Singer Target which intend to work with regular Singer Tap.

The Goal is to use this package as a foundation to build other taps focusing on the core tools, reducing the energy spent on maintaining the common parts.

Packages extending the target-core

Install

First, make sure Python 3 is installed on your system or follow these installation instructions for Mac or Ubuntu.

It's recommended to use a venv:

Defaults

python -m venv venv
. venv/bin/activate
pip install --upgrade target-core

Head

python -m venv venv
. venv/bin/activate
pip install --upgrade https+git@gitlab.com:omegax/target-core.git

Isolated virtual environment

python -m venv ~/.virtualenvs/target-core
~/.virtualenvs/target-core/bin/pip install target-core

Alternative

python -m venv ~/.virtualenvs/target-core
source ~/.virtualenvs/target-core/bin/activate
pip install target-core
deactivate

To run

Like any other target that's following the singer specificiation:

some-singer-tap | target-core --config [config.json]

It's reading incoming messages from STDIN and using the properites in config.json to upload the data.

Note: To avoid version conflicts run tap and targets in separate virtual environments.

Configuration settings

Running the the target connector requires a config.json file. An example with the minimal settings:

{
    "path_template": "{stream}_{date_time:%Y%m%d_%H%M%S}_part_{part:0>3}.json"
}

Profile based authentication

Profile based authentication used by default using the default profile. To use another profile set aws_profile parameter in config.json or set the AWS_PROFILE environment variable.

Non-Profile based authentication

For non-profile based authentication set aws_access_key_id , aws_secret_access_key and optionally the aws_session_token parameter in the config.json. Alternatively you can define them out of config.json by setting AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables.

Full list of options in config.json:

Property Type Mandatory? Description
path_template String (Default: None) Custom naming convention of the s3 key. Replaces tokens stream, and date_time with the appropriate values.

Supports datetime and other python advanced string formatting e.g. {stream}_{date_time:%FT%T.%f}.jsonl or {stream:_>8}/{date_time:%Y}/{date_time:%m}/{date_time:%d}/{date_time:%Y%m%d_%H%M%S_%f}.json.

Supports "folders" in s3 keys e.g. my_folder/my_sub_folder/{stream}/export_date={date}/{date_time}.json.
timezone_offset Integer Offset value in hour. Use offset 0 hours is you want the path_template to use utc time zone. The null values is used by default.
memory_buffer Integer Memory buffer's size used before storing the data into the temporary file. 64Mb used by default if unspecified. This value is used to partition the files by size.
file_size Integer File size limit. File parts will be created. The path_template must contain a part section for the part number. Example "path_template": "{stream}_{date_time:%Y%m%d_%H%M%S}_part_{part:0>3}.json".
work_dir String (Default: platform-dependent) Directory of temporary JSONL files with RECORD messages.
compression String The type of compression to apply before uploading. Supported options are none (default), gzip, and lzma. For gzipped files, the file extension will automatically be changed to .json.gz for all files. For lzma compression, the file extension will automatically be changed to .json.xz for all files.

Test

Install the tools

pip install .[test,lint]

Run pytest

pytest -p no:cacheprovider

Lint

flake8 --show-source --statistics --count --extend-exclude .virtualenvs

Release

  1. Update the version number at the beginning of target-core/target_core/__init__.py
  2. Merge the changes PR into main
  3. Create a tag git tag -a 1.0.0 -m 'Release version 1.0.0'
  4. Release the new version in github

License

Apache License Version 2.0

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

target-core-0.1.0.tar.gz (28.3 kB view hashes)

Uploaded Source

Built Distribution

target_core-0.1.0-py3-none-any.whl (23.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page