Singer.io simple field transformator between taps and targets - PipelineWise compatible
Project description
pipelinewise-transform-field
Transformation component between Singer taps and targets.
This is a PipelineWise compatible component.
How to use it
The recommended method of running this component is to use it from PipelineWise. When running it from PipelineWise you don't need to configure this tap with JSON files and most of things are automated. Please check the related documentation at Transformations
If you want to run this Singer compatible component independently please read further.
Install
First, make sure Python 3 is installed on your system or follow these installation instructions for Mac or Ubuntu.
It's recommended to use a virtualenv:
python3 -m venv venv
pip install pipelinewise-transform-field
or
python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip
pip install .
To run
Put it between a tap and a target with simple unix pipes:
some-singer-tap | transform-field --config [config.json] | some-singer-target
It's reading incoming messages from STDIN and using config.json
to transform incoming RECORD messages.
Note: To avoid version conflicts run tap
, transform
and targets
in separate virtual environments.
Configuration
You need to defines which columns have to be transformed by which method and in which condition the transformation needs to be applied.
Example config.json:
{
"transformations": [
{
"field_id": "password_hash",
"tap_stream_name": "stream-id-sent-by-the-tap",
"type": "SET-NULL"
},
{
"field_id": "salt",
"tap_stream_name": "stream-id-sent-by-the-tap",
"type": "SET-NULL"
},
{
"field_id": "value",
"tap_stream_name": "stream-id-sent-by-the-tap",
"type": "SET-NULL",
"when": [
{"column": "string_column_1", "equals": "Property" },
{"column": "numeric_column", "equals": 200 },
{"column": "string_column_2", "regex_match": "sensitive.*PII" }
]
}
]
}
(Tip: PipelineWise generating this for you from a more readable YAML format)
Transformation types
- SET-NULL: Transforms any input to NULL
- HASH: Transfroms string input to hash
- HASH-SKIP-FIRST-n: Transforms string input to hash skipping first n characters, e.g. HASH-SKIP-FIRST-2
- MASK-DATA: Transforms any date to stg
- MASK-NUMBER: Transforms any number to zero
- MASK-HIDDEN: Transforms any value to 'hidden'
To run tests:
- Install python dependencies in a virtual env and run nose unit and integration tests
python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip
pip install .
pip install nose
- To run tests:
nosetests --where=tests
License
Apache License Version 2.0
See LICENSE to see the full text.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pipelinewise-transform-field-1.2.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 78a6e17c0d448c5e60dbef2c3142a68e9853480eacb86c96c21cbb49478f5f4a |
|
MD5 | a9bc3240924c58b47e561c88cd260015 |
|
BLAKE2b-256 | 2429ca26785bd9a543a10fef44b22060165fe904b6b57a6274db82d7380e165a |
Hashes for pipelinewise_transform_field-1.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4cfa966c1a10d8aa6251f78d924c64186be4465531b90cf61aa5d1b5b3cda0e7 |
|
MD5 | a49dd170dee4f789f81f1c009c9943e9 |
|
BLAKE2b-256 | 29e34ea25ec2bb27aca4b19650fd9948db6548f65af0ca3886e59100f14a48a9 |