Skip to main content

Singer.io simple field transformator between taps and targets - PipelineWise compatible

Project description

pipelinewise-transform-field

PyPI version PyPI - Python Version License: Apache2

Transformation component between Singer taps and targets.

This is a PipelineWise compatible component.

How to use it

The recommended method of running this component is to use it from PipelineWise. When running it from PipelineWise you don't need to configure this tap with JSON files and most of things are automated. Please check the related documentation at Transformations

If you want to run this Singer compatible component independently please read further.

Install

First, make sure Python 3 is installed on your system or follow these installation instructions for Mac or Ubuntu.

It's recommended to use a virtualenv:

  python3 -m venv venv
  pip install pipelinewise-transform-field

or

  python3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip
  pip install .

To run

Put it between a tap and a target with simple unix pipes:

some-singer-tap | transform-field --config [config.json] | some-singer-target

It's reading incoming messages from STDIN and using config.json to transform incoming RECORD messages.

Note: To avoid version conflicts run tap, transform and targets in separate virtual environments.

Configuration

You need to defines which columns have to be transformed by which method and in which condition the transformation needs to be applied.

Example config.json:

{
  "transformations": [
      {
          "field_id": "password_hash",
          "tap_stream_name": "stream-id-sent-by-the-tap",
          "type": "SET-NULL"
      },
      {
          "field_id": "salt",
          "tap_stream_name": "stream-id-sent-by-the-tap",
          "type": "SET-NULL"
      },
      {
          "field_id": "value",
          "tap_stream_name": "stream-id-sent-by-the-tap",
          "type": "SET-NULL",
          "when": [
              {"column": "string_column_1", "equals": "Property" },
              {"column": "numeric_column", "equals": 200 },
              {"column": "string_column_2", "regex_match": "sensitive.*PII" }
            ]
      }

  ]
}

(Tip: PipelineWise generating this for you from a more readable YAML format)

Transformation types

  • SET-NULL: Transforms any input to NULL
  • HASH: Transfroms string input to hash
  • HASH-SKIP-FIRST-n: Transforms string input to hash skipping first n characters, e.g. HASH-SKIP-FIRST-2
  • MASK-DATA: Transforms any date to stg
  • MASK-NUMBER: Transforms any number to zero
  • MASK-HIDDEN: Transforms any value to 'hidden'

To run tests:

  1. Install python dependencies in a virtual env and run nose unit and integration tests
  python3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip
  pip install .
  pip install nose
  1. To run tests:
  nosetests --where=tests

License

Apache License Version 2.0

See LICENSE to see the full text.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pipelinewise-transform-field, version 2.0.0
Filename, size File type Python version Upload date Hashes
Filename, size pipelinewise_transform_field-2.0.0-py3-none-any.whl (11.5 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size pipelinewise-transform-field-2.0.0.tar.gz (6.9 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page