pipelinewise-transform-field 2.2.0

Singer.io simple field transformer between taps and targets - PipelineWise compatible

pipelinewise-transform-field

Transformation component between Singer taps and targets.

This is a PipelineWise compatible component.

How to use it

The recommended method of running this component is to use it from PipelineWise. When running it from PipelineWise you don't need to configure this tap with JSON files, and most of things are automated. Please check the related documentation at Transformations

If you want to run this Singer compatible component independently please read further.

Install

First, make sure Python 3 is installed on your system or follow these installation instructions for Mac or Ubuntu.

It's recommended to use a virtualenv:

  python3 -m venv venv
  pip install pipelinewise-transform-field

or

  python3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip setuptools
  pip install .

To validate transformations

transform-field --validate --config [config.json] --catalog [catalog.json]

To run

Put it between a tap and a target with simple unix pipes:

some-singer-tap | transform-field --config [config.json] | some-singer-target

It's reading incoming messages from STDIN and using config.json to transform incoming RECORD messages.

Note: To avoid version conflicts run tap, transform and targets in separate virtual environments.

Configuration

You need to define which columns have to be transformed by which method and in which condition the transformation needs to be applied.

Example config.json:

{
  "transformations": [
      {
          "field_id": "password_hash",
          "tap_stream_name": "stream-id-sent-by-the-tap",
          "type": "SET-NULL"
      },
      {
          "field_id": "salt",
          "tap_stream_name": "stream-id-sent-by-the-tap",
          "type": "SET-NULL"
      },
      {
          "field_id": "value",
          "tap_stream_name": "stream-id-sent-by-the-tap",
          "type": "SET-NULL",
          "when": [
              {"column": "string_column_1", "equals": "Property" },
              {"column": "numeric_column", "equals": 200 },
              {"column": "string_column_2", "regex_match": "sensitive.*PII" }
            ]
      }

  ]
}

(Tip: PipelineWise generating this for you from a more readable YAML format)

Transformation types

• SET-NULL: Transforms any input to NULL
• HASH: Transforms string input to hash
• HASH-SKIP-FIRST-n: Transforms string input to hash skipping first n characters, e.g. HASH-SKIP-FIRST-2
• MASK-DATE: Replaces the months and day parts of date columns to be always 1st of Jan
• MASK-NUMBER: Transforms any numeric value to zero
• MASK-HIDDEN: Transforms any string to 'hidden'
• MASK-STRING-SKIP-ENDS-n: Transforms string input to masked version skipping first and last n characters, e.g. MASK-STRING-SKIP-ENDS-3

To check code style:

1. Install python dependencies in a virtual env

  python3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip setuptools
  pip install .[test]

2. Run pylint

pylint transform_field

To run tests:

1. Install python dependencies in a virtual env and run unit and integration tests

  python3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip setuptools
  pip install .[test]

2. Run tests:

• Unit tests

  pytest -v tests/unit

• Integration tests

  pytest -v tests/integration

• All tests

  pytest -v tests

License

Apache License Version 2.0

See LICENSE to see the full text.