tmg-etl-pipeline 0.0.9

TMG ETL pipeline

TMG ETL pipeline is the base structure of our ETL pipelines. Any pipeline should inherit from TMGETLPipeline class and implement the necessary methods.

Quick Start

Installation

Install this library in a virtualenv using pip. virtualenv is a tool to create isolated Python environments. The basic problem it addresses is one of dependencies and versions, and indirectly permissions.

Supported Python Versions

Python >= 3.5

Mac/Linux

pip install virtualenv
virtualenv <your-env>
source <your-env>/bin/activate
<your-env>/bin/pip install tmg-etl-pipeline

Example Usage

from tmg_etl_pipeline.etl import TMGETLPipeline


class MyETL(TMGETLPipeline):

    def run(self):
        # your ETL logic goes here
        # access the configs
        # self.config['some-config-variable']
        # access the secrets
        # self.secrets['some-secret-variable']

    def cleanup(self):
        # your clean up code goes here

etl = MyETL(app_name='your-app-name', config_path='path/to-the-config-file')
etl.execute()