py-simple-flow 2020.8.23.3

Simple data processing (ETL) library with support for multi-processing

Python Simple Flow

• Simple framework with a defined structure for data processing with support for multi-process
• Data processing flow is split into four phases:
  • Source: Fetch the data from a source
  • Ingress: Register the data received, log or in DB for trace-ability
  • Transform: Business logic of transforming the data to your business needs
  • Egress: Push the data to downstream or your storage
• Processing can be done in one process, like running it in-process or in multi processing mode
• Mode to be chosen depends on the task you are doing
• Easy setting up logger even in multi processing mode