Skip to main content

This is a 3TL module for pipeline creation

Project description

### step class ### step class consist common functions which can be inherited by other classes – setTaskID, setUUID,setTaskExecutionID,startRedisConn,setLogger,loadParams,connectToAPIForKey,createAndGetResponseFromURL,getLogger,exceptionTraceback,getRelativeFile

### extract class### extract class consist of init method which sets all loadParams and log files .. also contains super class startup which can be defined from client extract .. it also consist of startWs- starting websocket connection functions .

### ML class ### Ml class consist of startup method ,startMLSubscriber to subscribr to the incoming data ,process ml to process data , compute ml to do client ML computations and handoff it to next step

### Transform class ### Transform class consist of startup ,startTRSubscriber to subscribr to the incoming data , tranformExData -function has to be defined inside clientTranform with own tranform logic .It also consist of microBatchProcessRedisSortedSet for microbatching streaming data ./

### Load Class ### Load class consist of startup method , loadsubscribers to listen to the data and client logic to loading final data needs to be added in client load file License-File: LICENSE.txt

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

libs3tl-0.1.11.tar.gz (6.5 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page