A simple data workflow runner that helps you write better ETL scripts using reusable code pieces.
Project description
datarunner
A simple data workflow runner that helps you write better ETL scripts using reusable code pieces.
Quick Start Tutorial
Install using pip:
pip install datarunner
Then write a few steps (functions, classes, etc) that can be called, pass to datarunner.Workflow, and call run():
from datarunner import Workflow, Step
def setup():
print('Ready to go!')
def extract():
return 'data'
def transform(data):
return data + ' using reusable code pieces, like Lego.'
class Load(Step):
""" Sub-class Step to customize the callable """
def __init__(self, to_dataset):
super().__init__()
self.to_dataset = to_dataset
def __str__(self):
return f'Load("{self.to_dataset}")'
def run(self, data):
print(f'Loading {data}')
Workflow(setup,
etl=[extract, transform, Load('example')]).run()
It should produce the following output:
setup Ready to go! etl -------------------------------------------------------------------------------- extract >> transform >> Load("example") Loading data using reusable code pieces, like Lego.
If we skip setup, then we can also use >> operator to convey the same flow:
flow = Workflow() >> extract >> transform >> Load('example')
flow.run()
Links & Contact Info
PyPI Package: https://pypi.python.org/pypi/datarunner
GitHub Source: https://github.com/maxzheng/datarunner
Report Issues/Bugs: https://github.com/maxzheng/datarunner/issues
Creator: https://www.linkedin.com/in/maxzheng
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
datarunner-1.1.2.tar.gz
(7.4 kB
view hashes)
Built Distribution
Close
Hashes for datarunner-1.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d48d6d656baee0c8860785ca083d31caa4b3c341958a19a27e84dc3c87d34d04 |
|
MD5 | af1597e0d5f528e3f73891bf1406e54f |
|
BLAKE2b-256 | 2c015b928f1db6ab90009a77219a9e734e4269ae88f5647a542b3ee4f139769f |