gokart pipeline
Project description
gokart-pipeliner
gokart pipeline project
Usage
Please show SampleTask.py or Eaxmple.ipynb
from gokart_pipeliner import GokartPipeliner
from ExampleTasks import *
# make pipeline
preprocess = [TaskA, {'task_b': TaskB, 'task_c': TaskC}, TaskD]
modeling = preprocess + [TaskE, {'task_f': TaskF}, TaskF]
predict = [{'model': modeling, 'task_a': TaskA}, TaskG]
# instantiation (setting static params)
params = {'TaskA': {'param1':0.1, 'param2': 'sample'}, 'TaskD': {'param1': 'foo'}}
config_path_list = ['./conf/param.ini']
gp = GokartPipeliner(
params=params,
config_path_list=config_path_list)
# run (setting dynamic params)
running_params = {'TaskB': {'param1':'bar'}}
gp.run(predict, params=running_params)
task example
class Task(gokart.TaskOnKart):
foo = gokart.TaskInstanceParameter()
def run(self):
x = self.load('foo')
self.dump(x)
get task result
We can get result of latest pipeline tasks.
task_b_result = gp.run([TaskA, TaskB], return_value=True)
write requires
If you say "want to write requires" or "want to reuse existing tasks", we can use override_requires
parameter.
params = {'ExistingTask': {'override_requires': False}}
gp.run([ExistingTask], params=params)
for jupyter notebook
off logger
gp.run([Task], params=params, verbose=False)
Develop
pip install poetry
pip install poetry-dynamic-versioning
# poetry install
poetry build
# poetry lock
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
gokart_pipeliner-0.0.7.tar.gz
(4.5 kB
view hashes)
Built Distribution
Close
Hashes for gokart_pipeliner-0.0.7-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1ea5c70b2959b2a86027c2f35cdcaa65279077b3cb6d2e44b60b4273176abad6 |
|
MD5 | 67f78b4790b8abfa862ab5ebfa1611ee |
|
BLAKE2b-256 | 4c1f66fd3321d215c2a65442ddb2cd91754e60159e889a645227c78e671908bf |