Sunflow is a workflow management tool that provides fast organization of algorithm applications, primarily using pluggy based hook technology.
Project description
SunBasket-算法信息存储管理工具
介绍
- SunFlow是一个生产级的轻量工作流管理工具,主要功能提供算法应用快捷组织功能,主要技术使用基于pluggy的hook技术。
安装
SunFlow采用Python开发,得益于Python良好的社区环境,安装支持Pythonic风格的各种管理器。
$ pip install sunflow-0.1.1-xxxxxxxxxxxx.whl
快速指南
- SunFlow提供python-sdk的使用方式,以下是SunFlow主程脚本示例:
from sunflow.node import Node
from sunflow.scheduler import Pipeline
from sunflow.io import LocalDataCatalog
from sunflow.hook import HookManager
from sunflow.runner import SequentialRunner
### 开始sunflow测试
### 设置函数节点对象
def first_func(a,b):
c = a + b
return c
def second_func(c,d):
e = c * d
return e
def third_func(e):
f = e + 1
return f
first_node = Node(func=first_func,inputs=['a','b'],outputs='c',name='firstfunc')
second_node = Node(func=second_func,inputs=['c','d'],outputs='e',name='secondfunc')
third_node = Node(func=third_func,inputs=['e'],outputs=['f'],name='thirdfunc')
### 创建一个pipeline
test_pipeline = Pipeline(nodes=[first_node,second_node,third_node])
### 使用本地catalog加载初始参数
test_cache_data = LocalDataCatalog()
test_cache_data.save(data_name='a',data_obj=5)
test_cache_data.save(data_name='b',data_obj=6)
test_cache_data.save(data_name='d',data_obj=7)
### hook_manager加载已挂载的前后处理功能函数
hook_manager = HookManager()
### 使用序列执行器(scheduler使用流程管道,catalog使用本地数据目录,hook_manager载入默认插件集合对象)
sequential_runner = SequentialRunner(scheduler=test_pipeline,catalog=test_cache_data,hook_manager=hook_manager)
# print(sequential_runner.pipeline,sequential_runner.catalog,sequential_runner.hook_manager)
print('--------------------------------------------------------------------------------------------------------------')
### 选择不清除缓存运行执行器
sequential_runner.execute(is_release=False)
print(test_cache_data.load(data_name='f'))
print(test_cache_data.cache_data)
### 测试过程(5+6)*7+1=78
设计
- 基于pluggy的hook技术实现灵活扩展性
- 设计关键概念,Node,Scheduler,Runner
- 函数节点化-Node
- 开放挂载点-HookManager
- 扩展运行方式(顺序运行和并行运行)-Runner
技术列表
- property动态属性
- 基于pluggy的hook
- __call__
- __add__
- cache_dict
设计UML图
以下是设计的UML图:
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
shihua-sunflow-0.1.1.tar.gz
(10.2 kB
view hashes)
Built Distribution
Close
Hashes for shihua_sunflow-0.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 048f5f01d7454e214533856661e0d32a86d2feffd09d83db0939d45d59907c17 |
|
MD5 | 773ac90d843ddcb6fab9ba9ffa362d38 |
|
BLAKE2b-256 | c1429717a044d43026cbe4b20931ef7140cd6aeb16c4d1031204dd491514d136 |