LitePipeline: distributed pipeline system
Project description
LitePipeline
A distributed pipeline system, based on Python3, tornado, venv-pack, pyinstaller.
All code based on Python3, do not use Python2!
It still under development, so, maybe have some bugs or not stable enough!
See more details at https://github.com/fiefdx/LitePipeline
Install
# install from pip
$ pip3 install -U litepipeline
# or install from source code
$ cd ./litepipeline
$ python3 ./setup.py install
# this will install 4 commands: liteconfig, litemanager, litenode, litepipeline
# liteconfig: to generate manager's or node's configuration file
# litemanager: to start LitePipeline manager
# litenode: to start LitePipeline node
# litepipeline: command line tool to communicate with LitePipeline cluster
run
Run Manager
# create manager's data directory
$ mkdir ./manager_data
# generate manager's configuration file
$ cd ./manager_data
# this will generate a configuration.yml file under ./manager_data
$ liteconfig -s manager -o ./
# run manager
$ litemanager -c ./configuration.yml
# test
$ curl localhost:8000
# return this message
{"message": "LitePipeline manager service"}
Run Node
# create node's data directory
$ mkdir ./node_data
# generate node's configuration file
$ cd ./node_data
# this will generate a configuration.yml file under ./node_data
$ liteconfig -s node -o ./
# run node
# after start node, node will register to manager, and get a unique node id
$ litenode -c ./configuration.yml
# test
$ curl localhost:8001
# return this message
{"message": "LitePipeline node service"}
Communicate with LitePipeline cluster
$ litepipeline localhost:8000 cluster info
# | node_id | http_host | http_port | action_slots | app_path | data_path
1 | 32313239-e7ee-4f90-8c05-4e08fb48be70 | 127.0.0.1 | 8001 | 2 | /usr/local/lib/python3.7/dist-packages/litepipeline/node | /home/pi/Develop/litenode/data
# use -h/--help parameter to see help message
$ litepipeline --help
usage: litepipeline [-h] [-r] [-v] address {app,task,cluster} ...
positional arguments:
address manager address, host:port
{app,task,cluster} sub-command help
app operate with app API
task operate with task API
cluster operate with cluster API
optional arguments:
-h, --help show this help message and exit
-r, --raw display raw json data
-v, --version show program's version number and exit
$ litepipeline localhost:8000 app --help
usage: litepipeline address app [-h]
{create,delete,update,list,info,download} ...
positional arguments:
{create,delete,update,list,info,download}
sub-command app help
create create application
delete delete application
update update application
list list applications
info application's info
download download application
optional arguments:
-h, --help show this help message and exit
$ litepipeline localhost:8000 task --help
usage: litepipeline address task [-h] {create,delete,list,info,stop} ...
positional arguments:
{create,delete,list,info,stop}
sub-command task help
create create task
delete delete task
list list tasks
info task's info
stop stop task
optional arguments:
-h, --help show this help message and exit
$ litepipeline localhost:8000 cluster --help
usage: litepipeline address cluster [-h] {info} ...
positional arguments:
{info} sub-command cluster help
info cluster's info
optional arguments:
-h, --help show this help message and exit
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
litepipeline-0.0.13.tar.gz
(31.0 kB
view hashes)
Built Distribution
Close
Hashes for litepipeline-0.0.13-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 112cac2a08cc23da7d7b9cf5d5ec33d72e0b017405232bcdc783f1b9ecb9d933 |
|
MD5 | 8a30c2d5dd85ad36cd792ca6949b9972 |
|
BLAKE2b-256 | 0e09ccb6beb6f84211bc5e491de730957d27c60150987759eae9422650f65573 |