LitePipeline: distributed pipeline system
Project description
LitePipeline
A distributed pipeline system, based on Python3, tornado, venv-pack, pyinstaller.
All code based on Python3, do not use Python2!
It still under development, so, maybe have some bugs or not stable enough!
See more details at https://github.com/fiefdx/LitePipeline
Install
# install from pip
$ pip3 install -U litepipeline
# or install from source code
$ cd ./litepipeline
$ python3 ./setup.py install
# this will install 5 commands: liteconfig, litemanager, litenode, litepipeline, liteviewer
# liteconfig: to generate manager's or node's configuration file
# litemanager: to start LitePipeline manager
# litenode: to start LitePipeline node
# litepipeline: command line tool to communicate with LitePipeline cluster
# liteviewer: the web UI service, for communicate with LitePipeline cluster with web browser.
run
Run Manager
# create manager's data directory
$ mkdir ./manager_data
# generate manager's configuration file
$ cd ./manager_data
# this will generate a configuration.yml file under ./manager_data
$ liteconfig -s manager -o ./
# run manager
$ litemanager -c ./configuration.yml
# test
$ curl localhost:8000
# return this message
{"message": "LitePipeline manager service"}
Run Node
# create node's data directory
$ mkdir ./node_data
# generate node's configuration file
$ cd ./node_data
# this will generate a configuration.yml file under ./node_data
$ liteconfig -s node -o ./
# run node
# after start node, node will register to manager, and get a unique node id
$ litenode -c ./configuration.yml
# test
$ curl localhost:8001
# return this message
{"message": "LitePipeline node service"}
Run Viewer
# create viewer's data directory
$ mkdir ./viewer_data
# generate viewer's configuration file
$ cd ./viewer_data
# this will generate a configuration.yml file and other scripts under ./viewer_data
$ liteconfig -s viewer -o ./
# run viewer
$ liteviewer -c ./configuration.yml
# test
# use web browser open: http://localhost:8088
Use Web Browser Communicate With LitePipeline Cluster
Use web browser open the liteviewer's host & port in it's configuration file
Use Command Line Communicate With LitePipeline Cluster
$ litepipeline localhost:8000 cluster info
# | node_id | http_host | http_port | action_slots | app_path | data_path
1 | 32313239-e7ee-4f90-8c05-4e08fb48be70 | 127.0.0.1 | 8001 | 2 | /usr/local/lib/python3.7/dist-packages/litepipeline/node | /home/pi/Develop/litenode/data
# use -h/--help parameter to see help message
$ litepipeline --help
usage: litepipeline [-h] [-r] [-v] address {app,task,cluster} ...
positional arguments:
address manager address, host:port
{app,task,cluster} sub-command help
app operate with app API
task operate with task API
cluster operate with cluster API
optional arguments:
-h, --help show this help message and exit
-r, --raw display raw json data
-v, --version show program's version number and exit
$ litepipeline localhost:8000 app --help
usage: litepipeline address app [-h]
{create,delete,update,list,info,download} ...
positional arguments:
{create,delete,update,list,info,download}
sub-command app help
create create application
delete delete application
update update application
list list applications
info application's info
download download application
optional arguments:
-h, --help show this help message and exit
$ litepipeline localhost:8000 task --help
usage: litepipeline address task [-h] {create,delete,list,info,stop} ...
positional arguments:
{create,delete,list,info,stop}
sub-command task help
create create task
delete delete task
list list tasks
info task's info
stop stop task
optional arguments:
-h, --help show this help message and exit
$ litepipeline localhost:8000 cluster --help
usage: litepipeline address cluster [-h] {info} ...
positional arguments:
{info} sub-command cluster help
info cluster's info
optional arguments:
-h, --help show this help message and exit
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
litepipeline-0.0.18.tar.gz
(1.8 MB
view hashes)
Built Distribution
Close
Hashes for litepipeline-0.0.18-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e0e06e6ea8b6f6fbce4bc9d5be3cda2fc2b8ac0beab04d65b099e7e5dc686f45 |
|
MD5 | 01b6472471d337224b851be305cbf777 |
|
BLAKE2b-256 | 9f7a12310ba2ba9b3ec9a01b77a9f9e3a4cc5200fa330545f5089765afd8d3ad |