LitePipeline: distributed pipeline system
Project description
LitePipeline
A distributed pipeline system, based on Python3, tornado, venv-pack, pyinstaller.
All code based on Python3, do not use Python2!
It still under development, so, maybe have some bugs or not stable enough!
See more details at https://github.com/fiefdx/LitePipeline
Install
# install from pip
$ pip3 install -U litepipeline
# or install from source code
$ cd ./litepipeline
$ python3 ./setup.py install
# this will install 5 commands: liteconfig, litemanager, litenode, litepipeline, liteviewer
# liteconfig: to generate manager's or node's configuration file
# litemanager: to start LitePipeline manager
# litenode: to start LitePipeline node
# litepipeline: command line tool to communicate with LitePipeline cluster
# liteviewer: the web UI service, for communicate with LitePipeline cluster with web browser.
run
Run Manager
# create manager's data directory
$ mkdir ./manager_data
# generate manager's configuration file
$ cd ./manager_data
# this will generate a configuration.yml file under ./manager_data
$ liteconfig -s manager -o ./
# run manager
$ litemanager -c ./configuration.yml
# test
$ curl localhost:8000
# return this message
{"message": "LitePipeline manager service"}
Run Node
# create node's data directory
$ mkdir ./node_data
# generate node's configuration file
$ cd ./node_data
# this will generate a configuration.yml file under ./node_data
$ liteconfig -s node -o ./
# run node
# after start node, node will register to manager, and get a unique node id
$ litenode -c ./configuration.yml
# test
$ curl localhost:8001
# return this message
{"message": "LitePipeline node service"}
Run Viewer
# create viewer's data directory
$ mkdir ./viewer_data
# generate viewer's configuration file
$ cd ./viewer_data
# this will generate a configuration.yml file and other scripts under ./viewer_data
$ liteconfig -s viewer -o ./
# run viewer
$ liteviewer -c ./configuration.yml
# test
# use web browser open: http://localhost:8088
Use Web Browser Communicate With LitePipeline Cluster
Use web browser open the liteviewer's host & port in it's configuration file
Use Command Line Communicate With LitePipeline Cluster
$ litepipeline localhost:8000 cluster info
# | node_id | http_host | http_port | action_slots | app_path | data_path
1 | 32313239-e7ee-4f90-8c05-4e08fb48be70 | 127.0.0.1 | 8001 | 2 | /usr/local/lib/python3.7/dist-packages/litepipeline/node | /home/pi/Develop/litenode/data
# use -h/--help parameter to see help message
$ litepipeline --help
usage: litepipeline [-h] [-r] [-v] address {app,task,cluster} ...
positional arguments:
address manager address, host:port
{app,task,cluster} sub-command help
app operate with app API
task operate with task API
cluster operate with cluster API
optional arguments:
-h, --help show this help message and exit
-r, --raw display raw json data
-v, --version show program's version number and exit
$ litepipeline localhost:8000 app --help
usage: litepipeline address app [-h]
{create,delete,update,list,info,download} ...
positional arguments:
{create,delete,update,list,info,download}
sub-command app help
create create application
delete delete application
update update application
list list applications
info application's info
download download application
optional arguments:
-h, --help show this help message and exit
$ litepipeline localhost:8000 task --help
usage: litepipeline address task [-h] {create,delete,list,info,stop} ...
positional arguments:
{create,delete,list,info,stop}
sub-command task help
create create task
delete delete task
list list tasks
info task's info
stop stop task
optional arguments:
-h, --help show this help message and exit
$ litepipeline localhost:8000 cluster --help
usage: litepipeline address cluster [-h] {info} ...
positional arguments:
{info} sub-command cluster help
info cluster's info
optional arguments:
-h, --help show this help message and exit
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
litepipeline-0.0.19.tar.gz
(1.8 MB
view hashes)
Built Distribution
Close
Hashes for litepipeline-0.0.19-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec8a10fffd31c3c5635082a1f11f7b5d112f4abc957d452eb0fe0a547ca5863b |
|
MD5 | 9119f16805f422350142b633506e40d0 |
|
BLAKE2b-256 | 456dcd766d2de2ab9253e3939062ba3a92e00cf7d388430d902f82c85318d87d |