A pipeline system for efficient execution.
Project description
Pyturbo Package
Author: Lijun Yu
Email: lijun@lj-y.com
A pipeline system for efficient execution.
Installation
pip install py-turbo
Introduction
Pyturbo
utilizes multiple level of abstract to efficiently execute parallel tasks.
- Worker: a process.
- Stage: a group of peer workers processing the same type of tasks.
- Task: a data unit transferred between stages. At each stage, a task is processed by one worker and will result in one or multiple downstream tasks.
- Pipeline: a set of sequential stages.
- Job: a data unit for a pipeline, typically a wrapped task for the first stage.
- Result: output of a job processed by one pipeline, typically a set of output tasks from the last stage.
- System: a set of peer pipelines processing the same type of jobs.
Get Started
from pyturbo import ReorderStage, Stage, System
class Stage1(Stage): # Define a stage
def __init__(self, resources):
... # Optional: set resources and number of workers
def process(self, task):
... # Process function for each worker process. Returns one or a series of downstream tasks.
... # Repeat for Stage2, Stage3
class Stage4(ReorderStage): # Define a reorder stage, typically for the final stage
def get_sequence_id(self, task):
... # Return the order of each task. Start from 0.
def process(self, task):
...
class MySystem(System):
def get_stages(self, resources):
... # Define the stages in a pipeline with given resources.
def get_results(self, results_gen):
... # Define how to extract final results from output tasks.
def main():
system = MySystem(num_pipeline) # Set debug=True to run in a single process
system.start() # Build and start system
system.add_job(...) # Submit one job
finished_job = system.result_queue.get() # Wait for result
system.end() # End system
Options
See options.md
Demo
See demo.py for an example implementation.
Version History
See version.md.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
py-turbo-0.3.0.tar.gz
(8.8 kB
view details)
Built Distribution
py_turbo-0.3.0-py3-none-any.whl
(23.5 kB
view details)
File details
Details for the file py-turbo-0.3.0.tar.gz
.
File metadata
- Download URL: py-turbo-0.3.0.tar.gz
- Upload date:
- Size: 8.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.8.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
9b4ca6eda87d618a6cc70d3a30170b2ddf3049f30c1be64e089cf8c49499aad5
|
|
MD5 |
1db93688cab57a1a00473d34157e66f6
|
|
BLAKE2b-256 |
0e096904a55411b33c4017e2ed3cf447125da3c68096044117770a16f430a7fc
|
File details
Details for the file py_turbo-0.3.0-py3-none-any.whl
.
File metadata
- Download URL: py_turbo-0.3.0-py3-none-any.whl
- Upload date:
- Size: 23.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.8.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
74088a48417aff5c522cdf06e98e9224405b721e6ed423da17e22c18c57c3152
|
|
MD5 |
b7d76393901e56a020b50a2da12d062e
|
|
BLAKE2b-256 |
7b2851677d1ee9d0b4f162b49f2b43b8b74125bcdca46094bcc2368d4c7dc41f
|