Pyturbo Package
Author: Lijun Yu
Email: lijun@lj-y.com
A pipeline system for efficient execution.
Installation
pip install py-turbo
Introduction
Pyturbo utilizes multiple level of abstract to efficiently execute parallel tasks.
- Worker: a process.
- Stage: a group of peer workers processing the same type of tasks.
- Task: a data unit transferred between stages. At each stage, a task is processed by one worker and will result in one or multiple downstream tasks.
- Pipeline: a set of sequential stages.
- Job: a data unit for a pipeline, typically a wrapped task for the first stage.
- Result: output of a job processed by one pipeline, typically a set of output tasks from the last stage.
- System: a set of peer pipelines processing the same type of jobs.
Get Started
from pyturbo import ReorderStage, Stage, System
class Stage1(Stage): # Define a stage
def allocate_resource(self, resources, ...):
... # Optional: split resources and determine number of workers.
def process(self, task):
... # Process function for each worker process. Returns one or a series of downstream tasks.
... # Repeat for Stage2, Stage3
class Stage4(ReorderStage): # Define a reorder stage, typically for the final stage
def get_sequence_id(self, task):
... # Return the order of each task. Start from 0.
def process(self, task):
...
class MySystem(System):
def get_stages(self, resources):
... # Define the stages in a pipeline with given resources.
def get_results(self, results_gen):
... # Define how to extract final results from output tasks.
def main():
system = MySystem(num_pipeline) # Set debug=True to run in a single process
system.start() # Build and start system
jobs = [...]
system.add_jobs(jobs) # Submit jobs
for job in system.wait_jobs(len(jobs)):
print(job.results) # Process result
system.end() # End system
Options
See options.md
Demo
See demo.py for an example implementation.
Version History
See version.md.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
py-turbo-0.6.10.tar.gz
(22.6 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
py_turbo-0.6.10-py3-none-any.whl
(25.5 kB
view details)
File details
Details for the file py-turbo-0.6.10.tar.gz.
File metadata
- Download URL: py-turbo-0.6.10.tar.gz
- Upload date:
- Size: 22.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/4.0.0 CPython/3.10.4
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
12e932d4d77a68378390d8c8145036fc4f24f2629a20294b177067f5a8224ec4
|
|
| MD5 |
0ef4596cf28a2ee9c1dc53084136ce8d
|
|
| BLAKE2b-256 |
6c45604db50ab909637a901f291d4865f60ae197da960e0ce116cd421411cb42
|
File details
Details for the file py_turbo-0.6.10-py3-none-any.whl.
File metadata
- Download URL: py_turbo-0.6.10-py3-none-any.whl
- Upload date:
- Size: 25.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/4.0.0 CPython/3.10.4
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e710735bbea38c2d1513804dec9becc2dde4bcf64ad5a9964fb55a5a31482648
|
|
| MD5 |
32ef53f73b6862e3f3c67a7ead157af7
|
|
| BLAKE2b-256 |
86ca61566c8a1b50195ecf681fa42e11548cab600271fa6726d2724787124111
|
