Pyturbo Package
Author: Lijun Yu
Email: lijun@lj-y.com
A pipeline system for efficient execution.
Installation
pip install py-turbo
Introduction
Pyturbo utilizes multiple level of abstract to efficiently execute parallel tasks.
- Worker: a process.
- Stage: a group of peer workers processing the same type of tasks.
- Task: a data unit transferred between stages. At each stage, a task is processed by one worker and will result in one or multiple downstream tasks.
- Pipeline: a set of sequential stages.
- Job: a data unit for a pipeline, typically a wrapped task for the first stage.
- Result: output of a job processed by one pipeline, typically a set of output tasks from the last stage.
- System: a set of peer pipelines processing the same type of jobs.
Get Started
from pyturbo import ReorderStage, Stage, System
class Stage1(Stage): # Define a stage
def allocate_resource(self, resources, ...):
... # Optional: split resources and determine number of workers.
def process(self, task):
... # Process function for each worker process. Returns one or a series of downstream tasks.
... # Repeat for Stage2, Stage3
class Stage4(ReorderStage): # Define a reorder stage, typically for the final stage
def get_sequence_id(self, task):
... # Return the order of each task. Start from 0.
def process(self, task):
...
class MySystem(System):
def get_stages(self, resources):
... # Define the stages in a pipeline with given resources.
def get_results(self, results_gen):
... # Define how to extract final results from output tasks.
def main():
system = MySystem(num_pipeline) # Set debug=True to run in a single process
system.start() # Build and start system
jobs = [...]
system.add_jobs(jobs) # Submit jobs
for job in system.wait_jobs(len(jobs)):
print(job.results) # Process result
system.end() # End system
Options
See options.md
Demo
See demo.py for an example implementation.
Version History
See version.md.
1 maintainer
Metadata
1 maintainer
Release files for py-turbo 0.6.9
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| py-turbo-0.6.9.tar.gz | 22.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| py_turbo-0.6.9-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 47.7 kB
Release files / py-turbo-0.6.9.tar.gz
| Download URL | py-turbo-0.6.9.tar.gz |
|---|---|
| Size | 22.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
37e88fe0043b2f63ec566747e999bd84d4d644147ffd3aaa331127e3528fea6b
|
|
BLAKE2b-256 checksum How to use checksums |
f3d757c85f4e39e387a92eb7dca4a1f0832a04f4cbc7d3953793eaa12035fd40
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.2
|
Release files / py_turbo-0.6.9-py3-none-any.whl
| Download URL | py_turbo-0.6.9-py3-none-any.whl |
|---|---|
| Size | 25.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6ffdd52149ac11ff8f0e6b24f284314bb68fa54f89eb786d7dea827e3befdd8e
|
|
BLAKE2b-256 checksum How to use checksums |
2746a66bc98852f487c3938364875e4e333921323e3d7d500a5c447426e3d050
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.2
|
