Skip to main content

A pipeline system for efficient execution.

Project description

Pyturbo Package

PyPI version Publish to PyPI

Author: Lijun Yu

Email: lijun@lj-y.com

A pipeline system for efficient execution.

Installation

pip install py-turbo

Introduction

Pyturbo utilizes multiple level of abstract to efficiently execute parallel tasks.

  • Worker: a process.
  • Stage: a group of peer workers processing the same type of tasks.
  • Task: a data unit transferred between stages. At each stage, a task is processed by one worker and will result in one or multiple downstream tasks.
  • Pipeline: a set of sequential stages.
  • Job: a data unit for a pipeline, typically a wrapped task for the first stage.
  • Result: output of a job processed by one pipeline, typically a set of output tasks from the last stage.
  • System: a set of peer pipelines processing the same type of jobs.

abstract.png

Get Started

from pyturbo import ReorderStage, Stage, System

class Stage1(Stage): # Define a stage

    def __init__(self, resources):
        ... # Optional: set resources and number of workers

    def process(self, task):
        ... # Process function for each worker process. Returns one or a series of downstream tasks.

... # Repeat for Stage2, Stage3

class Stage4(ReorderStage): # Define a reorder stage, typically for the final stage

    def get_sequence_id(self, task):
        ... # Return the order of each task. Start from 0.

    def process(self, task):
        ...

class MySystem(System):

    def get_stages(self, resources):
        ... # Define the stages in a pipeline with given resources.

    def get_results(self, results_gen):
        ... # Define how to extract final results from output tasks.

def main():
    system = MySystem(num_pipeline) # Set debug=True to run in a single process
    system.start() # Build and start system
    system.add_job(...) # Submit one job
    finished_job = system.result_queue.get() # Wait for result
    system.end() # End system

Options

See options.md

Demo

abstract.png

See demo.py for an example implementation.

Version History

See version.md.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

py-turbo-0.3.0.tar.gz (8.8 kB view details)

Uploaded Source

Built Distribution

py_turbo-0.3.0-py3-none-any.whl (23.5 kB view details)

Uploaded Python 3

File details

Details for the file py-turbo-0.3.0.tar.gz.

File metadata

  • Download URL: py-turbo-0.3.0.tar.gz
  • Upload date:
  • Size: 8.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.8.5

File hashes

Hashes for py-turbo-0.3.0.tar.gz
Algorithm Hash digest
SHA256 9b4ca6eda87d618a6cc70d3a30170b2ddf3049f30c1be64e089cf8c49499aad5
MD5 1db93688cab57a1a00473d34157e66f6
BLAKE2b-256 0e096904a55411b33c4017e2ed3cf447125da3c68096044117770a16f430a7fc

See more details on using hashes here.

File details

Details for the file py_turbo-0.3.0-py3-none-any.whl.

File metadata

  • Download URL: py_turbo-0.3.0-py3-none-any.whl
  • Upload date:
  • Size: 23.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.8.5

File hashes

Hashes for py_turbo-0.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 74088a48417aff5c522cdf06e98e9224405b721e6ed423da17e22c18c57c3152
MD5 b7d76393901e56a020b50a2da12d062e
BLAKE2b-256 7b2851677d1ee9d0b4f162b49f2b43b8b74125bcdca46094bcc2368d4c7dc41f

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page