Skip to main content

A pipeline system for efficient execution.

Project description

Pyturbo Package

PyPI version Downloads Publish to PyPI

Author: Lijun Yu

Email: lijun@lj-y.com

A pipeline system for efficient execution.

Installation

pip install py-turbo

Introduction

Pyturbo utilizes multiple level of abstract to efficiently execute parallel tasks.

  • Worker: a process.
  • Stage: a group of peer workers processing the same type of tasks.
  • Task: a data unit transferred between stages. At each stage, a task is processed by one worker and will result in one or multiple downstream tasks.
  • Pipeline: a set of sequential stages.
  • Job: a data unit for a pipeline, typically a wrapped task for the first stage.
  • Result: output of a job processed by one pipeline, typically a set of output tasks from the last stage.
  • System: a set of peer pipelines processing the same type of jobs.

abstract.png

Get Started

from pyturbo import ReorderStage, Stage, System

class Stage1(Stage): # Define a stage

    def __init__(self, resources):
        ... # Optional: set resources and number of workers

    def process(self, task):
        ... # Process function for each worker process. Returns one or a series of downstream tasks.

... # Repeat for Stage2, Stage3

class Stage4(ReorderStage): # Define a reorder stage, typically for the final stage

    def get_sequence_id(self, task):
        ... # Return the order of each task. Start from 0.

    def process(self, task):
        ...

class MySystem(System):

    def get_stages(self, resources):
        ... # Define the stages in a pipeline with given resources.

    def get_results(self, results_gen):
        ... # Define how to extract final results from output tasks.

def main():
    system = MySystem(num_pipeline) # Set debug=True to run in a single process
    system.start() # Build and start system
    jobs = [...]
    system.add_jobs(jobs) # Submit jobs
    for job in system.wait_jobs(len(jobs)):
        print(job.results) # Process result
    system.end() # End system

Options

See options.md

Demo

abstract.png

See demo.py for an example implementation.

Version History

See version.md.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

py-turbo-0.5.0.tar.gz (9.9 kB view details)

Uploaded Source

Built Distribution

py_turbo-0.5.0-py3-none-any.whl (24.6 kB view details)

Uploaded Python 3

File details

Details for the file py-turbo-0.5.0.tar.gz.

File metadata

  • Download URL: py-turbo-0.5.0.tar.gz
  • Upload date:
  • Size: 9.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.6.0 requests/2.24.0 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.50.2 CPython/3.9.0

File hashes

Hashes for py-turbo-0.5.0.tar.gz
Algorithm Hash digest
SHA256 3b98173ea65438d91bf384a5863d2035fb8d161c994501cff83d2f9a31d49c4f
MD5 d31fdd278d09e444737b9f07375b1126
BLAKE2b-256 57d048e8f7a419d0c03c8dffc61a37779aa8b21f7b4f2ce0bebbec155f5e47a7

See more details on using hashes here.

File details

Details for the file py_turbo-0.5.0-py3-none-any.whl.

File metadata

  • Download URL: py_turbo-0.5.0-py3-none-any.whl
  • Upload date:
  • Size: 24.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.6.0 requests/2.24.0 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.50.2 CPython/3.9.0

File hashes

Hashes for py_turbo-0.5.0-py3-none-any.whl
Algorithm Hash digest
SHA256 ab8f758bcafbf958272fb4b2088ce2b5e8d19d3af36f4929e2e6f200b0bef6fc
MD5 e72ab0042766bc49af1bf2b4ba8b79f6
BLAKE2b-256 12a9e23c372a5a3b948d9a10856dd735cb50034cfca68c4aae6ee6d6e21bc537

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page