Skip to main content

Manage parallel tasks

Project description

Summary

A simple framework to run tasks in parallel. It’s similar to multiprocessing.Pool, but has a few enhancements over that. For example, mp.Pool is only useful for multiprocessing functions (not objects). You can wrap a function around the object, but it’s nicer just to deal with task objects themselves.

Polymer is mostly useful for its Worker error logging and run-time statistics. It also restarts crashed multiprocessing workers automatically (not true with multiprocessing.Pool). When a worker crashes, Polymer knows what the worker was doing and resubmits that task as well. This definitely is not fool-proof; however, it’s a helpful feature.

Once TaskMgr().supervise() finishes, a list of object instances is returned. You can store per-task results as an attribute of each object instance.

Usage

import time

from polymer.Polymer import ControllerQueue, TaskMgr
from polymer.abc_task import BaseTask

class SimpleTask(BaseTask):
    def __init__(self, text="", wait=0.0):
        super(SimpleTask, self).__init__()
        self.text = text
        self.wait = wait

    def run(self):
        """run() is where all the work is done; this is called by TaskMgr()"""
        ## WARNING... using try / except in run() could squash Polymer's
        ##      internal error logging...
        #time.sleep(float(self.wait/10))
        print(self.text, self.wait/10.0)

    def __eq__(self, other):
        """Define how tasks are uniquely identified"""
        if isinstance(other, SimpleTask) and (other.text==self.text):
            return True
        return False

    def __repr__(self):
        return """<{0}, wait: {1}>""".format(self.text, self.wait)

    def __hash__(self):
        return id(self)

def Controller():
    """Controller() builds a list of tasks, and queues them to the TaskMgr
    There is nothing special about the name Controller()... it's just some
    code to build a list of SimpleTask() instances."""

    tasks = list()

    ## Build ten tasks... do *not* depend on execution order...
    num_tasks = 10
    for ii in range(0, num_tasks):
        tasks.append(SimpleTask(text="Task {0}".format(ii), wait=ii))

    targs = {
        'work_todo': tasks,  # a list of SimpleTask() instances
        'hot_loop': False,   # If True, continuously loop over the tasks
        'worker_count': 3,           # Number of workers (default: 5)
        'resubmit_on_error': False,  # Do not retry errored jobs...
        'queue': ControllerQueue(),
        'worker_cycle_sleep': 0.001, # Worker sleep time after a task
        'log_stdout': False,         # Don't log to stdout (default: True)
        'log_path':  "taskmgr.log",  # Log file name
        'log_level': 0,              # Logging off is 0 (debugging=3)
        'log_interval': 10,          # Statistics logging interval
    }

    ## task_mgr reads and executes the queued tasks
    task_mgr = TaskMgr(**targs)

    ## a set() of completed task objects are returned after supervise()
    results = task_mgr.supervise()
    return results

if __name__=='__main__':
    Controller()

License

GPLv3

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

polymer-0.0.69.tar.gz (26.6 kB view details)

Uploaded Source

Built Distribution

polymer-0.0.69-py3-none-any.whl (25.6 kB view details)

Uploaded Python 3

File details

Details for the file polymer-0.0.69.tar.gz.

File metadata

  • Download URL: polymer-0.0.69.tar.gz
  • Upload date:
  • Size: 26.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.2

File hashes

Hashes for polymer-0.0.69.tar.gz
Algorithm Hash digest
SHA256 bf8da94d1a656f1ebfdd08b477192bd6e9c72c2c5e3260587d8b60d67ddbeda5
MD5 aa22c51eb74bb58addd4eabc50ef2047
BLAKE2b-256 3151b6dfa1282a3c624dd13d46477319bae5602e91113b0422366f517e844803

See more details on using hashes here.

File details

Details for the file polymer-0.0.69-py3-none-any.whl.

File metadata

  • Download URL: polymer-0.0.69-py3-none-any.whl
  • Upload date:
  • Size: 25.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.2

File hashes

Hashes for polymer-0.0.69-py3-none-any.whl
Algorithm Hash digest
SHA256 a1611e811cf61e05bf1bea60bb1af794ef3ae4c109b96382e4fc38cafc306860
MD5 b3fbbaa26a08b3c3bee732a451d93fd7
BLAKE2b-256 aedc3d4bfe0663e2dcf364f36217e37d192d91ac02cbec8ca5f810c7ea446eec

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page