Skip to main content

Priority queue with task categorisation support

Project description

Priority queue with task categorisation support

Current build status Latest PyPI version

Overview

taskq is a heap based priority queue that adds support to partition tasks into categories for selective removal.

The main driver for the project was work/task queueing where jobs need to be distributed to workers with different capabilities and non-uniform processing times (for calculating efficient prefetch).

This library is not thread-safe and is not intended as a replacement for python’s queue.Queue class.

Getting Started

Install using pip:

pip install taskq

Basic FIFO usage:

>>> from taskq import Queue
>>> q = Queue()
>>> q.push('task1')
>>> q.push('task2')
>>> q.pop()
'task1'
>>> q.pop()
'task2'

Task priorities (complex types that define __cmp__ are also supported):

>>> from taskq import Queue
>>> q = Queue()
>>> q.push('task1', 2)
>>> q.push('task2', 1)
>>> q.pop()
'task2'
>>> q.pop()
'task1'

Multi-pop:

>>> from taskq import Queue
>>> q = Queue()
>>> q.push('task1')
>>> q.push('task2')
>>> q.push('task3')
>>> q.pop(2)
['task1', 'task2']

Task categories:

>>> from taskq import Queue
>>> q = Queue()
>>> q.push('task1', category='foo')
>>> q.push('task2', category='bar')
>>> q.push('task3', category='foo')
>>> q.pop(2, categories=['dog', 'foo'])
['task1', 'task2']

Category ratios:

>>> from taskq import Queue
>>> q = Queue()
>>> q.push('task1', category='foo')
>>> q.push('task2', category='bar')
>>> q.push('task3', category='foo')
>>> q.pop(2, categories=['bar', 'foo'], ratios=[1, 0.5])
['task1', 'task2', 'task3']

The above example is useful when prefetching/distributing tasks that have non-uniform processing durations. If the category ratios represent average duration in seconds, the count can be used to pull approximately count seconds worth of the higest priority tasks from the queue.

See source documentation for other feature examples.

Issues

Source code for taskq is hosted on GitHub. Any bug reports or feature requests can be made using GitHub’s issues system.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
taskq-0.3-py2.py3-none-any.whl (8.8 kB) Copy SHA256 hash SHA256 Wheel 2.7
taskq-0.3.tar.gz (16.8 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page