Skip to main content

Priority queue with task categorisation support

Project description

Priority queue with task categorisation support

Overview

taskq is a heap based priority queue that adds support to partition tasks into categories for selective removal.

The main driver for the project was work/task queueing where jobs need to be distributed to workers with different capabilities and non-uniform processing times (for calculating efficient prefetch).

This library is not thread-safe and is not intended as a replacement for python’s queue.Queue class.

Getting Started

Install using pip:

pip install taskq

Basic FIFO usage:

>>> from taskq import Queue
>>> q = Queue()
>>> q.push('task1')
>>> q.push('task2')
>>> q.pop()
'task1'
>>> q.pop()
'task2'

Task priorities (complex types that define __cmp__ are also supported):

>>> from taskq import Queue
>>> q = Queue()
>>> q.push('task1', 2)
>>> q.push('task2', 1)
>>> q.pop()
'task2'
>>> q.pop()
'task1'

Multi-pop:

>>> from taskq import Queue
>>> q = Queue()
>>> q.push('task1')
>>> q.push('task2')
>>> q.push('task3')
>>> q.pop(2)
['task1', 'task2']

Task categories:

>>> from taskq import Queue
>>> q = Queue()
>>> q.push('task1', category='foo')
>>> q.push('task2', category='bar')
>>> q.push('task3', category='foo')
>>> q.pop(2, categories=['dog', 'foo'])
['task1', 'task2']

Category ratios:

>>> from taskq import Queue
>>> q = Queue()
>>> q.push('task1', category='foo')
>>> q.push('task2', category='bar')
>>> q.push('task3', category='foo')
>>> q.pop(2, categories=['bar', 'foo'], ratios=[1, 0.5])
['task1', 'task2', 'task3']

The above example is useful when prefetching/distributing tasks that have non-uniform processing durations. If the category ratios represent average duration in seconds, the count can be used to pull approximately count seconds worth of the higest priority tasks from the queue.

See source documentation for other feature examples.

Issues

Source code for taskq is hosted on GitHub. Any bug reports or feature requests can be made using GitHub’s issues system.

Project details


Release history Release notifications

History Node

0.3

History Node

0.2

This version
History Node

0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
taskq-0.1.tar.gz (16.6 kB) Copy SHA256 hash SHA256 Source None Jun 30, 2014

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page