Skip to main content

huey, a little task queue

Project description

http://media.charlesleifer.com/blog/photos/huey2-logo.png

a lightweight alternative.

huey is:

huey supports:

  • multi-process, multi-thread or greenlet task execution models

  • schedule tasks to execute at a given time, or after a given delay

  • schedule recurring tasks, like a crontab

  • automatically retry tasks that fail

  • task prioritization

  • task result storage

  • task expiration

  • task locking

  • task pipelines and chains

http://i.imgur.com/2EpRs.jpg

At a glance

from huey import RedisHuey, crontab

huey = RedisHuey('my-app', host='redis.myapp.com')

@huey.task()
def add_numbers(a, b):
    return a + b

@huey.task(retries=2, retry_delay=60)
def flaky_task(url):
    # This task might fail, in which case it will be retried up to 2 times
    # with a delay of 60s between retries.
    return this_might_fail(url)

@huey.periodic_task(crontab(minute='0', hour='3'))
def nightly_backup():
    sync_all_data()

Calling a task-decorated function will enqueue the function call for execution by the consumer. A special result handle is returned immediately, which can be used to fetch the result once the task is finished:

>>> from demo import add_numbers
>>> res = add_numbers(1, 2)
>>> res
<Result: task 6b6f36fc-da0d-4069-b46c-c0d4ccff1df6>

>>> res()
3

Tasks can be scheduled to run in the future:

>>> res = add_numbers.schedule((2, 3), delay=10)  # Will be run in ~10s.
>>> res(blocking=True)  # Will block until task finishes, in ~10s.
5

For much more, check out the guide or take a look at the example code.

Running the consumer

Run the consumer with four worker processes:

$ huey_consumer.py my_app.huey -k process -w 4

To run the consumer with a single worker thread (default):

$ huey_consumer.py my_app.huey

If your work-loads are mostly IO-bound, you can run the consumer with threads or greenlets instead. Because greenlets are so lightweight, you can run quite a few of them efficiently:

$ huey_consumer.py my_app.huey -k greenlet -w 32

Storage

Huey’s design and feature-set were informed by the capabilities of the Redis database. Redis is a fantastic fit for a lightweight task queueing library like Huey: it’s self-contained, versatile, and can be a multi-purpose solution for other web-application tasks like caching, event publishing, analytics, rate-limiting, and more.

Although Huey was designed with Redis in mind, the storage system implements a simple API and many other tools could be used instead of Redis if that’s your preference.

Huey comes with builtin support for Redis, Sqlite and in-memory storage.

Documentation

See Huey documentation.

Project page

See source code and issue tracker on Github.

Huey is named in honor of my cat:

http://m.charlesleifer.com/t/800x-/blog/photos/p1473037658.76.jpg?key=mD9_qMaKBAuGPi95KzXYqg

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

huey-2.5.3.tar.gz (836.9 kB view details)

Uploaded Source

File details

Details for the file huey-2.5.3.tar.gz.

File metadata

  • Download URL: huey-2.5.3.tar.gz
  • Upload date:
  • Size: 836.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: Python-urllib/3.7

File hashes

Hashes for huey-2.5.3.tar.gz
Algorithm Hash digest
SHA256 089fc72b97fd26a513f15b09925c56fad6abe4a699a1f0e902170b37e85163c7
MD5 9f6e4433b6559ba5a6d255b6f7cf2771
BLAKE2b-256 868c2dfecf3705f5e522097b4e9fb6fb38e627a0340f8362cc05bd7a845e4279

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page