huey 3.3.0

a little task queue

a lightweight alternative.

huey is:

• a task queue

• written in python

• clean and simple API

huey has:

• support for redis (or valkey/redict), postgres, sqlite, file-system, or in-memory storage

• zero dependencies (redis-py required to use redis-like brokers, psycopg for postgres).

• example code.

• django integration (native or via django.tasks) with admin integration for visibility and management.

• documentation.

huey supports:

• multi-process, multi-thread or greenlet task execution models

• schedule tasks to execute at a given time, or after a given delay

• schedule recurring tasks, like a crontab

• automatically retry tasks that fail

• task prioritization

• task result storage

• task expiration

• task locking, rate-limits and timeouts

• task pipelines and chains

• groups (fan-out), chords (map / reduce)

At a glance

from huey import RedisHuey, crontab

# Or PostgresHuey, SqliteHuey, FileHuey, etc...
huey = RedisHuey('my-app', host='redis.myapp.com')

@huey.task()
def add_numbers(a, b):
    return a + b

@huey.task(retries=2, retry_delay=60)
def flaky_task(url):
    # This task might fail, in which case it will be retried up to 2 times
    # with a delay of 60s between retries.
    return this_might_fail(url)

@huey.periodic_task(crontab(minute='0', hour='3'))
def nightly_backup():
    sync_all_data()

Calling a task-decorated function will enqueue the function call for execution by the consumer. A special result handle is returned immediately, which can be used to fetch the result once the task is finished:

>>> from demo import add_numbers
>>> res = add_numbers(1, 2)
>>> res
<Result: task 6b6f36fc-da0d-4069-b46c-c0d4ccff1df6>

>>> res()
3

Tasks can be scheduled to run in the future:

>>> res = add_numbers.schedule((2, 3), delay=10)  # Will be run in ~10s.
>>> res(blocking=True)  # Will block until task finishes, in ~10s.
5

For much more, check out the guide or take a look at the example code.

Running the consumer

Run the consumer with four worker processes:

$ huey_consumer my_app.huey -k process -w 4

To run the consumer with a single worker thread (default):

$ huey_consumer my_app.huey

If your work-loads are mostly IO-bound, you can run the consumer with threads or greenlets instead. Because greenlets are so lightweight, you can run quite a few of them efficiently:

$ huey_consumer my_app.huey -k greenlet -w 32

Storage

Huey’s design and feature-set were informed by the capabilities of the Redis database. Redis is a fantastic fit for a lightweight task queueing library like Huey: it’s self-contained, versatile, and can be a multi-purpose solution for other web-application tasks like caching, event publishing, analytics, rate-limiting, and more.

Although Huey was designed with Redis in mind, the storage system implements a simple API and many other tools could be used instead of Redis if that’s your preference.

Huey comes with builtin support for Redis, Postgres, Sqlite, File-system, and in-memory storage.

Frameworks

Huey provides Django integration either natively or via django.tasks. Huey also provides an optional admin integration for Django:

Huey also provides flask-peewee admin integration based on the same underlying stat-tracking system:

Other frameworks can use the stats extension to collect and display this information.

Documentation

See Huey documentation.

Project page

See source code and issue tracker on Github.

Huey is named in honor of my cat: