a lightweight alternative.
huey is:
a task queue
written in python
clean and simple API
huey has:
support for redis (or valkey/redict), postgres, sqlite, file-system, or in-memory storage
zero dependencies (redis-py required to use redis-like brokers, psycopg for postgres).
django integration (native or via django.tasks) with admin integration for visibility and management.
huey supports:
multi-process, multi-thread or greenlet task execution models
schedule tasks to execute at a given time, or after a given delay
schedule recurring tasks, like a crontab
automatically retry tasks that fail
task prioritization
task result storage
task expiration
task locking, rate-limits and timeouts
task pipelines and chains
groups (fan-out), chords (map / reduce)
At a glance
from huey import RedisHuey, crontab
# Or PostgresHuey, SqliteHuey, FileHuey, etc...
huey = RedisHuey('my-app', host='redis.myapp.com')
@huey.task()
def add_numbers(a, b):
return a + b
@huey.task(retries=2, retry_delay=60)
def flaky_task(url):
# This task might fail, in which case it will be retried up to 2 times
# with a delay of 60s between retries.
return this_might_fail(url)
@huey.periodic_task(crontab(minute='0', hour='3'))
def nightly_backup():
sync_all_data()Calling a task-decorated function will enqueue the function call for execution by the consumer. A special result handle is returned immediately, which can be used to fetch the result once the task is finished:
>>> from demo import add_numbers
>>> res = add_numbers(1, 2)
>>> res
<Result: task 6b6f36fc-da0d-4069-b46c-c0d4ccff1df6>
>>> res()
3Tasks can be scheduled to run in the future:
>>> res = add_numbers.schedule((2, 3), delay=10) # Will be run in ~10s.
>>> res(blocking=True) # Will block until task finishes, in ~10s.
5For much more, check out the guide or take a look at the example code.
Running the consumer
Run the consumer with four worker processes:
$ huey_consumer my_app.huey -k process -w 4To run the consumer with a single worker thread (default):
$ huey_consumer my_app.hueyIf your work-loads are mostly IO-bound, you can run the consumer with threads or greenlets instead. Because greenlets are so lightweight, you can run quite a few of them efficiently:
$ huey_consumer my_app.huey -k greenlet -w 32Storage
Huey’s design and feature-set were informed by the capabilities of the Redis database. Redis is a fantastic fit for a lightweight task queueing library like Huey: it’s self-contained, versatile, and can be a multi-purpose solution for other web-application tasks like caching, event publishing, analytics, rate-limiting, and more.
Although Huey was designed with Redis in mind, the storage system implements a simple API and many other tools could be used instead of Redis if that’s your preference.
Huey comes with builtin support for Redis, Postgres, Sqlite, File-system, and in-memory storage.
Frameworks
Huey provides Django integration either natively or via django.tasks. Huey also provides an optional admin integration for Django:
Huey also provides flask-peewee admin integration based on the same underlying stat-tracking system:
Other frameworks can use the stats extension to collect and display this information.
Documentation
Project page
See source code and issue tracker on Github.
Huey is named in honor of my cat:
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file huey-3.3.1.tar.gz.
File metadata
- Download URL: huey-3.3.1.tar.gz
- Upload date:
- Size: 612.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
de3f4b9a9cb045f365599c50558ffab0416d5555b3c87bd35531c7b75c1149d1
|
|
| MD5 |
09e204dbac9b43c7555e679b3801daea
|
|
| BLAKE2b-256 |
b95f823732401f1e3a7ebf3ad6e79d45d4d27a16ad88d55a6a6efc7ac71d17bc
|
File details
Details for the file huey-3.3.1-py3-none-any.whl.
File metadata
- Download URL: huey-3.3.1-py3-none-any.whl
- Upload date:
- Size: 123.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
b0ab6bd65fa39774e601429c1224e3c345374b06056345161132cdff19d83ebb
|
|
| MD5 |
601fd81d79f7d7252342159b330710ce
|
|
| BLAKE2b-256 |
821ff03f2dc13d3d92abf0d0c0d8be44bffdd86e3e17f1fe6f8c657a34672645
|
