redis-tq
redis-tq is a Redis-based multi-producer, multi-consumer Queue. Allows for sharing data between multiple processes or hosts.
Tasks support a "lease time". After that time other workers may consider this client to have crashed or stalled and pick up the item instead. The number of retries can be configured as well.
Based on this example but with many improvements added.
Installing
redis-tq is available on PyPI so you can simply install via:
$ pip install redis-tq
How to use
On the producing side, populate the queue with tasks and a respective lease timeout:
from redistq import TaskQueue
tq = TaskQueue('localhost', 'myqueue')
for i in range(10):
tq.add(some task, lease_timeout, ttl=3)
On the consuming side:
from redistq import TaskQueue
tq = TaskQueue('localhost', 'myqueue')
while True:
task, task_id = tq.get()
if task is not None:
# do something with task and mark it as complete afterwards
tq.complete(task_id)
if tq.is_empty():
break
# tq.get is non-blocking, so you may want to sleep a
# bit before the next iteration
time.sleep(1)
If the consumer crashes (i.e. the task is not marked as completed after
lease_timeout seconds), the task will be put back into the task queue. This
rescheduling will happen at most ttl times and then the task will be
dropped. A callback can be provided if you want to monitor such cases.
Running the tests
The tests will check the presence of a Redis instance on localhost, you can use
docker run --rm -d -p 6379:6379 redis
to get one. Then use make test, it will take care of creating an appropriate
virtualenv and use it to run the tests.
Metadata
Release files for redis-tq 0.0.11
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| redis-tq-0.0.11.tar.gz | 7.9 kB | Without provenance | Details for redis-tq-0.0.11.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| redis_tq-0.0.11-py3-none-any.whl | Python 3 | none | any | Details for redis_tq-0.0.11-py3-none-any.whl |
Total release size: 16.3 kB
Release files / redis-tq-0.0.11.tar.gz
| Download URL | redis-tq-0.0.11.tar.gz |
|---|---|
| Size | 7.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
13fdb4d8679c58383698e9d152bdcc29b8e00a89acf2578e3d36b84ff0bf7b37
|
|
BLAKE2b-256 checksum How to use checksums |
e8ac6fd24a63cf82b823449d08abf3aae0885c8814f279ce02def78bfab2f077
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.1
|
Release files / redis_tq-0.0.11-py3-none-any.whl
| Download URL | redis_tq-0.0.11-py3-none-any.whl |
|---|---|
| Size | 8.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4815d3bcfec709821f3e0cc8a908a9bec2934c22b23ca92cd1090bb4fdfcf41a
|
|
BLAKE2b-256 checksum How to use checksums |
4c919584dfb834de88376025417f58753fdb8cedfd580b50ecb69a61982bbe55
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.1
|
Security for redis-tq 0.0.11
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.