RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. It is backed by Redis/Valkey and is designed to have a low barrier to entry while scaling incredibly well for large applications. It can be integrated into your web stack easily, making it suitable for projects of any size—from simple applications to high-volume enterprise systems.
RQ requires Redis >= 5 or Valkey >= 7.2.
Full documentation can be found here.
Support RQ
If you find RQ useful, please consider supporting this project via Tidelift.
Getting started
First, run a Redis/Valkey server:
$ redis-server
To put jobs on queues, you don't have to do anything special, just define your typically lengthy or blocking function:
import requests
def count_words_at_url(url):
"""Just an example function that's called async."""
resp = requests.get(url)
return len(resp.text.split())
Then, create an RQ queue:
from redis import Redis
from rq import Queue
queue = Queue(connection=Redis())
And enqueue the function call:
from my_module import count_words_at_url
job = queue.enqueue(count_words_at_url, 'https://stamps.id')
Job Prioritization
By default, jobs are added to the end of a single queue. RQ offers two ways to give certain jobs higher priority:
1. Enqueue at the front
You can enqueue a job at the front of its queue so it’s picked up before other jobs:
job = queue.enqueue(count_words_at_url, 'https://stamps.id', at_front=True)
2. Use multiple queues
You can create multiple queues and enqueue jobs into different queues based on their priority:
from rq import Queue
high_priority_queue = Queue('high', connection=Redis())
low_priority_queue = Queue('low', connection=Redis())
# This job will be picked up before jobs in the low priority queue
# even if it was enqueued later
high_priority_queue.enqueue(urgent_task)
low_priority_queue.enqueue(non_urgent_task)
Then start workers with a prioritized queue list:
$ rq worker high low
This command starts a worker that listens to both high and low queues. The worker will process
jobs from the high queue first, followed by the low queue. You can also run different workers
for different queues, allowing you to scale your workers based on the number of jobs in each queue.
Scheduling Jobs
Scheduling jobs is also easy:
# Schedule job to run at 9:15, October 10th
job = queue.enqueue_at(datetime(2019, 10, 10, 9, 15), say_hello)
# Schedule job to run in 10 seconds
job = queue.enqueue_in(timedelta(seconds=10), say_hello)
Repeating Jobs
To execute a Job multiple times, use the Repeat class:
from rq import Queue, Repeat
# Repeat job 3 times after successful execution, with 30 second intervals
queue.enqueue(my_function, repeat=Repeat(times=3, interval=30))
# Repeat job 3 times with different intervals between runs
queue.enqueue(my_function, repeat=Repeat(times=3, interval=[5, 10, 15]))
Unique Jobs
You can prevent duplicate jobs from being enqueued by using the unique parameter:
job = queue.enqueue(send_email, user_id, job_id='welcome-42', unique=True)
More details can be found in the docs.
Retrying Failed Jobs
Retrying failed jobs is also supported:
from rq import Retry
# Retry up to 3 times, failed job will be requeued immediately
queue.enqueue(say_hello, retry=Retry(max=3))
# Retry up to 3 times, with configurable intervals between retries
queue.enqueue(say_hello, retry=Retry(max=3, interval=[10, 30, 60]))
For a more complete example, refer to the docs. But this is the essence.
Webhooks
RQ can send an HTTP request to a URL when a job finishes or fails, without writing a callback function:
from rq import Webhook
queue.enqueue(
say_hello,
webhooks=[
Webhook('https://example.com/finished', job_status='finished'),
Webhook('https://example.com/failed', job_status='failed', method='POST'),
],
)
More details can be found in the docs.
Interval and Cron Job Scheduling
RQ >= 2.5 provides built-in job scheduling functionality that supports both simple interval-based scheduling and flexible cron syntax.
First, create a configuration file (e.g., cron_config.py) that defines the jobs you want to run periodically.
from rq import cron
from myapp import cleanup_database, send_daily_report
# Run database cleanup every 5 minutes
from rq import cron
from myapp import cleanup_temp_files, generate_analytics_report
# Clean up temporary files every 30 minutes
cron.register(
cleanup_temp_files,
queue_name='maintenance',
interval=1800 # 30 minutes in seconds
)
# Generate analytics report every 6 hours
cron.register(
generate_analytics_report,
queue_name='reports',
args=('daily_metrics',),
kwargs={'format': 'json', 'recipients': ['bob@example.com']},
interval=21600 # 6 hours in seconds
)
And then start the rq cron command to enqueue these jobs at specified intervals:
$ rq cron cron_config.py
You can also use standard cron syntax for more flexible scheduling:
from rq import cron
from myapp import send_newsletter, backup_database
# Database backup every day at 3:00 AM
cron.register(
backup_database,
queue_name='maintenance',
cron='0 3 * * *'
)
# Monthly report on the first day of each month at 8:00 AM
cron.register(
generate_monthly_report,
queue_name='reports',
cron='0 8 1 * *'
)
Cron jobs can also send webhooks to a monitoring endpoint when a scheduled run finishes or fails:
```python
from rq import cron, Webhook
# Ping a monitoring endpoint after the nightly backup finishes or fails
cron.register(
backup_database,
queue_name='maintenance',
cron='0 3 * * *',
webhooks=[
Webhook('https://example.com/finished', 'finished'),
Webhook('https://example.com/failed', 'failed'),
],
)
More details on functionality can be found in the docs.
The Worker
To start executing enqueued function calls in the background, start a worker from your project's directory:
$ rq worker --with-scheduler
*** Listening for work on default
Got count_words_at_url('http://nvie.com') from default
Job result = 818
*** Listening for work on default
To run multiple workers in production, use process managers like systemd. RQ also ships with a worker-pool that lets you run multiple worker processes with a single command.
$ rq worker-pool -n 4
More options are documented on python-rq.org.
Security
Warning: RQ uses
pickleas its default serializer, which is not secure. Only run RQ against Redis instances that you trust. It is possible to construct malicious pickle data that will execute arbitrary code during unpickling.
To avoid pickle, use an alternative serializer, such as JSONSerializer, when enqueueing and processing jobs (see docs). JSON only supports primitive argument types (str, int, float, bool, list, dict, None).
Installation
Simply use the following command to install the latest released version:
$ pip install rq
Notes on Performance
TL;DR — run Worker or SpawnWorker in production.
In a simple hello world microbenchmark, SimpleWorker processed 1,000 jobs in just 1.02 seconds vs. 6.64 seconds with the default Worker), more than 6x faster.
SimpleWorker is faster because it skips fork() or spawn() and runs jobs in process. Worker and SpawnWorker run each job in a separate process, acting as a sandbox that isolates crashes, memory leaks and enforce hard time-outs.
Although SimpleWorker is faster in benchmarks, this overhead is negligible in most real world applications like sending emails, generating reports, processing images, etc. In production systems, the time spent performing jobs usually dwarfs any queueing/worker overhead.
Use SimpleWorker in production only if:
- Your jobs are extremely short-lived (single digit milliseconds).
- The
fork()orspawn()latency is a proven bottleneck at your traffic levels. - Your job code is 100% trusted and known to be free of resource leaks and the possibility of crashing/segfaults.
Docs
To build and run the docs, install jekyll and run:
cd docs
jekyll serve
Related Projects
If you use RQ, Check out these below repos which might be useful in your rq based project.
Project history
This project has been inspired by the good parts of Celery, Resque and this snippet, and has been created as a lightweight alternative to the heaviness of Celery or other AMQP-based queueing implementations.
RQ is maintained by Stamps, an Indonesian based company that provides enterprise grade CRM and order management systems.
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file rq-2.11.0.tar.gz.
File metadata
- Download URL: rq-2.11.0.tar.gz
- Upload date:
- Size: 759.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.11.29 {"installer":{"name":"uv","version":"0.11.29","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
1ed9db6b685707a7dfd535532a408f1a8f0a5ad720a307be6b844f4142999e78
|
|
| MD5 |
c3041cdb719c9cfff24bc0304f874a96
|
|
| BLAKE2b-256 |
d78a7fcf759b5b685b26ef029af24575359bf8f5b5b340d92b08bb193a4163b1
|
File details
Details for the file rq-2.11.0-py3-none-any.whl.
File metadata
- Download URL: rq-2.11.0-py3-none-any.whl
- Upload date:
- Size: 126.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.11.29 {"installer":{"name":"uv","version":"0.11.29","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2f54a31375d7c1b5a1642acef4948809ce6484775b9741fe7edd9399ec9306a9
|
|
| MD5 |
9f0b1e85e651263352726ce84caf92f2
|
|
| BLAKE2b-256 |
801ef85cba91fe58c63812b4acce073fc448230381c4785e03832f455305ff82
|
