arq-optimus 0.27.0

ARQ fork with key-prefix isolation, cancellable jobs, and multi-project Redis support

ARQ Optimus

ARQ Optimus is a production-ready fork of ARQ (Async Redis Queue) with key-prefix isolation, cancellable jobs, and multi-project Redis support.

Why ARQ Optimus?

Stock ARQ uses hardcoded Redis key names (arq:queue, arq:job:*, etc.), which means multiple projects sharing the same Redis instance will collide. ARQ Optimus solves this with a configurable key_prefix that namespaces all Redis keys per project.

Feature	Stock ARQ	ARQ Optimus
Key prefix isolation	❌	✅ myproject:arq:queue
Multi-project Redis	❌	✅ Safe sharing
Job tracking queues	❌	✅ Running/completed/failed/cancelled
Cancellable jobs	Partial	✅ Full support with status tracking
Drop-in compatible	—	✅ Import path unchanged

Installation

pip install arq-optimus

Quick Start

ARQ Optimus is a drop-in replacement for ARQ. Just change your install source — import paths stay the same:

# Before (stock arq)
# pip install arq

# After (arq-optimus)
# pip install arq-optimus
from arq import create_pool, Worker
from arq.connections import RedisSettings

Basic Usage (No Prefix)

Works exactly like stock ARQ — fully backward compatible:

import asyncio
from arq import create_pool
from arq.connections import RedisSettings

async def say_hello(ctx, name: str):
    return f"Hello {name}!"

async def main():
    pool = await create_pool(RedisSettings())
    job = await pool.enqueue_job('say_hello', 'world')
    result = await job.result()
    print(result)  # Hello world!

asyncio.run(main())

With Key Prefix (Multi-Project)

Namespace all Redis keys with a project prefix:

import asyncio
from arq import create_pool
from arq.connections import RedisSettings

async def main():
    # All keys will be prefixed: scrape_optimus:arq:queue, scrape_optimus:arq:job:*, etc.
    pool = await create_pool(RedisSettings(), key_prefix='scrape_optimus')
    await pool.enqueue_job('my_task', 'arg1')

asyncio.run(main())

Worker with Key Prefix

from arq import Worker
from arq.connections import RedisSettings

async def my_task(ctx, arg):
    return f"processed {arg}"

class WorkerSettings:
    functions = [my_task]
    redis_settings = RedisSettings()
    key_prefix = 'scrape_optimus'  # Must match the pool prefix!

Django Integration

Configure the prefix from Django settings:

# settings.py
PROJECT_NAME = 'my_project'
ARQ_OPTIMUS_KEY_PREFIX = PROJECT_NAME

# arq_worker.py
from django.conf import settings
from arq.connections import RedisSettings

class WorkerSettings:
    functions = [...]
    redis_settings = RedisSettings(host='localhost')
    key_prefix = getattr(settings, 'ARQ_OPTIMUS_KEY_PREFIX', '')

Key Prefix Details

When key_prefix='myproject' is set, all Redis keys are namespaced:

Default Key	Prefixed Key
arq:queue	myproject:arq:queue
arq:job:<id>	myproject:arq:job:<id>
arq:result:<id>	myproject:arq:result:<id>
arq:in-progress:<id>	myproject:arq:in-progress:<id>
arq:abort	myproject:arq:abort
arq:track:running	myproject:arq:track:running
arq:track:completed	myproject:arq:track:completed
arq:track:failed	myproject:arq:track:failed

CLI

Run your worker with the standard arq command:

arq my_worker.WorkerSettings

Dashboard & Job Management CLI

For a real-time monitoring dashboard and a job management CLI, install the companion package:

pip install arq-optimus-dashboard

See arq-optimus-dashboard on PyPI for details.

Compatibility

• Python: 3.9+
• Redis: 5, 6, 7+
• Drop-in replacement: Same import paths as stock ARQ (from arq import ...)

Credits

Based on ARQ by Samuel Colvin. Extended with key-prefix isolation, job tracking, and multi-project support by Arpan Sahu.

License

MIT License — see LICENSE for details.