redispatcher

Dispatch and run distributed work asynchronously, brokered by Redis

Documentation: https://rafalstapinski.github.io/redispatcher

Source Code: https://github.com/rafalstapinski/redispatcher

What is redispatcher

redispatcher allows you to dispatch work that needs to be done in a process separate from your main program loop. This is useful in cases when you need to process some long running work that doesn't necessarily need to be done synchronously within your code. A classic example of this is sending a welcome email to a user as they sign up for your service. It's not necessary to wait for the results of sending an email, and it may take a few seconds to do so. redispatcher lets you fire-and-forget this work (as a message put into a Redis server) and have it be executed in the background, asynchronously, in a separate process (or machine) entirely.

redispatcher comes in two parts:

1. A library that lets you define workers, define strongly typed messages sent to workers, and provides helper functions to facilitate dispatching that work
2. A daemon that runs your workers in a pool, consistently listening for any incoming messages to be processed

Why use it

There are certainly other solutions for orchestrating distributed workers. redispatcher aims to be super lightweight, very fast and simple to set up (there are many free cloud-hosted Redis solutions available), and has robust type validation and intellisense support.

Features

• Full intellisense support across your code, despite a distributed workload
• Strongly typed message contract between your publishing code and consumer
• Minimal boilerplate required to setup and start publishing compared than most alternatives
• Minimal performance overhead and completely non-blocking, thanks to asyncio (and works with alternatives like uvloop)

Dependencies

• aioredis is used under the hood to publish message to and read messages from Redis
• pydantic is used to to validate messages conform to your strongly typed contracts

Installation

Install with poetry

$ poetry add redispatcher

or with pip

$ pip install redispatcher

Basic Usage

Defining your worker

from redispatcher import BaseConsumer

class SendWelcomeEmail(BaseConsumer):

    QUEUE = "send-welcome-email"

    class Message(BaseConsumer.Message):
        email: str
        name: str
    
    async def process_message(self, message: Message):
        # construct an email and send it to the `message.email` address

Dispatching work

from clients import my_aioredis_client

@app.post("/register")
async def register(...)
    ...
    message = SendWelcomeEmail.Message(email=..., name=..., registered=True)
    await SendWelcomeEmail.dispatch(message, my_aioredis_client)
    ...

Running redispatcher

from redispatcher import Redispatcher, RedispatcherConfig, ConsumerConfig

config = RedispatcherConfig(
    redis_dsn="redis://localhost:6379/0",
    consumers=[
        ConsumerConfig(
            consumer_class=SendWelcomeEmail,
            count=2
        )
    ]
)

if __name__ == "__main__":
    dispatcher = Redispatcher(config)
    dispatcher.start() 

Contributing

redispatcher is already used in production, but is still in its infancy.

If you find a bug, open an issue with a detailed description and steps to reproduce.

If you're looking for a feature, open an issue with a detailed description and use case. Feel free open a pull request if you want to contribure directly!