Simple language-agnostic message queues: tools, conventions, examples
Project description
rfq
Simple language-agnostic message queues: tools, conventions, examples
Table of Contents
Overview
We are using redis as a message queue to communicate between services, backends, workers. Implementing a reliable message queue with redis is possible but has to follow certain best practices.
The goal of this project is to provide a simple reliable message queue on top of redis while following best practices and capturing conventions as code. See rfq.js for a simple Javascript/Typescript integration.
Usage
The following shows library functionality in terms of their thin command line wrappers.
The command line tool and library can be configured by setting the environment variables
export RFQ_REDIS_HOST=localhost
export RFQ_REDIS_PORT=6397
for the default redis host and port when not providing a custom redis instance.
For development
make
make run
$ rfq --help
$ exit
make down
Run the commands directly from the docker container with
docker run -e RFQ_REDIS_HOST=MyRedisHost robofarm/rfq --help
list-topics
List all active topics
rfq list-topics
ndvi
list-queue
List messages in the backlog (work not yet started)
rfq list-queue --topic 'ndvi'
eeba0c1642ab11eaa480a4c3f0958f5d
ede1296442ab11eabdb9a4c3f0958f5d
List messages in the nextlog (work started but not yet committed)
rfq list-queue--topic 'ndvi' --queue nextlog
eeba0c1642ab11eaa480a4c3f0958f5d
purge-queue
Deletes messages in a queue
rfq purge-queue --topic 'ndvi'
1
publish
Publish messages to a topic
rfq publish --topic 'ndvi' --message '{ "tile": "T32UNE" }'
eeba0c1642ab11eaa480a4c3f0958f5d
rfq publish --topic 'ndvi' --message '{ "tile": "T33UVU" }'
ede1296442ab11eabdb9a4c3f0958f5d
Note: messages and are a flat map of key-value pairs, see issue/1
consume
Consume a message, working on it (dummy sleep), then commit the message when done
rfq consume --topic 'ndvi'
{ "tile": "T32UNE" }
harvest
Harvest messages from the nextlog back into the backlog
rfq harvest --topic 'ndvi'
eeba0c1642ab11eaa480a4c3f0958f5d
Design
We provide a reliable FIFO-queue on top of redis' datastructures.
-
Namespace: by default redis provides a global namespace. To distinguish rfq specific keys we prefix them with
rfq:
-
Topics: to group messages we provide the concept of topics. We encode a topic
mytopic
as key prefixrfq:mytopic:
-
Messages: the content of a message is stored as a hash map at
rfq:mytopic:message:uuid
using a uuid -
Backlog: the backlog is a list at
rfq:mytopic:backlog
onto which producers append message identifiers to -
Nextlog: when consumers pop a message from the backlog, they append it to the next list at
rfq:mytopic:nextlog
. Only after successfully completing a message, the worker commits the message by removing it from the next queue -
Harvester: in case of crashing workers, the harvester checks the next list and puts messages back into the backlog
Notes:
- Use uuid version 1 as message identifiers; they are approximately globally sortable by their timestamp bits
- Use alphanumerical topics in
[a-z0-9]+
for simplicity and portability
Protocol
The following describes the low level protocol and conventions used in terms of redis commands.
Producer 1
hmset rfq:mytopic:message:fe84ff90428611eabb02a4c3f0958f5d v 1 op tile
lpush rfq:mytopic:backlog fe84ff90428611eabb02a4c3f0958f5d
Producer 2
hmset rfq:mytopic:message:d97acb20428711eabb02a4c3f0958f5d v 1 op ndvi
lpush rfq:mytopic:backlog d97acb20428711eabb02a4c3f0958f5d
Consumer 1
rpoplpush rfq:mytopic:backlog rfq:mytopic:nextlog
lrem rfq:mytopic:nextlog 0 fe84ff90428611eabb02a4c3f0958f5d
del rfq:mytopic:message:d97acb20428711eabb02a4c3f0958f5d
Consumer 2
rpoplpush rfq:mytopic:backlog rfq:mytopic:nextlog
# crashes, never commits d97acb20428711eabb02a4c3f0958f5d
Harvester
lrange rfq:mytopic:nextlog 0 -1
lpush rfq:mytopic:backlog d97acb20428711eabb02a4c3f0958f5d
lrem rfq:mytopic:nextlog 0 d97acb20428711eabb02a4c3f0958f5d
License
Copyright © 2020 robofarm
Distributed under the MIT License (MIT).
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.