RocksQ

An inproc RocksDB-based queues with Python bindings.

The library is implemented in Rust and PyO3, which allows to release GIL when necessary. The library does not require external dependencies to be installed in the environment.

Persistent queue

A persistent queue with following features:

• max capacity limit in number of elements;
• size calculation based on filesystem space usage;
• length calculation based on number of elements;
• supports only bytes-like objects;
• can operate in a multithreaded environment efficiently (push and pop methods can release GIL if necessary);
• keeps the state between restarts;
• two implementations: blocking and nonblocking;

What is not supported:

• pub/sub is not supported intentionally (implement it on top of RocksQ if necessary);
• TTL is not supported intentionally (implement it on top of RocksQ if necessary).

Implementation details

It works on RocksDB and uses a single column family. The keys are 64-bit integers, the values are byte arrays. The keys are generated by incrementing a counter. The read and write counters are stored in a separate key-value pairs.

MPMC queue

A persistent queue with following features:

• TTL in seconds;
• multiple consumers marked with labels;
• size calculation based on filesystem space usage;
• length calculation based on number of elements;
• supports only bytes-like objects;
• can operate in a multithreaded environment efficiently (add and next methods can release GIL if necessary);
• keeps the state between restarts;
• two implementations: blocking and nonblocking;

Implementation details

It works on RocksDB and uses three column families:

• data

  Stores queue elements. The keys are 64-bit integers, the values are byte arrays. The keys are generated by incrementing a counter.

• system

  Stores a system information like start and write counters, a timestamp of the last write.

• reader

  Stores an information about consumers like read counters, expiration of elements after last reading. The keys are string labels of consumers, the values are binary serialized objects.

TTL is implemented via RocksDB TTL feature. TTL is not strict. It means that the element will remain in the queue for TTL seconds after insertion and the queue will make efforts to remove the element after TTL seconds but it is not guaranteed to be done immediately. Thus, consumers can retrieve expired but not removed elements.

Supported Platforms and Python Versions

Windows: Python versions: 3.7-3.12.

Linux: ManyLinux Python versions: 3.7-3.12. CI does not build for PyPy, but it should work if you build it manually.

MacOS: Currently, I do not have MacOS environment to debug the build process in MacOS, all volunteers are welcome.

Installation

pip install rocksq

Usage

See the examples in the python directory.

API docs are located at: https://insight-platform.github.io/RocksQ/.

Performance

The performance is mostly limited by the throughput of the underlying filesystem. The queue is able to saturate the throughput of the filesystem.