Skip to main content

Rust persistent queue based on RocksDB with Python bindings

Project description

RocksQ

An inproc RocksDB-based queue with Python bindings. It is implemented in Rust.

Features:

  • max capacity limit in number of elements;
  • size calculation based on filesystem space usage;
  • length calculation based on number of elements;
  • supports only bytes-like objects;
  • can operate in a multithreaded environment efficiently (push and pop methods can release GIL if necessary);
  • keeps the state between restarts;
  • two implementations: blocking and nonblocking;

What is not supported:

  • pub/sub is not supported intentionally (implement it on top of RocksQ if necessary);
  • TTL is not supported intentionally (implement it on top of RocksQ if necessary).

Implementation details

It works on RocksDB and uses a single column family. The keys are 64-bit integers, the values are byte arrays. The keys are generated by incrementing a counter. The read and write counters are stored in a separate key-value pairs.

It is implemented in Rust and PyO3, thus allows to release GIL when necessary. The library does not require external dependencies installed in the environment.

Supported Platforms and Python Versions

Windows: Python versions: 3.7-3.12.

Linux: ManyLinux Python versions: 3.7-3.12. CI does not build for PyPy, but it should work if you build it manually.

MacOS: Currently, I do not have MacOS environment to debug the build process in MacOS, all volounteers are welcome.

Installation

pip install rocksq

Usage

See the examples in the python directory.

API docs are located at: https://insight-platform.github.io/RocksQ/.

Performance

The performance is mostly limited by the throughput of the underlying filesystem. The queue is able to saturate the throughput of the filesystem.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

rocksq-0.2.0-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl (4.0 MB view hashes)

Uploaded PyPy manylinux: glibc 2.28+ x86-64

rocksq-0.2.0-pp310-pypy310_pp73-manylinux_2_28_aarch64.whl (3.9 MB view hashes)

Uploaded PyPy manylinux: glibc 2.28+ ARM64

rocksq-0.2.0-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl (4.0 MB view hashes)

Uploaded PyPy manylinux: glibc 2.28+ x86-64

rocksq-0.2.0-pp39-pypy39_pp73-manylinux_2_28_aarch64.whl (3.9 MB view hashes)

Uploaded PyPy manylinux: glibc 2.28+ ARM64

rocksq-0.2.0-pp38-pypy38_pp73-manylinux_2_28_x86_64.whl (4.0 MB view hashes)

Uploaded PyPy manylinux: glibc 2.28+ x86-64

rocksq-0.2.0-pp38-pypy38_pp73-manylinux_2_28_aarch64.whl (3.9 MB view hashes)

Uploaded PyPy manylinux: glibc 2.28+ ARM64

rocksq-0.2.0-cp312-none-win_amd64.whl (2.1 MB view hashes)

Uploaded CPython 3.12 Windows x86-64

rocksq-0.2.0-cp312-cp312-manylinux_2_28_x86_64.whl (4.0 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.28+ x86-64

rocksq-0.2.0-cp312-cp312-manylinux_2_28_aarch64.whl (3.9 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.28+ ARM64

rocksq-0.2.0-cp311-none-win_amd64.whl (2.0 MB view hashes)

Uploaded CPython 3.11 Windows x86-64

rocksq-0.2.0-cp311-cp311-manylinux_2_28_x86_64.whl (4.0 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.28+ x86-64

rocksq-0.2.0-cp311-cp311-manylinux_2_28_aarch64.whl (3.9 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.28+ ARM64

rocksq-0.2.0-cp310-none-win_amd64.whl (2.0 MB view hashes)

Uploaded CPython 3.10 Windows x86-64

rocksq-0.2.0-cp310-cp310-manylinux_2_28_x86_64.whl (4.0 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.28+ x86-64

rocksq-0.2.0-cp310-cp310-manylinux_2_28_aarch64.whl (3.9 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.28+ ARM64

rocksq-0.2.0-cp39-none-win_amd64.whl (2.1 MB view hashes)

Uploaded CPython 3.9 Windows x86-64

rocksq-0.2.0-cp39-cp39-manylinux_2_28_x86_64.whl (4.0 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.28+ x86-64

rocksq-0.2.0-cp39-cp39-manylinux_2_28_aarch64.whl (3.9 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.28+ ARM64

rocksq-0.2.0-cp38-none-win_amd64.whl (2.0 MB view hashes)

Uploaded CPython 3.8 Windows x86-64

rocksq-0.2.0-cp38-cp38-manylinux_2_28_x86_64.whl (4.0 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.28+ x86-64

rocksq-0.2.0-cp38-cp38-manylinux_2_28_aarch64.whl (3.9 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.28+ ARM64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page