Skip to main content

Rust persistent queue based on RocksDB with Python bindings

Project description

RocksQ

An inproc RocksDB-based queues with Python bindings.

The library is implemented in Rust and PyO3, which allows to release GIL when necessary. The library does not require external dependencies to be installed in the environment.

Persistent queue

A persistent queue with following features:

  • max capacity limit in number of elements;
  • size calculation based on filesystem space usage;
  • length calculation based on number of elements;
  • supports only bytes-like objects;
  • can operate in a multithreaded environment efficiently (push and pop methods can release GIL if necessary);
  • keeps the state between restarts;
  • two implementations: blocking and nonblocking;

What is not supported:

  • pub/sub is not supported intentionally (implement it on top of RocksQ if necessary);
  • TTL is not supported intentionally (implement it on top of RocksQ if necessary).

Implementation details

It works on RocksDB and uses a single column family. The keys are 64-bit integers, the values are byte arrays. The keys are generated by incrementing a counter. The read and write counters are stored in a separate key-value pairs.

MPMC queue

A persistent queue with following features:

  • TTL in seconds;
  • multiple consumers marked with labels;
  • size calculation based on filesystem space usage;
  • length calculation based on number of elements;
  • supports only bytes-like objects;
  • can operate in a multithreaded environment efficiently (add and next methods can release GIL if necessary);
  • keeps the state between restarts;
  • two implementations: blocking and nonblocking;

Implementation details

It works on RocksDB and uses three column families:

  • data

    Stores queue elements. The keys are 64-bit integers, the values are byte arrays. The keys are generated by incrementing a counter.

  • system

    Stores a system information like start and write counters, a timestamp of the last write.

  • reader

    Stores an information about consumers like read counters, expiration of elements after last reading. The keys are string labels of consumers, the values are binary serialized objects.

TTL is implemented via RocksDB TTL feature. TTL is not strict. It means that the element will remain in the queue for TTL seconds after insertion and the queue will make efforts to remove the element after TTL seconds but it is not guaranteed to be done immediately. Thus, consumers can retrieve expired but not removed elements.

Supported Platforms and Python Versions

Windows: Python versions: 3.7-3.12.

Linux: ManyLinux Python versions: 3.7-3.12. CI does not build for PyPy, but it should work if you build it manually.

MacOS: Currently, I do not have MacOS environment to debug the build process in MacOS, all volunteers are welcome.

Installation

pip install rocksq

Usage

See the examples in the python directory.

API docs are located at: https://insight-platform.github.io/RocksQ/.

Performance

The performance is mostly limited by the throughput of the underlying filesystem. The queue is able to saturate the throughput of the filesystem.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

rocksq-0.3.0-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl (3.6 MB view details)

Uploaded PyPy manylinux: glibc 2.28+ x86-64

rocksq-0.3.0-pp310-pypy310_pp73-manylinux_2_28_aarch64.whl (3.4 MB view details)

Uploaded PyPy manylinux: glibc 2.28+ ARM64

rocksq-0.3.0-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl (3.6 MB view details)

Uploaded PyPy manylinux: glibc 2.28+ x86-64

rocksq-0.3.0-pp39-pypy39_pp73-manylinux_2_28_aarch64.whl (3.4 MB view details)

Uploaded PyPy manylinux: glibc 2.28+ ARM64

rocksq-0.3.0-pp38-pypy38_pp73-manylinux_2_28_x86_64.whl (3.6 MB view details)

Uploaded PyPy manylinux: glibc 2.28+ x86-64

rocksq-0.3.0-pp38-pypy38_pp73-manylinux_2_28_aarch64.whl (3.4 MB view details)

Uploaded PyPy manylinux: glibc 2.28+ ARM64

rocksq-0.3.0-cp312-none-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.12 Windows x86-64

rocksq-0.3.0-cp312-cp312-manylinux_2_28_x86_64.whl (3.6 MB view details)

Uploaded CPython 3.12 manylinux: glibc 2.28+ x86-64

rocksq-0.3.0-cp312-cp312-manylinux_2_28_aarch64.whl (3.4 MB view details)

Uploaded CPython 3.12 manylinux: glibc 2.28+ ARM64

rocksq-0.3.0-cp311-none-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.11 Windows x86-64

rocksq-0.3.0-cp311-cp311-manylinux_2_28_x86_64.whl (3.6 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.28+ x86-64

rocksq-0.3.0-cp311-cp311-manylinux_2_28_aarch64.whl (3.4 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.28+ ARM64

rocksq-0.3.0-cp310-none-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.10 Windows x86-64

rocksq-0.3.0-cp310-cp310-manylinux_2_28_x86_64.whl (3.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.28+ x86-64

rocksq-0.3.0-cp310-cp310-manylinux_2_28_aarch64.whl (3.4 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.28+ ARM64

rocksq-0.3.0-cp39-none-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.9 Windows x86-64

rocksq-0.3.0-cp39-cp39-manylinux_2_28_x86_64.whl (3.6 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.28+ x86-64

rocksq-0.3.0-cp39-cp39-manylinux_2_28_aarch64.whl (3.4 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.28+ ARM64

rocksq-0.3.0-cp38-none-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.8 Windows x86-64

rocksq-0.3.0-cp38-cp38-manylinux_2_28_x86_64.whl (3.6 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.28+ x86-64

rocksq-0.3.0-cp38-cp38-manylinux_2_28_aarch64.whl (3.4 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.28+ ARM64

File details

Details for the file rocksq-0.3.0-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 6306ed65fcee551aecf6e2152721eabde9b4188eed8854c10ed33f6cfc07ade3
MD5 282d47e9a6e4a31b09bf65ce6595bf12
BLAKE2b-256 f1b56537d8847cc39e68f4094de63bfe55a322b493e447167e8779faa1c7075a

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-pp310-pypy310_pp73-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-pp310-pypy310_pp73-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 8701b7764bbaa098ec08f43936c58ff29688b27dc2a3491eaf24b528a620b4f8
MD5 c4cad583554b18ed44a6560d518bb69f
BLAKE2b-256 b5e74b942eb4aef3414b36c0daf287b5f0034e7e4f24e30b7bb03e7098588d41

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 879284e63f3c7746817d6b1b9bbc9765011656b4659a30c623d7930de9e1e50d
MD5 b3626bd7f30c333df2eb831cd68d8236
BLAKE2b-256 ddbb2f3034cafb7c923ca8d1e509be483bd496e8bff0bb081d4a6f19898681c0

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-pp39-pypy39_pp73-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-pp39-pypy39_pp73-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 fcec2053d781abc7225e03f1da8b64d42fc0db4bb38edf0e0ae96cde011be261
MD5 ff16b1ef63e5a3482a4bf81060fe993b
BLAKE2b-256 38830a38a9ff1082f380dd148e8a480528a4ed3c212ccdeb2bea9c826c77135b

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-pp38-pypy38_pp73-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-pp38-pypy38_pp73-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 3bb8c0319dbf49dd37d6ee2145b333f980a02467ba095e15ca5fbb1ad3f5472b
MD5 4e07d7efb4cc7e299ca00fbc1555cf52
BLAKE2b-256 b641df84a6a0e069980a8231b8e50b90ab73d18b9b83e1840559577a85aa6a55

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-pp38-pypy38_pp73-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-pp38-pypy38_pp73-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 bcf603c06175a909938352ec790b8cb73b9d1afeccaef627d75b8dcdafb5f996
MD5 887fb0f872563c943fc163ab18005495
BLAKE2b-256 24cbff1d54b070013b2bf4ec56d675ad8c82b6f53db921bbf894f0bdfff1cade

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-cp312-none-win_amd64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-cp312-none-win_amd64.whl
Algorithm Hash digest
SHA256 d56cd7c07f1e2108e8a6f35e9b707fe747ee0ac9e8f294f6a007a162522e90a8
MD5 92beb065c3a705cbcca732db17d74cc6
BLAKE2b-256 dd6fdca21476c4cdc0d786f8d169320619cf9290dfcbdfe12f52a3285fae72a9

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-cp312-cp312-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-cp312-cp312-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 99fe8b80142a3d1d99e4eb37ed090e4cc75a39bbb6d40948a301e07ea017c182
MD5 669ced814d73f464061e2642318796cd
BLAKE2b-256 7798c2bb8414461dc6aa40581cb7612cba431aa53f771a6bf95bcc218345e384

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-cp312-cp312-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-cp312-cp312-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 ebb4dce97a294e4ef9e1d21f59273c4192bde763ecc3dd46be69c5b230e4527b
MD5 2cdb1b03be0dd79d8ec3acf5a025964b
BLAKE2b-256 301bb5ef8c80cb0becc7d7f23557e4344915382c7144b936d69072ba056edba0

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-cp311-none-win_amd64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-cp311-none-win_amd64.whl
Algorithm Hash digest
SHA256 624981e2490850239ca466e9f52e7cb95656fb7b3e3b94fd5fe630afde3d8d71
MD5 b558173ef6d75176bca73fe2ba5cea1a
BLAKE2b-256 0ec57846d8c82a6b932f7ae7365c4d86fc90b066cbcbc8ede72781af0a5c132b

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-cp311-cp311-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-cp311-cp311-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 2e44f689a8c0bc44aeaf0cb3287c0c50b5630dc115d73dbe3f7c62ec2be11010
MD5 250b24eda40a58e3b31d4b0a79346be2
BLAKE2b-256 9bfa17cdbadb54349b52b98e375d2f9c0a3344fd002df8fc5dd15f2d33b8ebdc

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-cp311-cp311-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-cp311-cp311-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 05311c7e5640398eedf6c330c3caed2b83f9172e35d759df9bf68f55d075ff81
MD5 ed4055ff1c159c0888c6f83a6cd76479
BLAKE2b-256 b9dc3828f6cf393c8e0819da84bf8efad8285dc306b278d53777030fcf2fbc76

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-cp310-none-win_amd64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-cp310-none-win_amd64.whl
Algorithm Hash digest
SHA256 b8669830aa1ad228f23632d4624c534c7b0731688500a27c0b05ca6d27b74e4d
MD5 f6ce5121a5c5c5f974e3e4faef32c1b6
BLAKE2b-256 48ac138a16d315b472d2f6b7bab872bb43d3112b1759e20230297833a605a899

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-cp310-cp310-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-cp310-cp310-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 9e81f6f7374fdc80f746603269602a4865ac3e3599ede73c59970606fe47d6da
MD5 6034f02226ae755368466c73e158640a
BLAKE2b-256 99174d2ccce1a7e0485fcb8d16d6536345620358d54caf7ac097c915df575861

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-cp310-cp310-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-cp310-cp310-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 af4fa38ef070e1e3b949266459cd68a6a4af75a7579f5c89b8a80038b51067be
MD5 a7a7fbf2092349fbeb3eec0d750454a5
BLAKE2b-256 4f048d6eafa9a3dcabcafb9ba6e21cf80115abdb7e16e75efdf79b6fd802c83e

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-cp39-none-win_amd64.whl.

File metadata

  • Download URL: rocksq-0.3.0-cp39-none-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: maturin/1.7.1

File hashes

Hashes for rocksq-0.3.0-cp39-none-win_amd64.whl
Algorithm Hash digest
SHA256 6658af8eaecfeec9cf1eea09e78950fb1c8109bcf87f2c8562a0d708206b1628
MD5 2c1fadc3c69d6b8fa85046476cde4c0b
BLAKE2b-256 baf271e5fef99b4c8ae467375a185a89d257f35314bbe1896b7835e6978ebd24

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-cp39-cp39-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-cp39-cp39-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 3b82090af02cf51a001c01a72af47e61111c232d4fddddb23993da8a9a68e749
MD5 c4d0b0044baa1e689f2592a470be1ba7
BLAKE2b-256 becbc17b79ece097338b601079010ae6100e556568a83dd1f51104c4cdd40063

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-cp39-cp39-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-cp39-cp39-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 7b59daf4c68285a79d5817c79188cbfa01ac3543d02df3de4584f5d68409c63e
MD5 3e8235fe0b73a1f8c737cefac0802a89
BLAKE2b-256 fe44ec546ef76bfd3bfbf6d03f96aa41a463c8c4233190883308208f62eccfb8

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-cp38-none-win_amd64.whl.

File metadata

  • Download URL: rocksq-0.3.0-cp38-none-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: maturin/1.7.1

File hashes

Hashes for rocksq-0.3.0-cp38-none-win_amd64.whl
Algorithm Hash digest
SHA256 8e0579d57151adac99e09e9b1525efc72843a5ae452f2cbe9ade3a5d19f034b5
MD5 e6ce24f68ac5cb492be96c411d9a2990
BLAKE2b-256 3a7fa1789a4046a35e425967dca3e5d39efa423a7515c3320d16063d7828b603

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-cp38-cp38-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-cp38-cp38-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 089b93d2385341fd08e059791292c42c45cb3f2df6d251dfcacb1641ef78e273
MD5 f54e9f0d282319492676ca9bbcc1a0d8
BLAKE2b-256 21eac6c911352eac3d9298a54a7686ce40403df84121e8ae753f6b7c46c5c8a5

See more details on using hashes here.

File details

Details for the file rocksq-0.3.0-cp38-cp38-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for rocksq-0.3.0-cp38-cp38-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 28a49fd386243ea16ed63b28a5cf5a90c84f0c4dd666d913bec0966e0297d058
MD5 692ebca269ea02fc66c1bdadc9de8131
BLAKE2b-256 ba722825b55cd2c4b50070b9c3bfd700fb90c8faecb91e1a830f80aabebbeec2

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page