A faster alternative to Python's standard multiprocessing.Queue (IPC FIFO queue)
Project description
faster-fifo
Faster alternative to Python's standard multiprocessing.Queue (IPC FIFO queue). Up to 30x faster in some configurations. (requires Python 3.6 or newer and MacOS or Linux)
Implemented in C++ using POSIX mutexes with PTHREAD_PROCESS_SHARED attribute. Based on a circular buffer, low footprint, brokerless. Completely mimics the interface of the standard multiprocessing.Queue, so can be used as a drop-in replacement.
Adds get_many()
method to receive multiple messages at once on a consumer for the price of a single lock.
Installation
pip install faster-fifo
Manual build instructions
pip install Cython
python setup.py build_ext --inplace
(For MacOS, currently only manual build is available)
Usage example
from faster_fifo import Queue
from queue import Full, Empty
q = Queue(1000 * 1000) # specify the size of the circular buffer in the ctor
# any pickle-able Python object can be added to the queue
py_obj = dict(a=42, b=33, c=(1, 2, 3), d=[1, 2, 3], e='123', f=b'kkk')
q.put(py_obj)
assert q.qsize() == 1
retrieved = q.get()
assert q.empty()
assert py_obj == retrieved
for i in range(100):
try:
q.put(py_obj, timeout=0.1)
except Full:
log.debug('Queue is full!')
num_received = 0
while num_received < 100:
# get multiple messages at once, returns a list of messages for better performance in many-to-few scenarios
# get_many does not guarantee that all max_messages_to_get will be received on the first call, in fact
# no such guarantee can be made in multiprocessing systems.
# get_many() will retrieve as many messages as there are available AND can fit in the pre-allocated memory
# buffer. The size of the buffer is increased gradually to match demand.
messages = q.get_many(max_messages_to_get=100)
num_received += len(messages)
try:
q.get(timeout=0.1)
assert True, 'This won\'t be called'
except Empty:
log.debug('Queue is empty')
Performance comparison (faster-fifo vs multiprocessing.Queue)
System #1 (Intel(R) Core(TM) i9-7900X CPU @ 3.30GHz, 10 cores)
(measured execution times in seconds)
multiprocessing.Queue | faster-fifo, get() | faster-fifo, get_many() | |
---|---|---|---|
1 producer 1 consumer (200K msgs per producer) | 2.54 | 0.86 | 0.92 |
1 producer 10 consumers (200K msgs per producer) | 4.00 | 1.39 | 1.36 |
10 producers 1 consumer (100K msgs per producer) | 13.19 | 6.74 | 0.94 |
3 producers 20 consumers (100K msgs per producer) | 9.30 | 2.22 | 2.17 |
20 producers 3 consumers (50K msgs per producer) | 18.62 | 7.41 | 0.64 |
20 producers 20 consumers (50K msgs per producer) | 36.51 | 1.32 | 3.79 |
System #2 (Intel(R) Core(TM) i5-4200U CPU @ 1.60GHz, 2 cores)
(measured execution times in seconds)
multiprocessing.Queue | faster-fifo, get() | faster-fifo, get_many() | |
---|---|---|---|
1 producer 1 consumer (200K msgs per producer) | 7.86 | 2.09 | 2.2 |
1 producer 10 consumers (200K msgs per producer) | 11.68 | 4.01 | 3.88 |
10 producers 1 consumer (100K msgs per producer) | 44.48 | 16.68 | 5.98 |
3 producers 20 consumers (100K msgs per producer) | 22.59 | 7.83 | 7.49 |
20 producers 3 consumers (50K msgs per producer) | 66.3 | 22.3 | 6.35 |
20 producers 20 consumers (50K msgs per producer) | 78.75 | 14.39 | 15.78 |
Footnote
Originally designed for SampleFactory, a high-throughput asynchronous RL codebase https://github.com/alex-petrenko/sample-factory.
Programmed by Aleksei Petrenko and Tushar Kumar at USC RESL.
Developed under MIT License, feel free to use for any purpose, commercial or not, at your own risk ;)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for faster_fifo-1.0.5-cp38-cp38-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b6f1f853af64b71595670034127aed9f9588b44f416e1901ab9e7fda8694309d |
|
MD5 | 9d007c3a82ae7de5c005e5eaca3ebb4b |
|
BLAKE2b-256 | 9198cbf7fffd46dcb1e582c9bfcf80c8650dfd3a53a22653b51fb6434d7d0154 |
Hashes for faster_fifo-1.0.5-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 899a5e444f98c926792190b721bf50f3743ee2a847b0dea0ffc1b74edca46aee |
|
MD5 | 0ea1038c07f89ba49bcb767c2be2ce88 |
|
BLAKE2b-256 | eab9c0fcd325fc81ac1c989c8d370984e772ed442fc9f2949d1b74de94074e94 |
Hashes for faster_fifo-1.0.5-cp37-cp37m-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d7ed73d935aa5c5231e314ef9680ec0c2c245cef1734a982ab353a24a3d8ebbb |
|
MD5 | fd2f7565937b655822e10d63a901fffa |
|
BLAKE2b-256 | f28828193a99bcd472f4ed9ad4317ba5f80fbfb9392fca09dc3ce99e061a36e2 |
Hashes for faster_fifo-1.0.5-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af62d60d23e89361e2549aa02bf484c220980dd403d1bdc563b57b7bd309092e |
|
MD5 | cb0cc1450078f8d46a99a406f65897a5 |
|
BLAKE2b-256 | 75daed9e5fec10d27bbb9421a9aa3df3d68c465db188a0c4bb615bd1bc23dead |
Hashes for faster_fifo-1.0.5-cp36-cp36m-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7da8b55f8c67804ad476c2a31baf83901ee7496e7593cedbd62b203200710057 |
|
MD5 | 5bb9f86c6620b7a3cb979e622b82c59a |
|
BLAKE2b-256 | 0403a673b70b3106216da32bcce343d7a3c82ac44dd51a83539107c35afb5cde |
Hashes for faster_fifo-1.0.5-cp36-cp36m-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1f38fb77c6e93518d6908df8e808de312faaf2cdee535694660045a70d988314 |
|
MD5 | 446f95d939cec0d3707c545712e598e5 |
|
BLAKE2b-256 | e9cfb342ba0bb954144d01800d75512e927d8747e5cd7c6f0f53c71ec6b293df |