faster-fifo

Faster alternative to Python's standard multiprocessing.Queue (IPC FIFO queue). Up to 30x faster in some configurations.

Implemented in C++ using POSIX mutexes with PTHREAD_PROCESS_SHARED attribute. Based on a circular buffer, low footprint, brokerless. Completely mimics the interface of the standard multiprocessing.Queue, so can be used as a drop-in replacement.

Adds get_many() method to receive multiple messages at once on a consumer for the price of a single lock.

Requirements

Linux or MacOS
Python 3.6 or newer
GCC 4.9.0 or newer

Installation

pip install faster-fifo

Manual build instructions

pip install Cython
python setup.py build_ext --inplace
pip install -e .

Usage example

from faster_fifo import Queue
from queue import Full, Empty

q = Queue(1000 * 1000)  # specify the size of the circular buffer in the ctor

# any pickle-able Python object can be added to the queue
py_obj = dict(a=42, b=33, c=(1, 2, 3), d=[1, 2, 3], e='123', f=b'kkk')
q.put(py_obj)
assert q.qsize() == 1

retrieved = q.get()
assert q.empty()
assert py_obj == retrieved

for i in range(100):
    try:
        q.put(py_obj, timeout=0.1)
    except Full:
        log.debug('Queue is full!')

num_received = 0
while num_received < 100:
    # get multiple messages at once, returns a list of messages for better performance in many-to-few scenarios
    # get_many does not guarantee that all max_messages_to_get will be received on the first call, in fact
    # no such guarantee can be made in multiprocessing systems.
    # get_many() will retrieve as many messages as there are available AND can fit in the pre-allocated memory
    # buffer. The size of the buffer is increased gradually to match demand.
    messages = q.get_many(max_messages_to_get=100)
    num_received += len(messages)

try:
    q.get(timeout=0.1)
    assert True, 'This won\'t be called'
except Empty:
    log.debug('Queue is empty')

Performance comparison (faster-fifo vs multiprocessing.Queue)

System #1 (Intel(R) Core(TM) i9-7900X CPU @ 3.30GHz, 10 cores, Ubuntu 18.04)

(measured execution times in seconds)

	multiprocessing.Queue	faster-fifo, get()	faster-fifo, get_many()
1 producer 1 consumer (200K msgs per producer)	2.54	0.86	0.92
1 producer 10 consumers (200K msgs per producer)	4.00	1.39	1.36
10 producers 1 consumer (100K msgs per producer)	13.19	6.74	0.94
3 producers 20 consumers (100K msgs per producer)	9.30	2.22	2.17
20 producers 3 consumers (50K msgs per producer)	18.62	7.41	0.64
20 producers 20 consumers (50K msgs per producer)	36.51	1.32	3.79

System #2 (Intel(R) Core(TM) i5-4200U CPU @ 1.60GHz, 2 cores, Ubuntu 18.04)

(measured execution times in seconds)

	multiprocessing.Queue	faster-fifo, get()	faster-fifo, get_many()
1 producer 1 consumer (200K msgs per producer)	7.86	2.09	2.2
1 producer 10 consumers (200K msgs per producer)	11.68	4.01	3.88
10 producers 1 consumer (100K msgs per producer)	44.48	16.68	5.98
3 producers 20 consumers (100K msgs per producer)	22.59	7.83	7.49
20 producers 3 consumers (50K msgs per producer)	66.3	22.3	6.35
20 producers 20 consumers (50K msgs per producer)	78.75	14.39	15.78

Footnote

Originally designed for SampleFactory, a high-throughput asynchronous RL codebase https://github.com/alex-petrenko/sample-factory.

Programmed by Aleksei Petrenko and Tushar Kumar at USC RESL.

Developed under MIT License, feel free to use for any purpose, commercial or not, at your own risk ;)

Metadata

Release files for faster-fifo 1.0.9

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for faster-fifo 1.0.9
File	Size	Uploaded	Provenance
faster-fifo-1.0.9.tar.gz	63.3 kB	Mar 30, 2021	Without provenance	Details

Release files / faster-fifo-1.0.9.tar.gz

Download URL	faster-fifo-1.0.9.tar.gz
Size	63.3 kB
Tags	Source
SHA-256 checksum How to use checksums	`db98331255125ed518b390e25855dc3fee68063ca4ef0fcb59a6f01bf12452eb`
BLAKE2b-256 checksum How to use checksums	`23c4defd2f729b5cfdee780ce222f6a1d457905bc7f7bbedca122d6a5b31b5db`
Upload date	Mar 30, 2021
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.6.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.8.5`

Security for faster-fifo 1.0.9

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.