A single process, persistent multi-producer, multi-consumer queue.
The goals is to achieve following requirements:
- Disk-based: each queued item should be stored in disk in case of any crash.
- Thread-safe: can be used by multi-threaded producers and multi-threaded consumers.
- Recoverable: Items can be read after process restart.
- Green-compatible: It can be used in greenlet or eventlet environment.
While queuelib and python-pqueue cannot fulfil all of above. After some try, I found it’s hard achieve based on their current implementation without huge code change. this is the inspiration to start this project.
Besides, persist-queue can serialize any object instances supported by python pickle object serialization module. To support customized objects, please refer to Pickling and unpickling extension types(Python2) and Pickling Class Instances(Python3)
- Python 2.7 or Python3.X
- Fully support for Any POSIX-compatible OS(like Linux, Unix)
pip install persist-queue
from source code
git clone https://github.com/peter-wangxu/persist-queue cd persist-queue python setup.py install
>>> from persistqueue import Queue >>> q = Queue("mypath") >>> q.put('a') >>> q.put('b') >>> q.put('c') >>> q.get() 'a' >>> q.task_done()
Close the python console, and then we restart the queue from the same path,
>>> from persistqueue import Queue >>> q = Queue('mypath') >>> q.get() >>> q.get() 'b' >>> q.task_done()
example usage with multi-thread(referred from github project python-pqueue):
from persistqueue import Queue q = Queue() def worker(): while True: item = q.get() do_work(item) q.task_done() for i in range(num_worker_threads): t = Thread(target=worker) t.daemon = True t.start() for item in source(): q.put(item) q.join() # block until all tasks are done
persist-queue use tox to trigger tests.
to trigger tests based on python2.7/python3.4/python3.5, use:
tox -e py27
tox -e py34
tox -e py35
to trigger pep8 check, use:
tox -e pep8
Simply fork this repo and send PR for your code change(also tests to cover your change), remember to give a title and description of your PR. I am willing to enhance this project with you :).
Apache License Version 2.0
- WindowsError: [Error 183] raised when using persist-queue on Windows.
- It’s a known issue, since Windows cannot overwrite an existing opening file. For now, Please only use it in Linux/Unix/Mac environment.
Release history Release notifications | RSS feed
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size persist-queue-0.1.2.tar.gz (6.0 kB)||File type Source||Python version None||Upload date||Hashes View|