A thread-safe queue object which is interchangeable with the stdlib Queue. Any overflow goes into a compressed file to keep excessive amounts of queued items out of memory
Project description
Overview
filequeue is a Python library that provides a thread-safe queue which is a subclass of Queue.Queue from the stdlib.
filequeue.FileQueue will overflow into a compressed file if the number of items exceeds maxsize, instead of blocking or raising Full like the regular Queue.Queue.
There is also a filequeue.PriorityFileQueue and there will hopefully also be a LIFO Queue if I can figure out a nice way of implementing it.
Note filequeue.FileQueue will only behave the same as Queue.Queue if it is initialised with maxsize=0 (the default). See __init__ docstring for details (help(FileQueue))
Note filequeue.PriorityFileQueue won’t currently work exactly the same as a straight out replacement for Queue.PriorityQueue. The interface is very slightly different (extra optional kw argument on put and __init__), although it will work it won’t behave the same. It might still be useful to people though and hopefully I’ll be able to resolve this in a future version.
Requirements:
Python 2.5+ or Python 3.x
Why?
The motivation came from wanting to queue a lot of work, without consuming lots of memory.
The interface of filequeue.FileQueue matches that of Queue.Queue (or queue.Queue in python 3.x). With the idea being that most people will use Queue.Queue, and can swap in a filequeue.FileQueue only if the memory usage becomes an issue.
Licence
Made available as-is under the BSD Licence.
Issues
Any issues please post on the github page.
Changelog
0.2.3 (2012-11-27)
Fix for PriorityFileQueue where it wasn’t returning items in the correct order according to the priority.
Added import * into __init__.py to make the namespace a bit nicer.
Added the unit tests from stdlibs Queue (quickly edited out the full checks and LifoQueue tests)
0.2.2 (2012-11-27)
Initial public release.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.