queutils 0.8.4

Handy Python Queue utilies

Queutils

Queutils [Queue Utils] is a package of handy Python queue classes:

• AsyncQueue - An async wrapper for non-async queue.Queue
• IterableQueue - An AsyncIterable queue that terminates when finished
• FileQueue - Builds an iterable queue of filenames from files/dirs given as input

AsyncQueue

AsyncQueue is a async wrapper for non-async queue.Queue. It can be used to create an asyncio.Queue compatible interface to a (non-async) managed multiprocessing.Queue and thus enable async code in parent/child processes to communicate over multiprocessing.Queue as it were an asyncio.Queue.

Features

• asyncio.Queue compatible
• queue.Queue support
• multiprocessing.Queue support

IterableQueue

IterableQueue is an asyncio.Queue subclass that is AsyncIterable[T] i.e. it can be iterated in async for loop. IterableQueue terminates automatically when the queue has been filled and emptied.

The IterableQueue requires "producers" (functions adding items to the queue) to register themselves and it keeps count of registered producers which are "finished" adding items to the queue. Once all the registered producers are "finished", the queue enters into "filled" state and no new items can be added. Once an "filled" queue is emptied, the queue becomes "done" and all new get() calls to the queue will raise QueueDone exception.

Features

• asyncio.Queue interface, _nowait() methods are experimental
• AsyncIterable support: async for item in queue:
• Automatic termination of the consumers with QueueDone exception when the queue has been emptied
• Producers must be registered with add_producer() and they must notify the queue with finish() once they have finished adding items
• Countable interface to count number of items task_done() through count property
• Countable property can be disabled with count_items=False. This is useful when you want to sum the count of multiple IterableQueues

FileQueue

FileQueue builds a queue (IterableQueue[pathlib.Path]) of the matching files found based on search parameters given. It can search both list of files or directories or mixed. Async method FileQueue.mk_queue() searches subdirectories of given directories.

Features

• Input can be given both as str and pathlib.Path
• exclude: bool exclusive or inclusive filtering. Default is False.
• case_sensitive: bool case sensitive filtering (use of fnmatch or fnmatchcase). Default is True.
• follow_symlinks: bool whether to follow symlinks. Default is False.