An async task queue with live progress display
Project description
aqueue is an async task queue with live progress display.
You put items in, and they get processed, possibly creating more items which get processed, and so on, until all items are completed. A typical use case would be to scrape a website.
Installation
aqueue is a Python package hosted on PyPI. The recommended installation method is pip-installing into a virtual environment:
pip install aqueue
Getting Started
There’s two things you need to do to use aqueue:
Implement your Item subclasses.
Start your queue with one of those items.
Example
If you had a hierarchy of items like this…
Then, you might process it with aqueue like this…
import aqueue
class RootItem(aqueue.Item):
async def process(
self, enqueue: aqueue.EnqueueFn, set_desc: aqueue.SetDescFn
) -> None:
# display what we're doing in the worker status panel
set_desc("Processing RootItem")
# make an HTTP request, parse it, etc
...
# when you discover more items you want to process, enqueue them:
for _ in range(5):
enqueue(ChildItem())
async def after_children_processed(self) -> None:
# run this method when this Item and all other Items it enqueued are done
print("All done!")
class ChildItem(aqueue.Item):
# track the enqueueing and completion of these items in the overall panel
track_overall: bool = True
async def process(
self, enqueue: aqueue.EnqueueFn, set_desc: aqueue.SetDescFn
) -> None:
set_desc("Processing ChildItem")
if __name__ == "__main__":
aqueue.run_queue(
initial_items=[RootItem()],
num_workers=2,
)
Project Information
License: MIT
Source Code: https://github.com/t-mart/aqueue
Documentation: https://t-mart.github.io/aqueue/
Supported Python Versions: 3.10 and later
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.