In-memory FIFO queue for concurrent task execution
Project description
async-task-queue
In-memory FIFO queue for concurrent task execution. Used to execute tasks concurrently with optional control (via semaphore) over the max number of tasks running at the same time.
Features:
- Queue processing summary logging
- Introspection of failed, retried, and succeeded tasks
- Task retries (optional)
- Task execution timeout (optional)
- Queue processing with semaphore (optional)
- Batch size control (optional)
TOC:
Installation
async-task-queue requires Python 3.6 or above.
pip install async-task-queue
Guide
import logging
from async_task_queue import AsyncTask, AsyncTaskQueue
# Initialize a logger
logger = logging.getLogger("foo")
# Initialize an AsyncTaskQueue where:
# - At most 5 tasks are running concurrently
# - Number of tasks executing concurrently should be limited by a semaphore
# - Failed tasks should be retried (default behavior)
# - Executing the tasks queued should timeout and be cancelled after 5 minutes
task_queue = AsyncTaskQueue(
logger,
use_semaphore=True,
batch_size=5,
execution_timeout=300
)
# Add async tasks to the queue
task_queue.enqueue(
[
AsyncTask(some_coroutine, *args, **kwargs) for args, kwargs in some_args_kwargs
]
)
# Start processing the queue
await task_queue.execute()
Development
To develop async-task-queue, install dependencies and enable the pre-commit hook:
pip install pre-commit tox
pre-commit install
To run tests:
tox
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
async-task-queue-0.1.2.tar.gz
(8.8 kB
view hashes)
Built Distribution
Close
Hashes for async_task_queue-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6245a869bef4ae7fbc2db8b46cb8d854eea4feb69fe0f6405e90fcfc64a96282 |
|
MD5 | 89b4100a7dc42cd4c1bbf9405dd13a67 |
|
BLAKE2b-256 | 9364f86b640d834083378907ab5cb3f98da6ea9c555f4fc7d6ceff594fe5d321 |