Robust Async Task Runner
Overview
This project provides a Python class, ResilientTaskRunner, designed to execute asynchronous tasks with enhanced robustness features. It manages concurrency, handles retries with exponential backoff and jitter, implements dynamic attempt timeouts based on historical performance, and provides detailed success/failure reporting.
Features
- Concurrency Control: Limits the number of tasks running simultaneously using
asyncio.Semaphore. - Automatic Retries: Automatically retries failed tasks up to a configurable maximum number of attempts.
- Exponential Backoff & Jitter: Implements exponential backoff for retry delays, with added jitter to prevent thundering herd issues.
- Attempt Timeouts: Each execution attempt has a timeout.
- Dynamic Timeouts: Optionally adjusts attempt timeouts based on the performance (execution duration) of recently completed successful tasks. Uses a percentile of historical durations multiplied by a factor.
- Timeout Backoff: The timeout duration for the next attempt is doubled after any failure (timeout or exception) within the current attempt.
- Detailed Results: Returns separate lists of success and failure objects, including duration, attempts made, results, or error details.
- Clean Logging: Uses Python's
loggingmodule with task-specific identifiers.
Configuration Parameters
The ResilientTaskRunner class is initialized with the following parameters:
max_concurrency(int, default: 10): Maximum number of tasks to run concurrently.max_attempts(int, default: 3): Total maximum attempts per task (1 initial +max_attempts - 1retries).retry_delay_seconds(float, default: 1.0): Base delay in seconds before the first retry. Subsequent retries use exponential backoff (delay * 2^attempt_index).default_attempt_timeout_seconds(Optional[float], default: 30.0): The timeout for each individual task attempt if dynamic timeouts are disabled or haven't gathered enough data.enable_dynamic_timeouts(bool, default: True): IfTrue, attempts to calculate attempt timeouts based on historical execution times.min_completed_for_stats(int, default: 5): Minimum number of successful tasks needed before dynamic timeouts can be calculated.completed_duration_history_size(int, default: 50): Maximum number of recent successful task durations to keep for calculating dynamic timeouts.straggler_percentile(float, default: 90.0): The percentile of successful task durations used as the base for dynamic timeout calculation.straggler_factor(float, default: 2.5): The multiplier applied to the percentile duration to get the dynamic attempt timeout.
Usage
- Import:
from main import ResilientTaskRunner, asyncio - Define Async Tasks: Create your asynchronous functions that the runner will execute.
- Instantiate Runner: Create an instance of
ResilientTaskRunner, optionally overriding default configuration parameters.runner = ResilientTaskRunner( max_concurrency=5, max_attempts=4, default_attempt_timeout_seconds=10.0 )
- Add Tasks: Use the
add_taskmethod to add tasks to the runner's queue.async def my_task(arg1, arg2): # ... do async work ... await asyncio.sleep(1) return f"Result: {arg1 + arg2}" runner.add_task(my_task, 10, 20) runner.add_task(another_async_func, arg_for_it) # ... add more tasks
- Run: Call the
runmethod to start processing the tasks. This is an async method.async def main(): # ... instantiate runner and add tasks ... successes, failures = await runner.run() print("--- Successes ---") for s in successes: print(f" Task {s.task_id}: Result='{s.result}', Duration={s.duration:.2f}s, Attempts={s.total_attempts_made}") print("--- Failures ---") # ... print failure details ... if __name__ == "__main__": asyncio.run(main())
- Execute: Run the script containing your
mainfunction.python main.py(Oruv run python main.pyif usinguv)
Dependencies
This script requires only standard Python 3 libraries (asyncio, logging, time, random, statistics, collections, typing, dataclasses).
If using the provided pyproject.toml and uv.lock, you can manage the environment with uv.
# Sync environment (optional, if using uv)
uv sync
# Run the example
uv run python main.py
Metadata
Release files for resilient-runner 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| resilient_runner-0.1.1.tar.gz | 10.2 kB | Without provenance | Details for resilient_runner-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| resilient_runner-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for resilient_runner-0.1.1-py3-none-any.whl |
Total release size: 19.2 kB
Release files / resilient_runner-0.1.1.tar.gz
| Download URL | resilient_runner-0.1.1.tar.gz |
|---|---|
| Size | 10.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
048af364a75574324065ae290013ef0124d5e77105a4de0058dc38092c0c4a72
|
|
BLAKE2b-256 checksum How to use checksums |
85be7e7781f1a4ec5bc11af5e9d2da1a609ea4bbe5d2aaae02d89eee0a5d4541
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.6.17
|
Release files / resilient_runner-0.1.1-py3-none-any.whl
| Download URL | resilient_runner-0.1.1-py3-none-any.whl |
|---|---|
| Size | 9.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2c1d024316380c6634e480d132b2fa3bc610d9e5c83d4bf520459034f415875d
|
|
BLAKE2b-256 checksum How to use checksums |
3bcf7dab6320c5e11128ed9f2af01c70576c77b01bcfa99261d85b19a12f7506
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.6.17
|
Security for resilient-runner 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.