Create asynchronous tasks from Python functions
Project description
ICIJ's async worker library
Installation
With AMQP:
pip install icij-worker["amqp"]
With neo4j:
pip install icij-worker["neo4j"]
Usage
Create an async app and register tasks
Create asynchronous task tailored for long running Python functions:
Given the following pure Python function inside the app.py
module:
def long_running_task(greeted: str) -> str:
greeting = f"Hello {greeted} !"
return greeting
decorate your function with ICIJApp
class and register a new task:
from icij_worker import AsyncApp
my_app = AsyncApp(name="my_app")
@my_app.task
def long_running_task(greeted: str) -> str:
greeting = f"Hello {greeted} !"
return greeting
this will register the long_running_task
function under the long_running_task
task name.
Optionally add progress handlers for a better task monitoring:
@my_app.task
async def long_running_task(
greeted: str,
progress: Optional[Callable[[float], Awaitable]] = None
) -> str:
if progress is not None:
await progress(0.0)
greeting = f"Hello {greeted} !"
if progress is not None:
await progress(100.0)
return greeting
Launch a async worker pool
Start a worker pool using:
icij-worker workers start "app.my_app"
provide worker pool options using:
icij-worker workers start -c worker_config.json -n 2 --backend multiprocessing "app.my_app"
depending on the worker configuration additional setup might be required.
Async worker implementations
Worker asynchronous backends
Implemented
- multiprocessing for CPU bound tasks
To be implemented
- asyncio for I/O bound tasks
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file icij_worker-0.11.11.tar.gz
.
File metadata
- Download URL: icij_worker-0.11.11.tar.gz
- Upload date:
- Size: 56.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.8.4 CPython/3.10.12 Linux/6.5.0-1025-azure
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2887948ce11300c273eabf3e30d8b5a14f5c70fc4da0d76b6ef26893852407d6 |
|
MD5 | a1f620b87662ade8f9241612e9023800 |
|
BLAKE2b-256 | 099f306b42b92898caf5e61ce045f424a3a0a16912a13196bd55a6da5974062c |
File details
Details for the file icij_worker-0.11.11-py3-none-any.whl
.
File metadata
- Download URL: icij_worker-0.11.11-py3-none-any.whl
- Upload date:
- Size: 74.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.8.4 CPython/3.10.12 Linux/6.5.0-1025-azure
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | ba8a988e9bd0c0899933f279bc4e704d0797c1b8aa25a3e28bededee1c6728d1 |
|
MD5 | 947118a6261bdf1b4c3b144a11f0250b |
|
BLAKE2b-256 | 23432b4680127adca64726d1304335406f6ce3dcfe26928883654d416830d04c |