Create asynchronous tasks from Python functions
Project description
ICIJ's async worker library
Installation
With AMQP:
pip install icij-worker["amqp"]
With neo4j:
pip install icij-worker["neo4j"]
Usage
Create an async app and register tasks
Create asynchronous task tailored for long running Python functions:
Given the following pure Python function inside the app.py
module:
def long_running_task(greeted: str) -> str:
greeting = f"Hello {greeted} !"
return greeting
decorate your function with ICIJApp
class and register a new task:
from icij_worker import AsyncApp
my_app = AsyncApp(name="my_app")
@my_app.task
def long_running_task(greeted: str) -> str:
greeting = f"Hello {greeted} !"
return greeting
this will register the long_running_task
function under the long_running_task
task name.
Optionally add progress handlers for a better task monitoring:
@my_app.task
async def long_running_task(
greeted: str,
progress: Optional[Callable[[float], Awaitable]] = None
) -> str:
if progress is not None:
await progress(0.0)
greeting = f"Hello {greeted} !"
if progress is not None:
await progress(100.0)
return greeting
Launch a async worker pool
Start a worker pool using:
icij-worker workers start "app.my_app"
provide worker pool options using:
icij-worker workers start -c worker_config.json -n 2 --backend multiprocessing "app.my_app"
depending on the worker configuration additional setup might be required.
Async worker implementations
Worker asynchronous backends
Implemented
- multiprocessing for CPU bound tasks
To be implemented
- asyncio for I/O bound tasks
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
icij_worker-0.11.7.tar.gz
(56.9 kB
view hashes)
Built Distribution
Close
Hashes for icij_worker-0.11.7-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c6ce13d60f03c2fe88869a1552959d1688cc07936dc1200802810d85471e279a |
|
MD5 | c5d774ceca2cc0bd129aebeb1da3f915 |
|
BLAKE2b-256 | 403bb964ac7cbc53f62c585f775df25bc036ec158319a2e753a43ae6b6de355f |