Supports async / await pattern for CPU-bound operations.
Project description
Asynchronous CPU
Supports async / await pattern for CPU-bound operations.
Advantage
- Support async / await pattern for CPU-bound operations
- Free from handling event loop
1. Support async / await pattern for CPU-bound operations
The async / await syntax makes asynchronous code as:
- Simple
- Readable
This syntax is not only for I/O-bound but also CPU-bound operations.
This package supports Coroutine
function to run in ProcessPoolExecutor
and returns Awaitable
object.
2. Free from handling event loop
asyncio
is focusing not CPU-bound but I/O-bound operations.
High-level APIs of asyncio
doesn't support CPU-bound operations
since it works based on not ProcessPoolExecutor
but ThreadPoolExecutor
.
When we want to run CPU-bound operations concurrently with asyncio
,
we need to use Low-level APIs which need finer control over the event loop behavior.
Application developers should typically use the High-level asyncio
functions, such as asyncio.run(),
and should rarely need to reference Low-level APIs, such as the Event Loop object or call its methods.
See: Event Loop — Python 3.9.2 documentation
Quickstart
1. Install
pip install asynccpu
2. Implement
This package provides ProcessTaskPoolExecutor
extends ProcessPoolExecutor
,
And its instance has the method: create_process_task()
.
Ex:
from asynccpu import ProcessTaskPoolExecutor
async def process_cpu_bound(task_id):
"""
CPU-bound operations will block the event loop:
in general it is preferable to run them in a process pool.
"""
return f"task_id: {task_id}, result: {sum(i * i for i in range(10 ** 7))}"
with ProcessTaskPoolExecutor(max_workers=3, cancel_tasks_when_shutdown=True) as executor:
awaitables = {executor.create_process_task(process_cpu_bound, x) for x in range(10)}
results = await asyncio.gather(*awaitables)
for result in results:
print(result)
Note
The argument of Coroutine
requires not "raw Coroutine
object" but "Coroutine
function"
since raw Coroutine
object is not picklable.
This specification is depend on the one of Python multiprocessing
package:
multiprocessing — Process-based parallelism
Note When an object is put on a queue, the object is pickled and a background thread later flushes the pickled data to an underlying pipe.
See: Answer: Python multiprocessing PicklingError: Can't pickle <type 'function'> - Stack Overflow
Credits
This package was created with Cookiecutter and the yukihiko-shinoda/cookiecutter-pypackage project template.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.