trio-parallel
Welcome to trio-parallel!
CPU parallelism for Trio
License: Your choice of MIT or Apache License 2.0
Do you have CPU bound work that just keeps slowing down your event loop no matter what you try? Do you need to get all those cores humming at once? This is the library for you!
import multiprocessing
import trio
import trio_parallel
import time
def hard_work(x):
t = time.perf_counter() + 3
while time.perf_counter() < t:
x = not x
print(x, "transformed into", not x)
return not x
async def amain():
t0 = time.perf_counter()
async with trio.open_nursery() as nursery:
nursery.start_soon(trio_parallel.run_sync, hard_work, True)
nursery.start_soon(trio_parallel.run_sync, hard_work, False)
result = await trio_parallel.run_sync(hard_work, None)
print("got", result, "in", time.perf_counter() - t0, "seconds")
if __name__ == "__main__":
multiprocessing.freeze_support()
trio.run(amain)Features
Bypasses the GIL for CPU bound work
Minimal API complexity (looks and feels like Trio threads)
Cross-platform
Automatic LIFO caching of subprocesses
Cancel seriously misbehaving code
currently via SIGKILL/TerminateProcess
Convert segfaults and other scary things to catchable errors
This project aims to use the lightest-weight, lowest-overhead, lowest latency method to achieve CPU parallelism of arbitrary Python code. At the moment, that means subprocesses. However, this project is not at all constrained by that, and will be considering subinterpreters, or any other avenue as they become available.
Currently, this project is based on multiprocessing has all the usual multiprocessing caveats (freeze_support, pickleable objects only). The case for basing these workers on multiprocessing is that it keeps a lot of complexity outside of the project while offering a set of quirks that users are likely already familiar with.
FAQ
I want my workers to talk to each other
This is currently possible through the use of multiprocessing.Manager, but we don’t and will not support it. Instead, try using trio.run_process and having the various Trio runs talk to each other over sockets. Also, look into tractor?
I want my workers to outlive the main Trio process
The worker processes are started with the daemon flag for lifetime management, so this use case is not supported.
I want to map a function over a collection of arguments
This is fully possible but we leave the implementation of that up to you. Also, look into trimeter?
1 maintainer
Metadata
1 maintainer
Release files for trio-parallel 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| trio-parallel-0.1.0.tar.gz | 25.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| trio_parallel-0.1.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 44.2 kB
Release files / trio-parallel-0.1.0.tar.gz
| Download URL | trio-parallel-0.1.0.tar.gz |
|---|---|
| Size | 25.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e53a4b21496dab2a50b0b863831290824f80ba73797147a9a23e6bbcc2083dbb
|
|
BLAKE2b-256 checksum How to use checksums |
1e21f2cd570e6c73c5fb1be87a7ea6bcabcd1e4821042ba5571db6a0acd1c187
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/52.0.0.post20210125 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.8.5
|
Release files / trio_parallel-0.1.0-py3-none-any.whl
| Download URL | trio_parallel-0.1.0-py3-none-any.whl |
|---|---|
| Size | 19.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
458bad25471794615bf59964e4acab482c1ca59ced06ca94da71dd22391e27e2
|
|
BLAKE2b-256 checksum How to use checksums |
fec154bb724188d4aa91f564a0f423cdedd878dc41f940235ef6c003f2cea88f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/52.0.0.post20210125 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.8.5
|
