Skip to main content

trio-parallel: CPU parallelism for Trio

Do you have CPU bound work that just keeps slowing down your Trio event loop no matter what you try? Do you need to get all those cores humming at once? This is the library for you!

The aim of trio-parallel is to use the lightest-weight, lowest-overhead, lowest latency method to achieve CPU parallelism of arbitrary Python code with a dead-simple API.

Resources

License

MIT -or- Apache License 2.0

Documentation

Documentation

Chat

Chatroom

Forum

Forum

Issues

Issues

Repository

Repository

Tests

Tests

Coverage

Test coverage

Style

Code style

Distribution

Latest Pypi version
Supported Python versions
Supported Python interpreters

Example

import multiprocessing
import trio
import trio_parallel
import time


def hard_work(n, x):
    t = time.perf_counter() + n
    y = x
    while time.perf_counter() < t:
        x = not x
    print(y, "transformed into", x)
    return x


async def too_slow():
    await trio_parallel.run_sync(hard_work, 20, False, cancellable=True)


async def amain():
    t0 = time.perf_counter()
    async with trio.open_nursery() as nursery:
        nursery.start_soon(trio_parallel.run_sync, hard_work, 3, True)
        nursery.start_soon(trio_parallel.run_sync, hard_work, 1, False)
        nursery.start_soon(too_slow)
        result = await trio_parallel.run_sync(hard_work, 2, None)
        nursery.cancel_scope.cancel()
    print("got", result, "in", time.perf_counter() - t0, "seconds")


if __name__ == "__main__":
    multiprocessing.freeze_support()
    trio.run(amain)

The full API is documented at https://trio-parallel.readthedocs.io/

Features

  • Bypasses the GIL for CPU bound work

  • Minimal API complexity (looks and feels like Trio threads)

  • Cross-platform

  • Automatic LIFO caching of subprocesses

  • Cancel seriously misbehaving code

    • currently via SIGKILL/TerminateProcess

  • Convert segfaults and other scary things to catchable errors

FAQ

How does trio-parallel run Python code in parallel?

Currently, this project is based on multiprocessing subprocesses and has all the usual multiprocessing caveats (freeze_support, pickleable objects only). The case for basing these workers on multiprocessing is that it keeps a lot of complexity outside of the project while offering a set of quirks that users are likely already familiar with.

Can I have my workers talk to each other?

This is currently possible through the use of multiprocessing.Manager, but we don’t and will not support it. Instead, try using trio.run_process and having the various Trio runs talk to each other over sockets. Also, look into tractor?

Can I let my workers outlive the main Trio process?

The worker processes are started with the daemon flag for lifetime management, so this use case is not supported.

How should I map a function over a collection of arguments?

This is fully possible but we leave the implementation of that up to you. Also, look into trimeter?

Contributing

If you notice any bugs, need any help, or want to contribute any code, GitHub issues and pull requests are very welcome! Please read the code of conduct.

Metadata

Release files for trio-parallel 0.5.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for trio-parallel 0.5.0
File Size Uploaded Provenance Actions
trio-parallel-0.5.0.tar.gz 31.7 kB Without provenance Details for trio-parallel-0.5.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for trio-parallel 0.5.0
File Interpreter ABI Platform Actions
trio_parallel-0.5.0-py3-none-any.whl Python 3 none any Details for trio_parallel-0.5.0-py3-none-any.whl

Total release size: 53.2 kB

Release files / trio-parallel-0.5.0.tar.gz

Download URL trio-parallel-0.5.0.tar.gz
Size 31.7 kB
Tags Source
SHA-256 checksum
How to use checksums
243d9fa536e722be1bd2669524117931e8f6893876dbb1245d51bcc11c668933
BLAKE2b-256 checksum
How to use checksums
ba13220398948f674bdd64b66cc42d9b362a46dfe500c3f6f8f297fcece2fcbd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.9.4

Release files / trio_parallel-0.5.0-py3-none-any.whl

Download URL trio_parallel-0.5.0-py3-none-any.whl
Size 21.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
502fa30935a4da7009734c2075719c28eec6c487130f8d003227d0bd16b79ec7
BLAKE2b-256 checksum
How to use checksums
5f2aab9b4b20a3d3564c0007f54598a61e99950ef3227d994597a4756de1f530
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.9.4

Security for trio-parallel 0.5.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page