Loky logo

Reusable Process Pool Executor

Build Status Documentation Status codecov DOI

Goal

The aim of this project is to provide a robust, cross-platform and cross-version implementation of the ProcessPoolExecutor class of concurrent.futures. It notably features:

  • Consistent and robust spawn behavior: All processes are started using fork + exec on POSIX systems. This ensures safer interactions with third party libraries. On the contrary, multiprocessing.Pool uses fork without exec by default, causing third party runtimes to crash (e.g. OpenMP, macOS Accelerate...).

  • Reusable executor: strategy to avoid re-spawning a complete executor every time. A singleton executor instance can be reused (and dynamically resized if necessary) across consecutive calls to limit spawning and shutdown overhead. The worker processes can be shutdown automatically after a configurable idling timeout to free system resources.

  • Transparent cloudpickle integration: to call interactively defined functions and lambda expressions in parallel. It is also possible to register a custom pickler implementation to handle inter-process communications.

  • No need for if __name__ == "__main__": in scripts: thanks to the use of cloudpickle to call functions defined in the __main__ module, it is not required to protect the code calling parallel functions under Windows.

  • Deadlock free implementation: one of the major concern in standard multiprocessing and concurrent.futures modules is the ability of the Pool/Executor to handle crashes of worker processes. This library intends to fix those possible deadlocks and send back meaningful errors. Note that the implementation of concurrent.futures.ProcessPoolExecutor that comes with Python 3.7+ is as robust as the executor from loky but the latter also works for older versions of Python.

Installation

The recommended way to install loky is with pip,

pip install loky

loky can also be installed from sources using

git clone https://github.com/joblib/loky
cd loky
python setup.py install

Note that loky has an optional dependency on psutil to allow early memory leak detections.

Usage

The basic usage of loky relies on the get_reusable_executor, which internally manages a custom ProcessPoolExecutor object, which is reused or re-spawned depending on the context.

import os
from time import sleep
from loky import get_reusable_executor


def say_hello(k):
    pid = os.getpid()
    print(f"Hello from {pid} with arg {k}")
    sleep(.01)
    return pid


# Create an executor with 4 worker processes that will
# automatically shutdown after idling for 2s
executor = get_reusable_executor(max_workers=4, timeout=2)

res = executor.submit(say_hello, 1)
print("Got results:", res.result())

results = executor.map(say_hello, range(50))
n_workers = len(set(results))
print("Number of used processes:", n_workers)
assert n_workers == 4

For more advanced usage, see our documentation

Workflow to contribute

To contribute to loky, first create an account on github. Once this is done, fork the loky repository to have your own repository, clone it using 'git clone' on the computers where you want to work. Make your changes in your clone, push them to your github account, test them on several computers, and when you are happy with them, send a pull request to the main repository.

Running the test suite

To run the test suite, you need the pytest (version >= 3) and psutil modules. From the root of the project, run the test suite using:

    pip install -e .
    pytest .

Why was the project named loky?

While developing loky, we had some bad experiences trying to debug deadlocks when using multiprocessing.Pool and concurrent.futures.ProcessPoolExecutor, especially when calling functions with non-picklable arguments or returned values at the beginning of the project. When we had to choose a name, we had dealt with so many deadlocks that we wanted some kind of invocation to repel them! Hence loky: a mix of a god, locks and the y that make it somehow cooler and nicer : (and also less likely to result in name conflict in google results ^^).

Fixes to avoid those deadlocks in concurrent.futures were also contributed upstream in Python 3.7+, as a less mystical way to repel the deadlocks :D

Acknowledgement

This work is supported by the Center for Data Science, funded by the IDEX Paris-Saclay, ANR-11-IDEX-0003-02

Project links

Data verified by PyPI on Aug 31, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

4 maintainers

PyPI data
Data sourced directly from PyPI's database.
Avatar for jeremiedbb from gravatar.com jeremiedbb Avatar for ogrisel from gravatar.com ogrisel Avatar for pierreglaser from gravatar.com pierreglaser
+1 more Avatar for tommoral from gravatar.com tommoral

Credits

Author: Thomas Moreau

GitHub Statistics

Data verified by PyPI on Aug 31, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

BSD License (BSD 3-Clause License)

Requires

Python >=3.10

Provides Extra

tests

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

loky-3.6.0.tar.gz (115.3 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

loky-3.6.0-py3-none-any.whl (66.0 kB view details)

Uploaded Python 3

File details

Details for the file loky-3.6.0.tar.gz.

File metadata

  • Download URL: loky-3.6.0.tar.gz
  • Upload date:
  • Size: 115.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for loky-3.6.0.tar.gz
Algorithm Hash digest
SHA256 12c1655148c369dd1739a514a4f5b01b76538ffead916f9ad442ef6c0e9be896
MD5 51f92acb1184ff52ab398f1e0298b9d4
BLAKE2b-256 a622278d945801da206ad49ec30686a9faa140f977afd07143c136252546af0d

See more details on using hashes here.

Provenance

The following attestation bundles were made for loky-3.6.0.tar.gz:

Publisher: publish-to-pypi.yml on joblib/loky

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file loky-3.6.0-py3-none-any.whl.

File metadata

  • Download URL: loky-3.6.0-py3-none-any.whl
  • Upload date:
  • Size: 66.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for loky-3.6.0-py3-none-any.whl
Algorithm Hash digest
SHA256 992365f32e034277a995b170f49549ad7d0ee62d9c91d190ef2c267f8339df17
MD5 ed975b4f08d3237902508fc422c82396
BLAKE2b-256 42f9d9b7f7551d397c1a11b0aea666e8607fc95b7e3ba97b338d83b94e44a51e

See more details on using hashes here.

Provenance

The following attestation bundles were made for loky-3.6.0-py3-none-any.whl:

Publisher: publish-to-pypi.yml on joblib/loky

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

3.6.0 This release

2 files

Pre-release

3.6.0.dev0

2 files

3.5.6

2 files

3.5.5

2 files

3.5.3

2 files

3.5.2

2 files

3.5.1

2 files

3.5.0

2 files

3.4.1

2 files

3.4.0

2 files

3.3.0

2 files

3.2.0

2 files

3.1.0

2 files

3.0.0

2 files

2.9.0

2 files

2.8.0

2 files

2.7.0

2 files

2.6.0

2 files

2.5.1

2 files

2.5.0

2 files

2.4.2

2 files

2.4.1

2 files

2.4.0

2 files

2.3.1

2 files

2.3.0

2 files

2.2.2

2 files

2.2.1

2 files

2.2.0

2 files

2.1.4

2 files

2.1.3

2 files

2.1.2

2 files

2.1.1

2 files

2.1.0

2 files

2.0.0

2 files

1.2.2

2 files

1.2.1

2 files

1.2.0

2 files

1.1.4

2 files

1.1.3

2 files

1.1.2

2 files

1.1.1

2 files

1.1.0

2 files

1.0.0

2 files

0.3.0

2 files

0.2.0

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page