Skip to main content

nest-joblib

CI Status Documentation Status

Test coverage percentage uv Ruff prek

PyPI Version Supported Python versions License


Documentation: https://nest-joblib.readthedocs.io

Source Code: https://github.com/34j/nest-joblib


Patch joblib to allow nested parallelism

Installation

Install this via pip (or your favourite package manager):

pip install nest-joblib

Usage

from nest_joblib import apply

apply()

With the above code, LokyBackend supports nested-parallelism.

Advanced Usage

The following joblib specification of not doing nested-parallelism may be inefficient in an environment with sufficient memory.

joblib/_parallel_backends.py:

    def get_nested_backend(self):
        """Backend instance to be used by nested Parallel calls.

        By default a thread-based backend is used for the first level of
        nesting. Beyond, switch to sequential backend to avoid spawning too
        many threads on the host.
        """
        nesting_level = getattr(self, 'nesting_level', 0) + 1
        if nesting_level > 1:
            return SequentialBackend(nesting_level=nesting_level), None
        else:
            return ThreadingBackend(nesting_level=nesting_level), None

After calling nest_joblib.apply(), when joblib.parallel.register_parallel_backend(name, backend) is called, a subclass of backend with modified get_nested_backend is dynamically generated and registered with the name f"nested-{name}".

from joblib.parallel import parallel_backend
from nest_joblib import apply
from ray.util.joblib import register_ray

# use LokyBackend
apply()

# use DaskDistributedBackend
apply()
parallel_backend("nested-dask")

# use RayBackend
apply()
register_ray()
parallel_backend("nested-ray")

# use custom backend
from joblib.parallel import LokyBackend
apply()
class MyBackend(LokyBackend):
    pass
register_parallel_backend("custom", MyBackend)
parallel_backend("nested-custom")

Contributors ✨

Thanks goes to these wonderful people (emoji key):

34j
34j

💻 🤔 📖

This project follows the all-contributors specification. Contributions of any kind welcome!

Metadata

Release files for nest-joblib 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for nest-joblib 1.0.0
File Size Uploaded Provenance Actions
nest_joblib-1.0.0.tar.gz 6.4 kB With provenance Details for nest_joblib-1.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for nest-joblib 1.0.0
File Interpreter ABI Platform Actions
nest_joblib-1.0.0-py3-none-any.whl Python 3 none any Details for nest_joblib-1.0.0-py3-none-any.whl

Total release size: 12.4 kB

Release files / nest_joblib-1.0.0.tar.gz

Download URL nest_joblib-1.0.0.tar.gz
Size 6.4 kB
Tags Source
SHA-256 checksum
How to use checksums
0f994156643bd435a88a1448fd048ddd2b25002535b91e174edd6255b78a206a
BLAKE2b-256 checksum
How to use checksums
738fe3e21d738c2e4c47907fe893579de42448afd7b4ba5182644d45e590d1a9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.13

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.

Transparency log

Release files / nest_joblib-1.0.0-py3-none-any.whl

Download URL nest_joblib-1.0.0-py3-none-any.whl
Size 6.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f6180f661ccd5c3aeef48db5b5dcdbc1e0baa76b40a00c345787ca17d4612c73
BLAKE2b-256 checksum
How to use checksums
4450a6cee61628459f63cd7b9cd1965a69b76e512005a28857ee8fadc6811aad
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.13

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.

Transparency log

Security for nest-joblib 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for nest_joblib-1.0.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page