nest-joblib
Documentation: https://nest-joblib.readthedocs.io
Source Code: https://github.com/34j/nest-joblib
Patch joblib to allow nested parallelism
Installation
Install this via pip (or your favourite package manager):
pip install nest-joblib
Usage
from nest_joblib import apply
apply()
With the above code, LokyBackend supports nested-parallelism.
Advanced Usage
The following joblib specification of not doing nested-parallelism may be inefficient in an environment with sufficient memory.
joblib/_parallel_backends.py:
def get_nested_backend(self):
"""Backend instance to be used by nested Parallel calls.
By default a thread-based backend is used for the first level of
nesting. Beyond, switch to sequential backend to avoid spawning too
many threads on the host.
"""
nesting_level = getattr(self, 'nesting_level', 0) + 1
if nesting_level > 1:
return SequentialBackend(nesting_level=nesting_level), None
else:
return ThreadingBackend(nesting_level=nesting_level), None
After calling nest_joblib.apply(), when joblib.parallel.register_parallel_backend(name, backend) is called, a subclass of backend with modified get_nested_backend is dynamically generated and registered with the name f"nested-{name}".
from joblib.parallel import parallel_backend
from nest_joblib import apply
from ray.util.joblib import register_ray
# use LokyBackend
apply()
# use DaskDistributedBackend
apply()
parallel_backend("nested-dask")
# use RayBackend
apply()
register_ray()
parallel_backend("nested-ray")
# use custom backend
from joblib.parallel import LokyBackend
apply()
class MyBackend(LokyBackend):
pass
register_parallel_backend("custom", MyBackend)
parallel_backend("nested-custom")
Contributors ✨
Thanks goes to these wonderful people (emoji key):
34j 💻 🤔 📖 |
This project follows the all-contributors specification. Contributions of any kind welcome!
Metadata
Release files for nest-joblib 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| nest_joblib-1.0.0.tar.gz | 6.4 kB | With provenance | Details for nest_joblib-1.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| nest_joblib-1.0.0-py3-none-any.whl | Python 3 | none | any | Details for nest_joblib-1.0.0-py3-none-any.whl |
Total release size: 12.4 kB
Release files / nest_joblib-1.0.0.tar.gz
| Download URL | nest_joblib-1.0.0.tar.gz |
|---|---|
| Size | 6.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0f994156643bd435a88a1448fd048ddd2b25002535b91e174edd6255b78a206a
|
|
BLAKE2b-256 checksum How to use checksums |
738fe3e21d738c2e4c47907fe893579de42448afd7b4ba5182644d45e590d1a9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.13
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.
Transparency logRelease files / nest_joblib-1.0.0-py3-none-any.whl
| Download URL | nest_joblib-1.0.0-py3-none-any.whl |
|---|---|
| Size | 6.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f6180f661ccd5c3aeef48db5b5dcdbc1e0baa76b40a00c345787ca17d4612c73
|
|
BLAKE2b-256 checksum How to use checksums |
4450a6cee61628459f63cd7b9cd1965a69b76e512005a28857ee8fadc6811aad
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.13
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.
Transparency logSecurity for nest-joblib 1.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for nest_joblib-1.0.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.