Cluster Futures
This module provides a Python concurrent.futures executor that lets you run functions on remote systems in your HTCondor or Slurm cluster. Stop worrying about writing job files, scattering/gathering, and serialization—this module does it all for you.
It uses the cloudpickle library to allow (most) closures to be used transparently, so you’re not limited to “pure” functions.
Installation:
pip install clusterfutures
Usage:
import cfut
def square(n):
return n * n
with cfut.SlurmExecutor() as executor:
for result in executor.map(square, [5, 7, 11]):
print(result)See slurm_example.py and condor_example.py for further examples. The easiest way to get started is to ignore the fact that futures are being used at all and just use the provided map function, which behaves like itertools.imap but transparently distributes your work across the cluster.
Goals & design
clusterfutures is a simple wrapper to run Python functions in batch jobs on an HPC cluster. Each future corresponds to one batch job. The functions that you run through clusterfutures should normally run for at least a few seconds each: running smaller functions will be inefficient because of the overhead of launching jobs and moving data.
Functions, parameters and return values are sent by creating files; this assumes that the control process and the worker nodes have a shared filesystem. This mechanism is convenient for relatively small amounts of data; it’s probably not the best way to transfer large amounts of data to & from workers.
Release files for clusterfutures 0.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| clusterfutures-0.5.tar.gz | 9.1 kB | Without provenance | Details for clusterfutures-0.5.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| clusterfutures-0.5-py3-none-any.whl | Python 3 | none | any | Details for clusterfutures-0.5-py3-none-any.whl |
Total release size: 18.7 kB
Release files / clusterfutures-0.5.tar.gz
| Download URL | clusterfutures-0.5.tar.gz |
|---|---|
| Size | 9.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
261e82c44b500e39b71e3a2c41db66d9777e9674d58449cb90aa83a1955e9453
|
|
BLAKE2b-256 checksum How to use checksums |
9105667b80f05dfb2b175f8d36f3c737e5ae9cb6e65f0c0a2cea0d655d73c2c8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.0
|
Release files / clusterfutures-0.5-py3-none-any.whl
| Download URL | clusterfutures-0.5-py3-none-any.whl |
|---|---|
| Size | 9.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
391c9258da445366b7e859ac8ed2883aecfd26de364959fc1910e1a7c63bb933
|
|
BLAKE2b-256 checksum How to use checksums |
8a0cb3357f3f7fe40009f24c2aac4ad6ae902071c5eff340c162f4c895751a20
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.0
|
Security for clusterfutures 0.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
