Cluster Tools
This package provides python Executor classes for distributing tasks on a Slurm cluster, Kubernetes, Dask or via multi processing.
Example
import cluster_tools
def square(n):
return n * n
if __name__ == '__main__':
strategy = "slurm" # other valid values are "multiprocessing" and "sequential"
with cluster_tools.get_executor(strategy) as executor:
result = list(executor.map(square, [2, 3, 4]))
assert result == [4, 9, 16]
Installation
The cluster_tools package requires at least Python 3.9.
You can install it from pypi, e.g. via pip:
pip install cluster_tools
By default only the dependencies for running jobs on Slurm and via multiprocessing are installed. For Kubernetes and Dask run:
pip install cluster_tools[kubernetes]
pip install cluster_tools[dask]
Configuration
Slurm
The cluster_tools automatically determine the slurm limit for maximum array job size and split up larger job batches into multiple smaller batches.
Also, the slurm limit for the maximum number of jobs which are allowed to be submitted by a user at the same time is honored by looking up the number of currently submitted jobs and only submitting new batches if they fit within the limit.
If you would like to configure these limits independently, you can do so by setting the SLURM_MAX_ARRAY_SIZE and SLURM_MAX_SUBMIT_JOBS environment variables. You can also limit the maximum number of simultaneously running tasks within the slurm array job(s) by using the SLURM_MAX_RUNNING_SIZE environment variable.
Kubernetes
Resource configuration
| Key | Description | Example |
|---|---|---|
namespace |
Kubernetes namespace for the resources to be created. Will be created if not existent. | cluster-tools |
node_selector |
Which nodes to utilize for the processing. Needs to be a Kubernetes nodeSelector object. |
{"kubernetes.io/hostname": "node001"} |
image |
The docker image for the containerized jobs to run in. The image needs to have the same version of cluster_tools and the code to run installed and in the PYTHONPATH. |
scalableminds/voxelytics:latest |
mounts |
Additional mounts for the containerized jobs. The current working directory and the .cfut directory are automatically mounted. |
["/srv", "/data"] |
cpu |
CPU requirements for this job. | 4 |
memory |
Memory requirements for this job. Not required, but highly recommended to avoid congestion. Without resource requirements, all jobs will be run in parallel and RAM will run out soon. | 16G |
python_executable |
The python executable may differ in the docker image from the one in the current environment. For images based of FROM python, it should be python. Defaults to python. |
python3.8 |
umask |
umask for the jobs. |
0002 |
Notes
- The jobs are run with the current
uid:gid. - The jobs are removed 7 days after completion (successful or not).
- The logs are stored in the
.cfutdirectory. This is actually redundant, because Kubernetes also stores them. - Pods are not restarted upon error.
- Requires Kubernetes ≥ 1.23.
- Kubernetes cluster configuration is expected to be the same as for
kubectl, i.e. in~/.kube/configor similar.
Dev Setup
# See ./dockered-slurm/README.md for troubleshooting
cd dockered-slurm
docker compose up -d
docker exec -it slurmctld bash
docker exec -it c1 bash
Make sure to install all extra dependencies, such as Kubernetes, with uv sync --all-extras.
Tests can be executed with cd tests && uv run pytest -s tests.py after entering the container.
Linting can be run with ./lint.sh.
Code formatting (black) can be run with ./format.sh.
Credits
Thanks to sampsyo/clusterfutures for providing the slurm core abstraction and giovtorres/slurm-docker-cluster for providing the slurm docker environment which we use for CI based testing.
1 maintainer
Metadata
1 maintainer
Release files for cluster_tools 4.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| cluster_tools-4.2.0.tar.gz | 51.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| cluster_tools-4.2.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 102.1 kB
Release files / cluster_tools-4.2.0.tar.gz
| Download URL | cluster_tools-4.2.0.tar.gz |
|---|---|
| Size | 51.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
91abecd0990e032721a2151ee755abd55250479aca3db3f094203f9c017ee2ec
|
|
BLAKE2b-256 checksum How to use checksums |
fe83e8fd2be093fa400e911196b5c491304d819b25f98dafaa699982e56b45c3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / cluster_tools-4.2.0-py3-none-any.whl
| Download URL | cluster_tools-4.2.0-py3-none-any.whl |
|---|---|
| Size | 50.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9b6f5b5afa1d9640ef1605ac1259fb8d6fc259421365dcf99a71cf11e9196380
|
|
BLAKE2b-256 checksum How to use checksums |
d3406d514b0df51638fc3bb29d476a5c6865226e117155eab0cea43e65f9a1a9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
