MyCluster
Library and command line interface to support interacting with multiple HPC clusters
Provides the ability to interact with the most popular HPC job scheduling systems using a single interface and enables the creation of job submission scripts.
Tested with SGE, LSF and slurm (PBS/TORQUE support under development)
Getting started
MyCluster can be installed from PyPi.
pip install mycluster
Configuration
Storing your email address
MyCluster will write your email address into any submission files so you can recieve updates from the schedulers. You can supply this on the command line or store it in a configuration file. To store your email in the configuration file run:
mycluster configure
Setting a custom scheduler
By default MyCluster will try and detect the underlying scheduler but this can be overridden by setting the MYCLUSTER_SCHED environment variable. This should be set to a string name of a Python class that implements the mycluster.schedulers.base.Scheduler class.
Override the submission template
In some cases you may want to override the submission templates, for example if you want to include additional parameters or scheduler commands. To do this set the MYCLUSTER_TEMPLATE environment variable to the jinja template you wish to use. See mycluster/schedulers/templates for the base templates.
Command Line
MyClusyter installs the "mycluster" cli command to interact with the local scheduler via the command line.
Print command help
mycluster <command> --help
List all queues
mycluster queues
List jobs
mycluster list
Create a new submission file, see --help for more submission options.
mycluster create JOBFILE QUEUE RUNSCRIPT
Submit a job file
mycluster submit JOBFILE
Cancel a job
mycluster cancel JOBID
The RUNSCRIPT to be executed by the JOB_SCRIPT can make use of the following predefined environment variables
export NUM_TASKS=
export TASKS_PER_NODE=
export THREADS_PER_TASK=
export NUM_NODES=
# OpenMP configuration
export OMP_NUM_THREADS=$THREADS_PER_TASK
# Default mpiexec commnads for each flavour of mpi
export OMPI_CMD="mpiexec -n $NUM_TASKS -npernode $TASKS_PER_NODE -bysocket -bind-to-socket"
export MVAPICH_CMD="mpiexec -n $NUM_TASKS -ppn $TASKS_PER_NODE -bind-to-socket"
export IMPI_CMD="mpiexec -n $NUM_TASKS -ppn $TASKS_PER_NODE"
API
Mycluster can be used programatically using the mycluster module. All schedulers implement the base mycluster.schedulers.base.Scheduler class.
import mycluster
# Detect the local scheduler
scheduler = mycluster.detect_scheduling_sys()
print(f"Scheduler loaded: {scheduler.scheduler_type()}")
# Create a batch script to submit a 48 task run of script.sh to the skylake queue
script = scheduler.create("skylake", 48, "my_job", "script.sh", "01:00:00", tasks_per_node=24)
# Write to a file
with open("mysub.job", "w") as f:
f.write(script)
# Submit the batch script
job_id = scheduler.submit("mysub.job")
# Check the status of the job
print(scheduler.get_job_details(job_id))
# Cancel the job
scheduler.delete(job_id)
3 maintainers
Metadata
3 maintainers
Release files for mycluster 2.0.9
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| mycluster-2.0.9.tar.gz | 18.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| mycluster-2.0.9-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 53.3 kB
Release files / mycluster-2.0.9.tar.gz
| Download URL | mycluster-2.0.9.tar.gz |
|---|---|
| Size | 18.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a82d92bc95a5c59d228b0ee0925a858cc5d7e0734917643904a4af4c4affa466
|
|
BLAKE2b-256 checksum How to use checksums |
11636cd3dd44c62ec32e55b21129c08553f4ca0e302f24fa48eaac814b8a5da2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.8
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 28, 2025.
Transparency logRelease files / mycluster-2.0.9-py3-none-any.whl
| Download URL | mycluster-2.0.9-py3-none-any.whl |
|---|---|
| Size | 34.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
67409104d68a1e777ea0085d73e44dd4f52bc3606a413f3956b4ef236ce6ee25
|
|
BLAKE2b-256 checksum How to use checksums |
78ba31c8a9f0103f4812fc9a974632debf10974190197b196626db393e932dc3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.8
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 28, 2025.
Transparency log
