Node Vetting for Distributed Workloads
Ensure allocated nodes are vetted before executing a distributed workload through a series of configurable sanity checks. These checks are designed to detect highly dynamic issues (e.g., GPU temperature) and should be performed immediately before executing the main distributed job.
Features
- ⚡ Fast and lightweight
- 🛠️ Modular and configurable
- 🚀 Easy to extend
Getting Started
# Install
pip install vetnode
# checks for dependencies and installs requirements
vetnode setup ./examples/local-test/config.yaml
# runs the vetting process
vetnode diagnose ./examples/local-test/config.yaml
Workflow Usage Example
The vetnode cli is intended to be embedded into your HPC workflow. The following is a node vetting example for a ML (machine learning) workflow on a Slurm HPC cluster.
#!/bin/bash
#SBATCH --nodes=6
#SBATCH --time=0-00:15:00
#SBATCH --account=a-csstaff
REQUIRED_NODES=4
MAIN_JOB_COMMAND="python -m torch.distributed.torchrun --nproc_per_node=$(wc -l < vetted-nodes.txt) main.py"
vetnode setup ../examples/slurm-job-with-vetting/config.yaml
srun vetnode diagnose ../examples/slurm-job-with-vetting/config.yaml >> results.txt
# Extract node lists
grep '^Cordon:' results.txt | awk '{print $2}' > cordoned-nodes.txt
grep '^Vetted:' results.txt | awk '{print $2}' > vetted-nodes.txt
#Run on healthy nodes only
if [ $(wc -l < vetted-nodes.txt) -ge $REQUIRED_NODES ]; then
srun -N $REQUIRED_NODES --exclude=./cordoned-nodes.txt $MAIN_JOB_COMMAND
else
echo "Job canceled!"
echo "Reason: too few vetted nodes."
fi
Quick Run
The following is a Slurm job example you can download and run as a test.
curl -o job.sh https://raw.githubusercontent.com/theely/vetnode/refs/heads/main/examples/slurm-ml-vetting/job.sh
sbatch --account=a-csstaff job.sh
#check job status
squeue -j {jobid} --long
#check vetting results
cat vetnode-{jobid}/results.txt
Development
Set-up Python Virtual environement
Create a virtual environment:
python3.11 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
pip install -r requirements.txt
Run the CLI
cd src
python -m vetnode diagnose ../examples/local-test/config.yaml
Running Tests
From the FirecREST root folder run pytest to execute all unit tests.
source .venv/bin/activate
pip install -r ./requirements.txt -r ./requirements-testing.txt
pytest
Distribute
pip install -r ./requirements-testing.txt
python3 -m build --wheel
twine upload dist/*
Metadata
Release files for VetNode 0.0.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| vetnode-0.0.3-py3-none-any.whl | Python 3 | none | any | Details for vetnode-0.0.3-py3-none-any.whl |
Release files / vetnode-0.0.3-py3-none-any.whl
| Download URL | vetnode-0.0.3-py3-none-any.whl |
|---|---|
| Size | 11.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
878f8121c35af0adeba4acf6d07b6b0a0598ac257d83689d8e711dc65d10fba9
|
|
BLAKE2b-256 checksum How to use checksums |
834d210bea71bc85fc266f2604d9e3b50f227ed8f529e399858998cd70508b8a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.11.0
|
Security for VetNode 0.0.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.