JZ-hydra-submitit-launcher
A Jean Zay (JZ) tailored Hydra submitit launcher based on Hydra and its submitit-launcher plugin. Basically it extends the submitit-launcher plugin with defaults that make sense for JZ.
Install
This package can be installed from pypi (using --user is required on JZ):
pip install --user jz-hydra-submitit-launcher
You can also install it from source:
git clone https://github.com/zaccharieramzi/jz-hydra-submitit-launcher.git
cd jz-hydra-submitit-launcher
pip install --user .
The main command is then installed in your local binaries.
You need to add them to your path in order for the command to be found (and avoid the error -bash: hydra-submitit-launch: command not found):
echo "export PATH=$PATH:$HOME/.local/bin" >> $HOME/.bashrc
Don't forget to source $HOME/.bashrc before using the command.
Use
The primary use is with the hydra-submitit-launch command with your script name and the config type:
hydra-submitit-launch my_app.py dev
Available configs
6 different configs are available:
dev: with 2 hours, 1 gpu, and qos_gpu-dev.t3: with 20 hours, 1 gpu, and qos_gpu-t3.t4: with 100 hours, 1 gpu, and qos_gpu-t4.4gpus_dev: with 2 hours, 4 gpus, and qos_gpu-dev.4gpus_t3: with 20 hours, 4 gpus, and qos_gpu-t3.4gpus_t4: with 100 hours, 4 gpus, and qos_gpu-t4.
By default, all the configs select 32Gb GPUs, use a single node and use the gpu_p1 partition.
Advanced configs
You can override the SLURM config, the same way you would with any hydra configuration.
The parameters you can override are defined in the hydra-submitit-launcher plugin doc.
For example, if you want to use the gpu_p2 partition, you would need to do:
hydra-submitit-launch my_app.py dev hydra.launcher.setup=null hydra.launcher.partition=gpu_p2 hydra.launcher.cpus_per_task=3
In order to change the timeout on the SLURM job to for example 10 hours, you would need to do:
hydra-submitit-launch my_app.py base +hours=10
This will automatically select the right qos for you (and if you want to force a certain qos, you can always add hydra.launcher.qos=<your-qos>).
Logs
Your SLURM logs are stored in the following path: launch-dir/multirun/day-of-launch/time-of-launch/.submitit/job_array_task_id/
This is the default from hydra and submitit and this can be changed using both the submitit plugin parameters (see here) and the hydra job configurations (see here).
References
- Hydra: https://hydra.cc/docs/intro/
- submitit-launcher: https://hydra.cc/docs/plugins/submitit_launcher/
- submitit: https://github.com/facebookincubator/submitit
- JZ docs: http://www.idris.fr/ or https://jean-zay-doc.readthedocs.io/en/latest/
Metadata
Release files for jz-hydra-submitit-launcher 0.1.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| jz-hydra-submitit-launcher-0.1.5.tar.gz | 6.6 kB | Without provenance | Details for jz-hydra-submitit-launcher-0.1.5.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| jz_hydra_submitit_launcher-0.1.5-py3-none-any.whl | Python 3 | none | any | Details for jz_hydra_submitit_launcher-0.1.5-py3-none-any.whl |
Total release size: 15.0 kB
Release files / jz-hydra-submitit-launcher-0.1.5.tar.gz
| Download URL | jz-hydra-submitit-launcher-0.1.5.tar.gz |
|---|---|
| Size | 6.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
45153115a5f3e6410c13908b95e2daabd27040beacde5e31dde2476036dea643
|
|
BLAKE2b-256 checksum How to use checksums |
89e873ba3299ffb0f989a58e9d5b8ef29097dc572cf49c50ebd7828a13549789
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.2
|
Release files / jz_hydra_submitit_launcher-0.1.5-py3-none-any.whl
| Download URL | jz_hydra_submitit_launcher-0.1.5-py3-none-any.whl |
|---|---|
| Size | 8.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c6d2a7fcc20e84177b361bff4e5a6806a29366cf36fe62a1bfd2b0552270570a
|
|
BLAKE2b-256 checksum How to use checksums |
487143b8cdf6eefddaed5b9adbe955f190586d1f449a87065c18a1dc678c25a9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.2
|
Security for jz-hydra-submitit-launcher 0.1.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.