GPUQueue A very simple GPU tool - To run multiple jobs with assigned (limited) GPU resources
It provides very simple and basic function of dynamically utilize given GPUs with a large job array. It can be used to automatically identify the GPU that has been released by a newly-ended program.
Examples
python interface
from gpu_queue import JobSubmitter
job_array = [
"python -c 'import os, time;print(\"GPU num utilized\",os.environ[\"CUDA_VISIBLE_DEVICES\"]);time.sleep(3)'",
"python -c 'import os, time;print(\"GPU num utilized\",os.environ[\"CUDA_VISIBLE_DEVICES\"]);time.sleep(2)'",
"python -c 'import os, time;print(\"GPU num utilized\",os.environ[\"CUDA_VISIBLE_DEVICES\"]);time.sleep(0.5)'",
"python -c 'import os, time;print(\"GPU num utilized\",os.environ[\"CUDA_VISIBLE_DEVICES\"]);time.sleep(0.5)'",
"python -c 'import os, time;print(\"GPU num utilized\",os.environ[\"CUDA_VISIBLE_DEVICES\"]);time.sleep(3)'",
"python -c 'import os, time;print(\"GPU num utilized\",os.environ[\"CUDA_VISIBLE_DEVICES\"]);time.sleep(1)'",
]
J = JobSubmitter(job_array, [0, 1, 2])
J.submit_jobs()
Output:
6 jobs has been saved
GPU num utilized 0
GPU num utilized 2
GPU num utilized 1
GPU num utilized 2
GPU num utilized 2
GPU num utilized 1
all jobs has been run
sucessful jobs: 6
failed jobs: 0
gpuqueue can be directly used in the bash
Bash interface
#!/usr/bin/env bash
# example of typical machine learning hyper-parameter tuning
# mean teacher for semi supervised learning
save_dir=cifar10/labeled_sample_4000/augment_img
EMA_decay=0.999
declare -a StringArray=(
"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacherbaseline RegScheduler.max_value=0 Trainer.EMA_decay=${EMA_decay} "
"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacher_0.1 RegScheduler.max_value=0.1 Trainer.EMA_decay=${EMA_decay} "
"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacher_1 RegScheduler.max_value=1 Trainer.EMA_decay=${EMA_decay} "
"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacher_10 RegScheduler.max_value=10 Trainer.EMA_decay=${EMA_decay} "
"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacher_20 RegScheduler.max_value=20 Trainer.EMA_decay=${EMA_decay} "
"python classify_main.py Trainer.name=MeanTeacherTrainer Config=config/cifar_mt_config.yaml Trainer.save_dir=${save_dir}/meanteacher_50 RegScheduler.max_value=50 Trainer.EMA_decay=${EMA_decay} "
)
# just using 0 and 1 gpus for those jobs
gpuqueue "${StringArray[@]}" --available_gpus 0 1
install
pip install gpuqueue
Metadata
Release files for GPUQueue 0.0.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| GPUQueue-0.0.3.tar.gz | 4.0 kB | Without provenance | Details for GPUQueue-0.0.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| GPUQueue-0.0.3-py3-none-any.whl | Python 3 | none | any | Details for GPUQueue-0.0.3-py3-none-any.whl |
Total release size: 8.8 kB
Release files / GPUQueue-0.0.3.tar.gz
| Download URL | GPUQueue-0.0.3.tar.gz |
|---|---|
| Size | 4.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6b8c7fd192d922984893ceaac5e152f8408f0015f6eea07b9188d4efe5b405b0
|
|
BLAKE2b-256 checksum How to use checksums |
246ebebbb9b4d850a26829266f5976c725bf4b4b515ebcf633dfb51bae03109a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.3
|
Release files / GPUQueue-0.0.3-py3-none-any.whl
| Download URL | GPUQueue-0.0.3-py3-none-any.whl |
|---|---|
| Size | 4.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2c4f63b5e2cfe08e3f7e8caceacbeedc039caf610915ae1b076c83b59d863453
|
|
BLAKE2b-256 checksum How to use checksums |
20fcc0176ae76b31efea144c0d8fa1027751dc071bba85f59d2a4e80b18716e8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.3
|
Security for GPUQueue 0.0.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.