labtasker-plugin-script-generate
[!WARNING] This plugin does not cover all possible scenarios and does not guarantee the correctness of the generated scripts. It is strongly advised to double-check and manually adjust the generated scripts to your need.
This is a plugin for Labtasker that helps you to automatically split your bash script into submit script and job script.
Install
pip install labtasker-plugin-script-generate
Example
You can find the code in the demo directory.
Imagine you have a bash wrapper script for your machine learning experiment. To divide the script into a submit script
and a job script, simply add a few special comments (e.g., #@submit, #@task, #@end).
#!/bin/bash
export CUDA_HOME=/usr/local/cuda-12.1
+#@submit
for dataset in imagenet cifar10 mnist; do
for model in resnet50 vit transformer; do
LOG_DIR=/path/to/logs/$dataset/$model
+ #@task
python train.py --dataset $dataset \
--model $model \
--cuda-home $CUDA_HOME \
--log-dir $LOG_DIR
+ #@end
done
done
+#@end
echo "done"
Run:
labtasker generate original.sh
You will see original_submit.sh and original_run.sh generated at the same directory, with
the following content:
original_submit.sh:
#!/bin/bash
# This script is generated by Labtasker from original.sh
export CUDA_HOME=/usr/local/cuda-12.1
for dataset in imagenet cifar10 mnist; do
for model in resnet50 vit transformer; do
LOG_DIR=/path/to/logs/$dataset/$model
labtasker task submit -- --CUDA_HOME "$CUDA_HOME" --LOG_DIR "$LOG_DIR" --dataset "$dataset" --model "$model"
done
done
echo "done"
original_run.sh:
#!/bin/bash
# This script is generated by Labtasker from original.sh
export CUDA_HOME=/usr/local/cuda-12.1
LABTASKER_TASK_SCRIPT=$(mktemp)
cat <<'LABTASKER_LOOP_EOF' > "$LABTASKER_TASK_SCRIPT"
CUDA_HOME=%(CUDA_HOME)
LOG_DIR=%(LOG_DIR)
dataset=%(dataset)
model=%(model)
python train.py --dataset $dataset \
--model $model \
--cuda-home $CUDA_HOME \
--log-dir $LOG_DIR
LABTASKER_LOOP_EOF
labtasker loop --executable /bin/bash --script-path $LABTASKER_TASK_SCRIPT
echo "done"
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file labtasker_plugin_script_generate-0.1.3.tar.gz.
File metadata
- Download URL: labtasker_plugin_script_generate-0.1.3.tar.gz
- Upload date:
- Size: 13.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
fa74041ffab5fcffa6e3929fe5bae880547eaf17eaf2014169a280849666e49b
|
|
| MD5 |
69d17c1d91c3a9781db535514c810afb
|
|
| BLAKE2b-256 |
1a1f84bd0c9a0060f38d3700c4bcf0e166443d44ae555debb6306fb7bb85d5bb
|
Provenance
The following attestation bundles were made for labtasker_plugin_script_generate-0.1.3.tar.gz:
Publisher:
release.yml on luocfprime/labtasker-plugin-script-generate
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
labtasker_plugin_script_generate-0.1.3.tar.gz -
Subject digest:
fa74041ffab5fcffa6e3929fe5bae880547eaf17eaf2014169a280849666e49b - Sigstore transparency entry: 311446790
- Sigstore integration time:
-
Permalink:
luocfprime/labtasker-plugin-script-generate@801a9f6fa73e27923119757268e384ab48070d54 -
Branch / Tag:
refs/tags/v0.1.3 - Owner: https://github.com/luocfprime
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@801a9f6fa73e27923119757268e384ab48070d54 -
Trigger Event:
release
-
Statement type:
File details
Details for the file labtasker_plugin_script_generate-0.1.3-py3-none-any.whl.
File metadata
- Download URL: labtasker_plugin_script_generate-0.1.3-py3-none-any.whl
- Upload date:
- Size: 6.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
44522dbec492e0afc8bc4cf058630d351e1c47431e574f7c57053964ebe27e71
|
|
| MD5 |
92d08381b90c9512f6201cbbb1b61597
|
|
| BLAKE2b-256 |
a17938ce84ca45b05770a70d2edda986381c5caa0ced082d1bbb82a6d54bbfb6
|
Provenance
The following attestation bundles were made for labtasker_plugin_script_generate-0.1.3-py3-none-any.whl:
Publisher:
release.yml on luocfprime/labtasker-plugin-script-generate
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
labtasker_plugin_script_generate-0.1.3-py3-none-any.whl -
Subject digest:
44522dbec492e0afc8bc4cf058630d351e1c47431e574f7c57053964ebe27e71 - Sigstore transparency entry: 311446793
- Sigstore integration time:
-
Permalink:
luocfprime/labtasker-plugin-script-generate@801a9f6fa73e27923119757268e384ab48070d54 -
Branch / Tag:
refs/tags/v0.1.3 - Owner: https://github.com/luocfprime
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@801a9f6fa73e27923119757268e384ab48070d54 -
Trigger Event:
release
-
Statement type:
