mml-lsf 0.5.1

This is the MML LSF plugin, providing LSF cluster support on the specific setting of the DKFZ GPU cluster.

MML LSF plugin

This plugin provides LSF cluster support on the specific setting of the DKFZ GPU cluster.

Install

pip install mml-lsf

If you want to use the submission features of mml_lsf.runner.LSFJobRunner you need to set up the following (not required by any of the other features of this plugin):

• install sshpass for providing ssh with password

  sudo apt install sshpass
  

• set the following variables in your mml.env (alternatively provide them manually to mml_lsf.runner.LSFJobRunner)
  • export MML_AD_USER=...
  • export MML_CLUSTER_HOST=... (a LSF submission host)
  • export MML_CLUSTER_WORKER=... (a cluster worker node)

Usage

First and foremost it automatically ensures the number of workers used by MML to be conforming to the node the job will be executed (see mml_lsf.workers). In addition, it provides a suitable implementation for job planning on the LSF cluster, taking care of all necessary prefixes to the CLI (see mml_lsf.requirements). Finally, it offers the LSFJobRunner to automatically submit job. Alternatively it is also possible to submit via pre-rendering into a local file and ssh file tunneling.

Usage with `sshpass'

from mml.core.scripts.utils import load_env
from mml_lsf.requirements import LSFSubmissionRequirements
from mml_lsf.runner import LSFJobRunner
from mml.interactive.planning import MMLJobDescription

# make sure to load mml.env variables
load_env()  # if within a jupyter notebook, instead invoke mml.interactive.init()
# setup job requirements
reqs = LSFSubmissionRequirements(
    num_gpus=1, 
    vram_per_gpu=11.0, 
    queue='gpu-lowprio',
    mail='[e-mail]',  # optional
    script_name='mml.sh',  # name of my runner script to load CUDA, conda env, etc and finally invoke mml
    job_group='/USERNAME/JOB_GROUP_NAME',   # optional, used e.g. to limit max number of jobs 
    interactive=True  # optional, if True realtime updates are printed to terminal
    )
# setup runner, will prompt for password once
runner = LSFJobRunner() 
job = MMLJobDescription(prefix_req=reqs, mode='info', config_options={})  # simple job "mml info"
job.run(runner=runner)  # will submit job (no password prompt)
job2 = MMLJobDescription(prefix_req=reqs, mode='train', config_options={})  # another job "mml info"
job2.run(runner=runner) # will submit job (no password prompt)

Usage with file tunneling

from mml_lsf.requirements import LSFSubmissionRequirements
from mml.interactive.planning import MMLJobDescription, write_out_commands

# setup job requirements
reqs = LSFSubmissionRequirements(
    num_gpus=1, 
    vram_per_gpu=11.0, 
    queue='gpu-lowprio',
    mail='[e-mail]',  # optional
    script_name='mml.sh',  # name of my runner script to load CUDA, conda env, etc and finally invoke mml
    job_group='/USERNAME/JOB_GROUP_NAME',   # optional, used e.g. to limit max number of jobs 
    interactive=False  # setting True is not recommended for batched submission
    )

# create batch of cmds
cmds = list()
# cmd 1 some dummy task
prep_cmds.append(MMLJobDescription(prefix_req=reqs, mode='train', config_options={'tasks': 'fake', 'proj': 'dummy'}))
# cmd 2 another dummy task
prep_cmds.append(MMLJobDescription(prefix_req=reqs, mode='train', config_options={'tasks': 'fake', 'proj': 'dummy'}))
# now write
write_out_commands(cmd_list=cmds, name='exp1')
# this creates a 'exp1.txt' at current working directory

Now submit these jobs via:

ssh AD_USER@SUBMISSION_HOST 'bash -s' < /path/to/exp1.txt