decimate 0.9.6

A fault-tolerant SLURM scheduler extension

NAME

decimate - a fault-tolerant SLURM scheduler extension

SYNOPSIS

dbatch [ Slurm options ] [ --check <user_script> ]
[ --max-retry=<number of restart> ]
script [args...]

DESCRIPTION

Developped by the KAUST Supercomputing Laboratory (KSL),
decimate is a SLURM extension written in python designed to handle
dependent jobs more easely and efficiently.

Decimate transparently adds parameters to SLURM sbatch command
to check the correctness of jobs and automatically
reschedules jobs found faulty.

Using Decimate on Shaheen II, one can submit, run, monitor or
terminate a workflow composed of dependent jobs. If asked,
thanks to standardized or customized messages, the user will be
informed by mail of the progress of its workflow on the system.

In case of failure of one part of tne workflow, decimate
automatically detects the failure, signals it to the user and
launches the misbehaving part after having fixed the job
dependency. By default if the same failure happens three
consecutive times, decimate cancels the whole workfow removing
all the depending jobs from the scheduling. In a next version,
decimate will allow the automatic restarting of the workflow
once the problem causing its failure has been cured.

decimate also allows the user to define his own mail alerts
that can be sent at any point of the workflow through a call to
a python method. This feature will also be available from bash
in a next version.

Some customized checking functions can also be designed by the
user. Their purpose is to validate if a step of the workflow
was succesful or not. It could involved checking for the
presence of some result files, grepping some error or success
messages in them, computing ratio or checksum... These
intermediate results can be easely transmitted to decimate
validating or not the correctness of any step. They can also be
forwarded by mail to the user where as the workflow is
executing.

USE

At this moment, jobs only need to be submitted through the
dbatch
command that accepts exactely the same parameters as the
original SLURM sbatch command plus the new parameters

--check=SCRIPT_FILE
where SCRIPT_FILE is a python
or shell script
to check if results are ok.

--max-retry=MAX_RETRY
number of time a step can fail and be
restarted automatically before failing the
whole workflow (3 per default)

sslog tails out the decimate logging file attached to the
current directory, tracking all the jobs that were launched
with dbatch from this directory.

sstatus gives the current status of the workflow excecuting
in the current directory.

Decimate is still in a beta phase and under test with some of
our KSL users. More documentations will be provided once the
stabilized and fully tested version is made available by the
end of June 2018.

If interested in testing decimate or contributing, please send
a mail to [e-mail]

AUTHOR

Written by Samuel Kortas (samuel.kortas (at) kaust.edu.sa)

REPORTING BUGS

Report decimate bugs to [e-mail]


COPYRIGHT
Copyright (c) 2017, KAUST Supercomputing Laboratory
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

SEE ALSO

decimate official documentation pages:
<http://http://decimate.readthedocs.io>

KAUST Supercomputing Laboratory: <http://hpc.kaust.edu.sa/>