balsam-flow 0.3.5

HPC Workflows & Edge Service

Balsam: HPC Workflows & Edge Service

Balsam makes it easy to manage large compute campaigns on a supercomputer:

• Many independent application runs (i.e. classic ensemble jobs)
• Many instances of workflows, with inter-task dependencies forming graphs
• Dynamic workflows, where some tasks spawn other tasks with the Python API
• Remotely submit workflows and track their progress
• Multi-user workflow management

Use a command-line interface or Python API to fill a database with a few dozen or million tasks. The Balsam components will automatically bundle your work and talk to the system scheduler to allocate resources. On the inside, a pilot launcher process executes your workflows and keeps you informed of what's going on.

Read the Balsam Documentation online at balsam.readthedocs.io!

Installation

Prerequisites

Balsam requires Python 3.6 or later. You will need setuptools 39.2 or newer:

$ pip install --upgrade pip setuptools

Some Balsam components require mpi4py, so it is best to install Balsam in an environment with mpi4py already in place and configured for your platform. At the very least, a working MPI implementation and mpicc compiler wrapper should be in the search path so that the dependency can be automatically installed.

Cython is also used to compile some CPU-intensive portions of the Balsam service. While the Cython dependency will also be installed if it's absent, it is preferable to have an existing version built with your platform-tuned compiler wrappers.

Quick setup

$ pip insall balsam-flow
$ balsam init ~/myWorkflow
$ source balsamactivate myWorkflow

Citing Balsam

If you are referencing Balsam in a publication, please cite the following paper:

• M. Salim, T. Uram, J.T. Childers, P. Balaprakash, V. Vishwanath, M. Papka. Balsam: Automated Scheduling and Execution of Dynamic, Data-Intensive HPC Workflows. In Proceedings of the 8th Workshop on Python for High-Performance and Scientific Computing. ACM Press, 2018.

BSD 3-Clause License

Copyright (c) 2019, UChicago Argonne LLC All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of UChicago Argonne LLC nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.