Versatile computation submission tool
Project description
PySubmit
The versatile computation submission tool.
Purpose
Scientists who use computational methods often face the problem to test various parameter values as input for their numerical codes.
Having the power of high performance computer clusters (HPCs) at hand, it is easy to test up to multiple thousand input parameter values simultaneously. However, HPCs typically use batch systems that schedule computation jobs. Special files called start scripts are required to get your computation running. These start scripts are even useful for computations on your local workstations if they are adapted a little.
This tool enables to generate these start scripts with ease and for arbitrary interfaces to your numerical code. The parameters are supplied using Python and a boilerplate of your start script that needs to be filled with values. With that, this tool helps to make your numerical research more reproducible.
Installation
The package is on PyPI so that an installation is as simple as:
$ pip install pysubmit
Alternatively, you can install the package in developer mode with from this repository:
$ pip install -e .
with the current directory being the top level folder of the PySubmit project.
Configuration
Call pysubmit -h
to see possible configuration options.
Creating a template
PySubmit uses regular Python files as templates for the creation of the start scripts.
Create a Python file with a function render()
in it, which needs to return filenames
and parameter values for the variables in the actual start script. The start script itself
is added to this Python file with triple quotes ("""
) at the very end. The opening triple
quotes are followed by <<<
and the ending ones are preceded by >>>
. Hence, the limiting
syntax of the start script code inside the Python file that comes with the render()
function is """<<<
and >>>"""
.
This format keeps the PySubmit template very consise and contained in a single file. A basic example
is found in tests/template_example_new.py
. Try to use the template by executing:
$ pysubmit -pf <path>/<to>/<repo>/pysubmit/tests/template_example_new.py -o TEST
to generate the start scripts as defined in the example and save them into the folder TEXT/
.
The version 1.0 of PySubmit uses a different format for the templates. To use the file format
from v0.1 - and thereby be backwards compatible -, the command-line tool pysubmit
comes with an extra keyword --boilerplate-file
. If you would like to use the old format
with v1.0, use --pysubmit-file
to supply your render.py
file and --boilerplate-file
to supply the boilerplate
file. An example of the old format can be found under
tests/template_example_old.py
and tests/template_example_old.boilerplate
.
Upload to pypi
These are the steps to upload this project to pypi:
- Build the package, see https://packaging.python.org/tutorials/packaging-projects/#generating-distribution-archives.
- Upload the package, see https://packaging.python.org/tutorials/packaging-projects/#uploading-the-distribution-archives.
The summary of those two resources is:
- Build the package:
python setup.py sdist bdist_wheel
- Upload the package:
twine upload dist/*
Help
If anything is unclear, please contact me under martin.lellep@gmail.com :-). I personally find this tool incredibly useful and use it literally everyday to start my simulations on HPCs.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.