Skip to main content

Autosubmit: a versatile tool to manage Weather and Climate Experiments in diverse Supercomputing Environments

Project description


Autosubmit is a tool to create, manage and monitor experiments using
computing clusters, HPC platforms and supercomputers remotely via ssh.


HOW TO DEPLOY/SETUP AUTOSUBMIT FRAMEWORK
========================================

- Autosubmit has been tested:

with the following Operating Systems:
* Linux Debian

on the following HPC platforms:
* Ithaca (IC3 machine)
* Stargate (IC3 machine)
* MareNostrum (BSC machine)
* MareNostrum3 (BSC machine)
* HECToR (EPCC machine)
* Lindgren (PDC machine)
* C2A (ECMWF machine)
* CCA (ECMWF machine)
* ARCHER (EPCC machine)

- Pre-requisites: These packages (bash, python2, sqlite3, git-scm > 1.8.2, subversion) must be available at local
machine. These packages (argparse, dateutil, pyparsing, numpy, pydotplus, matplotlib, paramiko, saga-python,
pythondialog, mock, portalocker) must be available for python runtime. And the machine is also able to access
HPC platforms via password-less ssh.

- Install Autosubmit
> pip install autosubmit
or download, unpack and "python setup.py install"

- Create a repository for experiments: Say for example "/cfu/autosubmit" then "autosubmit configure" and follow
instructions

- Create a blank database: "autosubmit install"

HOW TO USE AUTOSUBMIT
=====================
> autosubmit -h

> autosubmit expid --HPC ithaca --description "experiment is about..."

Say for example, "cxxx" is 4 character based expid generated by system automatically.
First character is a letter, the other three alfa-numeric characters allow to identify uniquely the experiment.

> vi /cfu/autosubmit/cxxx/conf/expdef_cxxx.conf

> vi /cfu/autosubmit/cxxx/conf/autosubmit_cxxx.conf

> vi /cfu/autosubmit/cxxx/conf/platforms_cxxx.conf

> vi /cfu/autosubmit/cxxx/conf/jobs_cxxx.conf

> autosubmit create cxxx

> nohup autosubmit run cxxx &

Cautions:
- Before launching autosubmit check the following stuff:
> ssh ithaca # (for example) check other HPC platforms where password-less ssh is feasible.
- After launching autosubmit, one must be aware of login expiry limit and policy (if applicable for any HPC)
and renew the login access accordingly (by using token/key etc) before expiry.

HOW TO MONITOR EXPERIMENT
=========================

> autosubmit monitor -h

> autosubmit monitor cxxx
or
> autosubmit monitor cxxx -o png

Above generated plot with date & time stamp can be found at:

/cfu/autosubmit/cxxx/plot/cxxx_date_time.pdf
or
/cfu/autosubmit/cxxx/plot/cxxx_date_time.png


HOW TO RESTART EXPERIMENT
=========================

> autosubmit recovery -h

> autosubmit recovery cxxx -s # saving the pickle file

> nohup autosubmit run cxxx &


FULL DOCUMENTATION AND HOW TOs
==============================

Check the Autosubmit documentation provided in the docs/ folder of the package, in PDF format.
Check the online documentation in the following web page: http://www.bsc.es/projects/earthscience/autosubmit/

Project details


Release history Release notifications

This version
History Node

3.6.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
autosubmit-3.6.0.tar.gz (1.5 MB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page