Skip to main content

Autosubmit is a Python-based workflow manager to create, manage and monitor complex tasks involving different substeps, such as scientific computational experiments. These workflows may involve multiple computing systems for their completion, from HPCs to post-processing clusters or workstations. Autosubmit can orchestrate all the tasks integrating the workflow by managing their dependencies, interfacing with all the platforms involved, and handling eventual errors.

Project description

Autosubmit is a lightweight workflow manager and meta-scheduler. It was originally designed in 2011 for use in climate research for configuring and running experiments. In the last few years, it has been extended to cover additional use cases, and today it is used as a general orchestration tool. It supports scheduling jobs to remote batch servers (via SSH) such as PBS, LSF, SLURM, and SGE.

Autosubmit is a Python package provided in PyPI, which facilitates easy and fast integration and relocation on new platforms. Conda recipes are available on the project website. A containerized version for testing purposes is also available but not public yet.

The features found in Autosubmit characterize it as both an experiment manager and also as a workflow orchestrator. The experiment manager allows users to define and configure experiments, supported by a hierarchical database that ensures reproducibility and traceability. The orchestrator is designed to run complex workflows in research and operational mode by managing their dependencies and interfacing with local and remote hosts.

These multi-scale workflows can contain from a few steps to thousands of steps, and from a single platform to multiple platforms. Platform is a concept in Autosubmit to abstract servers. A workflow configuration can include one or multiple platforms, allowing the workflow to run on any number of servers via password-less SSH without any external deployment.

Due to its robustness it can handle different eventualities such as networking connectivity issues or I/O errors. The monitoring capabilities extend beyond the command-line application through a REST API that allows communication with workflow monitoring tools such as the Autosubmit web GUI.

It has contributed to various European research projects and runs different operational systems. It will support the Earth Digital Twins as the Digital Twin Ocean over the next years.

It is currently used at the Barcelona Supercomputing Centre (BSC) to run models (EC-Earth, MONARCH, NEMO, CALIOPE, HERMES, and others), operational toolchains (S2S4E), data-download workflows (ECMWF MARS), and for many other use cases. Autosubmit has been used to run workflows in different supercomputers at BSC, ECMWF, IC3, CESGA, EPCC, PDC, and OLCF.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

autosubmit-4.1.14.tar.gz (1.9 MB view details)

Uploaded Source

Built Distribution

autosubmit-4.1.14-py3-none-any.whl (2.0 MB view details)

Uploaded Python 3

File details

Details for the file autosubmit-4.1.14.tar.gz.

File metadata

  • Download URL: autosubmit-4.1.14.tar.gz
  • Upload date:
  • Size: 1.9 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.9.21

File hashes

Hashes for autosubmit-4.1.14.tar.gz
Algorithm Hash digest
SHA256 c10ace2a13d2d86d20e8757b78b138ed33b821e56ec06326d6dc8f47ba7a56ea
MD5 56e339860e70b0551575b8fbe463c0b1
BLAKE2b-256 f73ee956867dd8e20c61e6cac0daa83fbb24cc21e851653405ad9a8314f712bf

See more details on using hashes here.

File details

Details for the file autosubmit-4.1.14-py3-none-any.whl.

File metadata

  • Download URL: autosubmit-4.1.14-py3-none-any.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.9.21

File hashes

Hashes for autosubmit-4.1.14-py3-none-any.whl
Algorithm Hash digest
SHA256 0351689bcbcfc0098a6023c6b6f082aa919ce39e30a0d66bb09ba57e4c229a72
MD5 c9436b0255a563e9bec7fc3ecd18ac31
BLAKE2b-256 624273fe5bf658979b707690060f3f1d44389d5175099a5d5dd48e2cc3dd0e40

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page