Skip to main content

Anaconda-Server Badge PyPI version DOI Join the chat at https://gitter.im/vatlab/SoS Build Status Build Status

Script of Scripts (SoS) consists of SoS Notebook, a Jupyter-based polyglot notebook that allows the use of multiple Jupyter kernels in one notebook, and SoS Workflow, a workflow system for the execution of workflows in both process- and outcome-oriented styles. It is designed for data scientists and bioinformatics who routinely work with scripts in different languages such as bash, Python, R, and SAS. This repository contains the SoS Workflow workflow engine.

Please refer to relevant publications SoS Notebook: An Interactive Multi-Language Data Analysis Environment in Bioinformatics, Script of Scripts: a pragmatic workflow system for daily computational research in PLoS Computational Biology, and the SoS homepage for more information on SoS Notebook and SoS Workflow.

We welcome and value community contributions: please post issues to provide us feedback or get our support; please send pull requests if you have helped fixing bugs or making improvements to the source code. In addition, if you are interesting in extending SoS, for example adding a new target type or task engine, please have a look at our tutorial on extending SoS here.

Installation

Quick Install

Using uv (fastest, recommended)

uv is a fast Python package installer written in Rust.

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install SoS
uv pip install sos

# Install with optional dependencies
uv pip install "sos[dot]"  # For graphviz support

Using pip

# Install from PyPI
pip install sos

# Install with optional dependencies
pip install sos[dot]  # For graphviz support

Using conda

conda install sos -c conda-forge

# Install complete SoS suite
conda install sos sos-pbs sos-notebook jupyterlab-sos sos-bash sos-python sos-r -c conda-forge

Development Installation

For developers and contributors, please see CONTRIBUTING.md for detailed setup instructions.

Quick Start with uv

# Clone the repository
git clone https://github.com/vatlab/SoS.git
cd SoS

# Set up development environment with uv
uv venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
uv sync

# Use invoke tasks for development
invoke --list      # Show available tasks
invoke test        # Run tests
invoke format      # Format code
invoke lint        # Check code style
invoke check       # Run all checks

Traditional Setup

# Clone and install in development mode
git clone https://github.com/vatlab/SoS.git
cd SoS
pip install -e . --group dev   # needs pip >= 25.1

Building from Source

With the modern build system using pyproject.toml:

# Using uv (fastest)
uv build

# Using Python build module
pip install build
python -m build

# Or using pip directly
pip wheel . --no-deps

Additional Components

SoS Notebook

pip install sos-notebook
python -m sos_notebook.install

Language Modules

pip install sos-r sos-matlab sos-python sos-bash

Remote Execution (PBS, Slurm, LSF)

pip install sos-pbs

For more details, see the installation page.

Change Log of SoS Workflow and SoS Notebook

SoS 0.25.1:

  • sos#1542: Fix a compatibility issue with Python 3.12.

SoS 0.25.0:

  • sos#1535: Remove 'R' formatter, remote target type, remote push and pull, named (#name) path because SoS now assumes that all input and output files are available on shared drives with identical path names.

SoS 0.24.5:

  • sos#1530: Outputcontent of script to stderr when the script fails to execute.

SoS 0.22.5:

  • sos#1420: Allow checking the status, killing, and purge of remote workflows.

SoS Notebook 0.22.3

  • sos-notebook#303: Make task execution non-blocking in sos notebook, so that the buttons to check status and remove tasks actually operatable.

SoS Notebook 0.22.2

  • sos-notebook#307: Use a new template organization method for nbconvert > 0.6.0.

SoS 0.21.12

  • sos#1392: Allow the use of pem_file to authenticate to remote hosts.

SoS 0.20.9

  • sos#1319: Extend option -r host to allow workflow to be executed with templates and on PBS
  • sos#1322: Deprecate option -b BIN_DIR due to sos#1319

SoS 0.19.15

  • sos#1273: Allow workflows to be executed on cluster systems.
  • sos#1277: Allow master tasks to be executed on cluster systems with extended runtime option trunk_workers.
  • sos#1279: Extend option -j of commands sos run and sos execute to use remote workers.
  • sos#1288: Change the default value of option -q from localhost to None (no task).

SoS 0.19.0

  • sos#1056, sos#1218: Use a new shared execution model to enforce optio -j.
  • sos#1219: Make sure global sections are executed only once

SoS Notebook 0.18.5

  • sos-notebook#192: Allow language modules to support pattern matched kernel names such as julia-?.?

SoS 0.18.6

  • sos#1206: Allow output_from(step) to obtain output from a workflow.

SoS 0.18.5

  • sos#1186: Add command line option -T to enfore dependency tracing.
  • sos#1197: Introduce function traced to make dependencies that will always be traced
  • sos#1201: Introduce dependencies between compounded workflows

SoS 0.18.4

  • sos#1185: Disallow the use of parameters in tasks.
  • sos#1186: (reverted) Enforcing the concept that targets in depends statement always try to build dependency.

SoS Notebook 0.18.1

  • sos-notebook#178: Allow adding & to the end of magics %run, %sosrun, and %runfile to execute workflow in background.
  • sos-notebook#179: Remove magic %rerun and add magic %runfile
  • sos-notebook#180: Add option -r to %save to execute the cell after saving it.

SoS 0.18.0

  • sos#1115: Function output_from and named_output to support named inputs and outputs, among other new features summarized in this ticket.
  • sos#1120: Allow paremeters group_by, paired_with etc for functions output_from and named_output.
  • sos#1125: Set concurrent=True as default for substep execution.
  • sos#1132: Deprecate action stop_if and replace it with done_if and skip_if
  • sos#1175: Enforce the use of sos_variable to import shared variable in a step

SoS Notebook 0.18.0

  • sos-notebook#150: A new side panel that works in the same fashion as JupyterLab's console window.
  • sos-notebook#154: New task status table that allows reporting status, killing, and purging multiple tasks with the same tags.

SoS 0.9.16.10

SoS 0.9.16.0, SoS Notebook 0.9.16.0

  • sos#991: Use a new task file format to replace multiple files for each task. This change is not backward compatible so please upgrade only after you completed and removed all existing tasks.

SoS 0.9.15.1

  • sos-notebook#89: Added templates to highlight source code using codemirror, with optional auto-generated table of contents.

SoS 0.9.14.10

  • sos#983: Allow depending on an entire process-oriented workflow using a sos_step() target that matches multiple steps.

SoS 0.9.14.3:

  • sos#975: Add option -p to generate a summary report after the completion of workflow.
  • sos#976: Much improved workflow help message (sos run script -h).

SoS Notebook 0.9.14.4:

  • sos-notebook#79: Allow auto-completion and inspection in subkernel cells.

SoS Notebook 0.9.14.1

  • sos-notebook#74: Add a %revisions magic to display revision history of the current document.

SoS 0.9.14.1

  • sos#925: Output summary of executed and ignored step, substeps, and tasks after the execution of workflows.

SoS Notebook 0.9.13.4

SoS Notebook 0.9.12.12

SoS 0.9.12.11

  • sos#922: Use user-id for docker execution (-u)
  • sos#926: Add function zap() to SoS path classes path, paths, file_target, and sos_targets

SoS Notebook 0.9.12.11

SoS 0.9.12.9

  • sos#914: Allow option active of actions and tasks to accept conditions.
  • sos#915: Automatically expand user (~) for SoS path types path, paths and file_targets.
  • sos#916: Use hashlib instead of faster xxhash under windows

SoS Notebook 0.9.12.9

SoS 0.9.12.3

  • sos#859: Introduce automatic auxiliary steps to simplify the use of makefile steps.

SoS 0.9.11.3

  • sos#879: Add action options stdout and stderr to redict output from script-executing actions.
  • sos-notebook#42: Add option --append to magic %capture .

SoS 0.9.11.2

  • sos-notebook#39: Separation installation and deployment and use command python -m sos_notebook.install to install sos kernel to Jupyter.

SoS 0.9.10.19

  • sos#874: Add input option concurrent=True to allow parallel execution of input groups.
  • sos#874: Optimize task submission of task engines to reduce status checking

SoS Notebook 0.9.10.17

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sos-0.25.2.tar.gz (246.2 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

sos-0.25.2-py3-none-any.whl (278.8 kB view details)

Uploaded Python 3

File details

Details for the file sos-0.25.2.tar.gz.

File metadata

  • Download URL: sos-0.25.2.tar.gz
  • Upload date:
  • Size: 246.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for sos-0.25.2.tar.gz
Algorithm Hash digest
SHA256 ac67926617f7dc0224940d0a33132895728a081c598c1e7460c584a846ef2e1c
MD5 8d87143382e98f91ec92f6c0ba1301f0
BLAKE2b-256 fca94511a33b5f97e3010c6694752e79c6f0c6475cbbb0c40132e4d40e29129f

See more details on using hashes here.

Provenance

The following attestation bundles were made for sos-0.25.2.tar.gz:

Publisher: release.yml on vatlab/sos

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file sos-0.25.2-py3-none-any.whl.

File metadata

  • Download URL: sos-0.25.2-py3-none-any.whl
  • Upload date:
  • Size: 278.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for sos-0.25.2-py3-none-any.whl
Algorithm Hash digest
SHA256 71d217b6663cbfd912fd3f57c3e666a17c488687dcd46c5eb8606666497eed56
MD5 eb7113a3dafbe0a888239ed8b5d9d76f
BLAKE2b-256 0b74db24390647a249bcee26298b5f69d98f78994a551dd184d387b5e6385628

See more details on using hashes here.

Provenance

The following attestation bundles were made for sos-0.25.2-py3-none-any.whl:

Publisher: release.yml on vatlab/sos

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.25.2 This release

2 files

0.25.1

2 files

0.25.0

2 files

0.24.5

2 files

0.24.4

2 files

0.24.3

2 files

0.24.2

2 files

0.24.1

2 files

0.24.0

2 files

0.23.4

2 files

0.23.3

2 files

0.23.2

2 files

0.23.1

2 files

0.23.0

2 files

0.22.7

2 files

0.22.6

2 files

0.22.5

2 files

0.22.4

2 files

0.22.3

2 files

0.22.2

2 files

0.22.1

2 files

0.22.0

2 files

0.21.16

2 files

0.21.15

2 files

0.21.14

2 files

0.21.13

2 files

0.21.12

2 files

0.21.11

2 files

0.21.10

2 files

0.21.9

2 files

0.21.8

2 files

0.21.7

2 files

0.21.6

2 files

0.21.5

2 files

0.21.4

2 files

0.21.3

2 files

0.21.2

2 files

0.21.1

2 files

0.20.13

2 files

0.20.12

2 files

0.20.11

2 files

0.20.10

2 files

0.20.9

2 files

0.20.8

2 files

0.20.7

2 files

0.20.6

2 files

0.20.5

2 files

0.20.4

2 files

0.20.3

2 files

0.20.2

2 files

0.20.1

2 files

0.20.0

2 files

0.19.16

2 files

0.19.15

2 files

0.19.14

2 files

0.19.13

2 files

0.19.12

2 files

0.19.11

2 files

0.19.10

2 files

0.19.9

2 files

0.19.8

2 files

0.19.7

2 files

0.19.6

2 files

0.19.5

2 files

0.19.4

2 files

0.19.3

2 files

0.19.2

2 files

0.19.1

2 files

0.19.0

2 files

0.18.7

2 files

0.18.6

2 files

0.18.5

2 files

0.18.4

2 files

0.18.3

2 files

0.18.2

2 files

0.18.1

2 files

0.18.0

2 files

0.17.7

2 files

0.17.6

2 files

0.17.5

2 files

0.17.4

2 files

0.17.3

2 files

0.17.2

2 files

0.17.1

2 files

0.17.0

2 files

0.16.11

2 files

0.16.10

2 files

0.16.9

2 files

0.9.16.8

2 files

0.9.16.7

2 files

0.9.16.6

2 files

0.9.16.5

2 files

0.9.16.4

2 files

0.9.16.3

2 files

0.9.16.2

2 files

0.9.16.1

2 files

0.9.16.0

2 files

0.9.15.3

2 files

0.9.15.2

2 files

0.9.15.1

2 files

0.9.15.0

2 files

0.9.14.10

2 files

0.9.14.9

2 files

0.9.14.8

2 files

0.9.14.7

2 files

0.9.14.6

2 files

0.9.14.5

2 files

0.9.14.4

2 files

0.9.14.3

2 files

0.9.14.2

2 files

0.9.14.1

2 files

0.9.14.0

2 files

0.9.13.9

2 files

0.9.13.8

2 files

0.9.13.7

2 files

0.9.13.6

2 files

0.9.13.5

2 files

0.9.13.4

2 files

0.9.13.3

2 files

0.9.13.2

2 files

0.9.13.1

2 files

0.9.13.0

2 files

0.9.12.10

2 files

0.9.12.9

2 files

0.9.12.8

2 files

0.9.12.7

2 files

0.9.12.6

2 files

0.9.12.5

2 files

0.9.12.4

2 files

0.9.12.3

2 files

0.9.12.2

2 files

0.9.12.1

2 files

0.9.12.0

2 files

0.9.11.4

2 files

0.9.11.3

2 files

0.9.11.2

2 files

0.9.11.1

2 files

0.9.11.0

1 file

0.9.10.18

1 file

0.9.10.17

1 file

0.9.10.16

1 file

0.9.10.15

1 file

0.9.10.14

1 file

0.9.10.12

1 file

0.9.10.11

1 file

0.9.10.10

1 file

0.9.10.9

1 file

0.9.10.8

1 file

0.9.10.7

1 file

0.9.10.6

1 file

0.9.10.5

1 file

0.9.10.4

1 file

0.9.10.3

1 file

0.9.10.2

1 file

0.9.10.1

1 file

0.9.10.0

1 file

0.9.9.11

1 file

0.9.9.10

1 file

0.9.9.9

1 file

0.9.9.8

1 file

0.9.9.7

1 file

0.9.9.6

1 file

0.9.9.5

1 file

0.9.9.4

1 file

0.9.9.3

1 file

0.9.9.2

1 file

0.9.9.1

1 file

0.9.9.0

1 file

0.9.8.10

1 file

0.9.8.9

1 file

0.9.8.8

1 file

0.9.8.7

1 file

0.9.8.6

1 file

0.9.8.5

1 file

0.9.8.4

1 file

0.9.8.3

1 file

0.9.8.2

1 file

0.9.8.1

1 file

0.9.8

1 file

0.9.7.9

1 file

0.9.7.8

1 file

0.9.7.7

1 file

0.9.7.6

1 file

0.9.7.5

1 file

0.9.7.4

1 file

0.9.7.3

1 file

0.9.7.2

1 file

0.9.7.1

1 file

0.9.7

1 file

0.9.6.1

1 file

0.9.6

1 file

0.9.5

1 file

0.9.4

1 file

0.9.3.2

1 file

0.9.3.1

1 file

0.9.3

1 file

0.9.2

1 file

0.9.1

1 file

0.9.0

1 file

0.8.7

1 file

0.8.6

1 file

0.8.5

1 file

0.8.4.1

1 file

0.8.4

1 file

0.8.3

1 file

0.8.2

1 file

0.8.1

1 file

0.8.0

1 file

0.7.9.11

1 file

0.7.9.10

1 file

0.7.6

1 file

0.7.5

1 file

0.7.4

1 file

0.7.3

1 file

0.6.4

1 file

0.6.3.3

1 file

0.6.3.2

2 files

0.6.3.1

1 file

0.6.3

1 file

0.6.2

1 file

0.6.1

1 file

0.6.0

1 file

0.5.9

1 file

0.5.8

1 file

0.5.7

1 file

0.5.6

1 file

0.5.4

1 file

0.5.3

1 file

0.5.2

1 file

0.5.1

1 file

0.5.0

1 file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page