Skip to main content

Sophios

doc-build-status

Sophios is a workflow compiler. You write a workflow in Python, or in .wic, a short YAML format. Sophios holds it in a workflow model of its own, checks all of it before anything runs, and compiles it for a backend. Today that backend is the Common Workflow Language (CWL), so what it writes runs under any CWL runner, with or without Sophios installed: cwltool on your machine, or Toil, which sends the steps to a cloud or to Slurm, LSF and other cluster schedulers.

An analysis is rarely one program: you clean the raw data, fit a model and compute statistics on it side by side, and put both into a figure. Steps, and the data that flows between them, make a graph. Run by hand or from a shell script, that graph works on your machine today; Sophios makes it a workflow that runs the same way on next month's data, on a colleague's machine and on a cluster. You describe each step: which program it runs, what you give it and which earlier results it reads. Sophios fills in the connections you leave out and tells you what it chose, stops on a value of the wrong type, a missing input file or a container engine that is not running, and runs every program in the container it asks for.

Sophios's Python API has two halves. sophios.api.python.workflow builds workflows from Step and Workflow objects. sophios.api.python.tool_builder builds the tools themselves, so a program of your own becomes a step without hand-written CWL.

Documentation

The documentation is on Read the Docs. Start with Install, then Your first workflow.

Install

  1. Install Docker and start it. Sophios runs each tool in the container image it names.

  2. Create a conda environment with the programs pip cannot install, Node.js (the newest 26.x) and Graphviz:

    conda create --name sophios --channel conda-forge python=3.11 nodejs=26 graphviz
    conda activate sophios
    
  3. Install Sophios, and let it write its configuration and copy the tools that ship with it to ~/wic/:

    python -m pip install sophios
    sophios --generate_config
    
  4. Check that it works:

    mkdir sophios-check && cd sophios-check
    sophios --yaml ~/wic/examples/multistep1.wic --run_local --quiet
    

    The last line starts with Success! Runner outputs are under.

The install guide explains each step and what to do when one fails. To work on Sophios itself, start with the developer guide.

A first workflow

This workflow makes an empty file, writes a greeting into it, and reads the file back. In a project directory, save either form:

# greeting.wic
steps:
- id: touch
  in:
    filename: !ii greeting.txt
  out:
  - file: !& empty_file
- id: append
  in:
    file: !* empty_file
    str: !ii Hello, Sophios
  out:
  - file: !& greeting_file
- id: cat
  in:
    file: !* greeting_file
# greeting.py
"""Create a file, write a greeting into it, and read it back."""
from pathlib import Path

from sophios.api.python.workflow import Step, Workflow


#: The tools that ship with Sophios, where `sophios --generate_config` put them.
ADAPTERS = Path.home() / 'wic' / 'cwl_adapters'


def workflow() -> Workflow:
    """The same three steps as greeting.wic."""
    touch = Step(clt_path=ADAPTERS / 'touch.cwl')
    touch.inputs.filename = 'greeting.txt'
    append = Step(clt_path=ADAPTERS / 'append.cwl')
    append.inputs.file = touch.outputs.file
    append.inputs.str = 'Hello, Sophios'
    cat = Step(clt_path=ADAPTERS / 'cat.cwl')
    cat.inputs.file = append.outputs.file
    return Workflow([touch, append, cat], 'greeting')


if __name__ == '__main__':
    workflow().run()

!ii marks a value written into the workflow; !& names a step's output and !* feeds it into a later step's input. In Python, an assignment does each, and ADAPTERS is where sophios --generate_config put the tools that ship with Sophios. Run either form:

sophios --yaml greeting.wic --run_local
python greeting.py

Both compile to the same CWL, check the machine, fetch the container images and run. Sophios writes the compiled CWL, the job file and the results under autogenerated/ in the directory you run from. --basepath (or basepath= in Python) lets you choose another directory for them, and --outdir one for only the files the run produces. Every problem Sophios finds carries a code, and sophios --explain <code> says what to change.

Writing a tool

When a program has no CWL description yet, the tool builder writes one, and the tool goes straight into a workflow. Here wc -w counts the words of the greeting:

# count_words.py
"""Build a tool in Python, then run it in a workflow."""
from pathlib import Path

from sophios.api.python.tool_builder import (
    CommandLineTool, Input, Inputs, Output, Outputs, cwl,
)
from sophios.api.python.workflow import Step, Workflow

#: The tools that ship with Sophios, where `sophios --generate_config` put them.
ADAPTERS = Path.home() / 'wic' / 'cwl_adapters'

word_count = (
    CommandLineTool('word_count',
                    Inputs(text=Input(cwl.file, position=1)),
                    Outputs(counts=Output(cwl.file, glob='counts.txt')))
    .base_command('wc', '-w')
    .stdout('counts.txt')
    .docker('debian:bookworm-slim')
)


def workflow() -> Workflow:
    """Create a file, write a line into it, and count its words."""
    touch = Step(clt_path=ADAPTERS / 'touch.cwl')
    touch.inputs.filename = 'greeting.txt'
    append = Step(clt_path=ADAPTERS / 'append.cwl')
    append.inputs.file = touch.outputs.file
    append.inputs.str = 'Hello, Sophios'
    count = Step(word_count)
    count.inputs.text = append.outputs.file
    return Workflow([touch, append, count], 'count_words')


if __name__ == '__main__':
    word_count.write_cwl('word_count.cwl', validate=True)
    workflow().run()

The run writes word_count.cwl, a tool file any .wic workflow can use, and leaves counts.txt in the output directory. Writing tools in Python takes the builder one part at a time.

License

MIT; see LICENSE.

Metadata

Release files for sophios 0.7.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sophios 0.7.0
File Size Uploaded Provenance Actions
sophios-0.7.0.tar.gz 1.3 MB Without provenance Details for sophios-0.7.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sophios 0.7.0
File Interpreter ABI Platform Actions
sophios-0.7.0-py3-none-any.whl Python 3 none any Details for sophios-0.7.0-py3-none-any.whl

Total release size: 1.7 MB

Release files / sophios-0.7.0.tar.gz

Download URL sophios-0.7.0.tar.gz
Size 1.3 MB
Tags Source
SHA-256 checksum
How to use checksums
98fbd9b024c5c06702e3b575da1e0a394d422f438aef7d85ecdef467ff3eb447
BLAKE2b-256 checksum
How to use checksums
f55c46e53344c5339c6515ed52b745d0b73e944d3cc7472878b2d71802f6557c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.17

Release files / sophios-0.7.0-py3-none-any.whl

Download URL sophios-0.7.0-py3-none-any.whl
Size 433.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a993b5275bdeee0aedc44593adb3d6e61f58476f043fbde10fb20dbeb70b9033
BLAKE2b-256 checksum
How to use checksums
7a22fb83432f8eaf253c1a89b41ff4aadaf02d1547705572ab53e17f5b65ec86
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.17

Security for sophios 0.7.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page