Skip to main content

Jobflow

tests code coverage pypi version supported python versions DOI

Documentation | PyPI | GitHub | Paper

Jobflow is a free, open-source library for writing and executing workflows. Complex workflows can be defined using simple python functions and executed locally or on arbitrary computing resources using the jobflow-remote or FireWorks workflow managers.

Some features that distinguish jobflow are dynamic workflows, easy compositing and connecting of workflows, and the ability to store workflow outputs across multiple databases.

Is jobflow for me

jobflow is intended to be a friendly workflow software that is easy to get started with, but flexible enough to handle complicated use cases.

Some of its features include:

  • A clean and flexible Python API.
  • A powerful approach to compositing and connecting workflows — information passing between jobs is a key goal of jobflow. Workflows can be nested allowing for a natural way to build complex workflows.
  • Integration with multiple databases (MongoDB, S3, GridFS, and more) through the Maggma package.
  • Support for the jobflow-remote and FireWorks workflow management systems, allowing workflow execution on multicore machines or through a queue, on a single machine or multiple machines.
  • Support for dynamic workflows — workflows that modify themselves or create new ones based on what happens during execution.

Workflow model

Workflows in jobflows are made up of two main components:

  • A Job is an atomic computing job. Essentially any python function can be Job, provided its input and return values can be serialized to json. Anything returned by the job is considered an "output" and is stored in the jobflow database.
  • A Flow is a collection of Job objects or other Flow objects. The connectivity between jobs is determined automatically from the job inputs. The execution order of jobs is automatically determined based on their connectivity.

Python functions can be easily converted in to Job objects using the @job decorator. In the example below, we define a job to add two numbers.

from jobflow import job, Flow

@job
def add(a, b):
    return a + b

add_first = add(1, 5)
add_second = add(add_first.output, 5)

flow = Flow([add_first, add_second])
flow.draw_graph().show()

The output of the job is accessed using the output attribute. As the job has not yet been run, output contains be a reference to a future output. Outputs can be used as inputs to other jobs and will be automatically "resolved" before the job is executed.

Finally, we created a flow using the two Job objects. The connectivity between the jobs is determined automatically and can be visualised using the flow graph.

simple flow graph

Installation

jobflow is a Python 3.10+ library and can be installed using pip.

pip install jobflow

Quickstart and tutorials

To get a first glimpse of jobflow, we suggest that you follow our quickstart tutorial. Later tutorials delve into the advanced features of jobflow.

Need help?

Ask questions about jobflow on the jobflow support forum. If you've found an issue with jobflow, please submit a bug report on GitHub Issues.

What’s new?

Track changes to jobflow through the changelog.

Contributing

We greatly appreciate any contributions in the form of a pull request. Additional information on contributing to jobflow can be found here. We maintain a list of all contributors here.

License

jobflow is released under a modified BSD license; the full text can be found here.

Citation

If you use Jobflow in your work, please cite it as follows:

  • "Jobflow: Computational Workflows Made Simple", A.S. Rosen, M. Gallant, J. George, J. Riebesell, H. Sahasrabuddhe, J.X. Shen, M. Wen, M.L. Evans, G. Petretto, D. Waroquiers, G.‑M. Rignanese, K.A. Persson, A. Jain, A.M. Ganose, Journal of Open Source Software, 9(93), 5995 (2024) DOI: 10.21105/joss.05995

Acknowledgements

Jobflow was designed by Alex Ganose, Anubhav Jain, Gian-Marco Rignanese, David Waroquiers, and Guido Petretto. Alex Ganose implemented the first version of the package. Later versions have benefited from the contributions of several research groups. A full list of contributors is available here.

Metadata

Release files for jobflow 0.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for jobflow 0.4.0
File Size Uploaded Provenance Actions
jobflow-0.4.0.tar.gz 58.9 kB Without provenance Details for jobflow-0.4.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for jobflow 0.4.0
File Interpreter ABI Platform Actions
jobflow-0.4.0-py3-none-any.whl Python 3 none any Details for jobflow-0.4.0-py3-none-any.whl

Total release size: 121.1 kB

Release files / jobflow-0.4.0.tar.gz

Download URL jobflow-0.4.0.tar.gz
Size 58.9 kB
Tags Source
SHA-256 checksum
How to use checksums
fc7c7318f9d5525b857e1671e3275e7eb1a0dd6ec93727706b16449736aa6d40
BLAKE2b-256 checksum
How to use checksums
3b8563b93422b5297d5aa8e0f7abce112c9feaec6f8b78748e73d21659e189e8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Release files / jobflow-0.4.0-py3-none-any.whl

Download URL jobflow-0.4.0-py3-none-any.whl
Size 62.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
28c17b9d7fb048b8861cbf97096f401a56df758d25b11bbc440088f0e66bdcc5
BLAKE2b-256 checksum
How to use checksums
7b256f78e3c39263e4a5687bd4008e143bc490c8fa9721b30850f1d7c8a164f5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Security for jobflow 0.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page