Pantarei

A general-purpose workflow manager - because everything flows

$ rei run project.py 
$ rei ls -l project.py 
⬜ project.py  |███████████         | 50% [2/4] 0:00:14
✅ ended    50% [2/4]
⭐ running  50% [2/4]
   ✅ f/185e9a70bf662355983d79881877056a 0:00:05 f(x=0)
   ✅ f/216ec88e5485e1ae439c37d3f94cab8f 0:00:05 f(x=1)
   ⭐ f/4e17f7f9fc6073c3d511bc12c7b6722a 0:00:02 f(x=2)
   ⭐ f/1853daf3a769344e10cfcbdef036a8c9 0:00:02 f(x=3)

Quick start

Pantarei builds on four kinds of execution units:

• functions: stateless, Python callables
• tasks: wrapped functions that cache execution results
• threads: wrapped tasks for batch execution in /shared-memory/ parallel environments
• jobs: wrapped tasks for batch execution in /distributed-memory/ parallel environments

As a catch-all, it also provides a parallel execution unit that uses whatever scheduling system is available on your environment, defaulting to shared-memory parallelism.

To see it in action, say you have a Python function

def f(x):
    import time
    time.sleep(2)
    return x

Wrap the function with a Task and call it with a range of arguments

from pantarei import Task

task = Task(f)
for x in [1, 2]:
    task(x=x)

The task's results are cached: a successive execution will just fetch the results (like joblib)

results = task(x=1)

We wrap the task with Parallel and submit its execution to a local scheduler, such as SLURM, or to a multi-threading scheduler as fallback

from pantarei import Parallel

job = Parallel(task)
for x in [3, 4]:
    job(x=x)

If you want to get the jobs' results, wait until they are done

job.scheduler.wait()
results = job(x=3)

Command line interface

Pantarei comes with a command line interface to run and manage jobs. If you like working from the terminal, you'll find yourself at home.

Run a script with some jobs

rei run script.py

Check the status of the jobs

rei ls -l script.py

🟩 script.py |████████████████████| 0:00:00 [2/2]
   ✅ f/3de5a949fa3c880e35165fc6820ce82e 0:00:00 f(x=1)
   ✅ f/26766c2fda253b7aeb1adaa02f31e93b 0:00:00 f(x=2)

Inspect the jobs' cache (metadata and results)

rei cat script.py

Clear the jobs' cache and artifacts

rei rm -rf script.py

There is much more of course: rei --help and rei <command> --help are your best friends.

Documentation

Check out the tutorial for more examples and the public API for full details.

The CLI interface (rei) is documented via its own help pages.

Installation

From pypi

pip install pantarei

TODO

• submit on remote cluster
• handle task dependencies
• add Workflow / Queue
• perhaps add signac-like view() or checkout() method to check out a view of cache as folders

Contributing

Contributions to the project are welcome. If you wish to contribute, check out these guidelines.

Authors

• Daniele Coslovich