Directed Acyclic Workflow Graph Scheduling
dawgz provides a lightweight and intuitive Python interface to declare, schedule and execute job workflows. It can also delegate execution to resource management backends such as Slurm, which means you can write, configure, and submit your workflows without ever leaving Python.
Installation
The dawgz package is available on PyPi and can be installed with pip.
pip install dawgz
Alternatively, if you need the latest features, you can install it from source.
pip install git+https://github.com/francois-rozet/dawgz
Getting started
In dawgz, a job is a Python function. The dawgz.job decorator allows to declare the resources a job requires and capture its arguments. A job's dependencies within the workflow are declared with the dawgz.Job.after method. After declaration, the dawgz.schedule function takes care of scheduling the jobs and their dependencies, with a selected execution backend. For more information, check out the interface and the examples.
Follows a small example demonstrating how one could use dawgz to calculate π (very roughly) using the Monte Carlo method. We define two jobs, generate and estimate. Five instances of generate are declared that will be executed concurrently. The estimate job has all generate instances as dependencies, meaning that it will only start after they have completed successfuly.
import dawgz
import glob
import numpy as np
@dawgz.job(cpus=1, ram="2GB", time="5:00")
def generate(i: int):
print(f"Task {i + 1}")
x = np.random.random(10000)
y = np.random.random(10000)
within_circle = x**2 + y**2 <= 1
np.save(f"pi_{i}.npy", within_circle)
@dawgz.job(cpus=2, ram="4GB", time="15:00")
def estimate():
files = glob.glob("pi_*.npy")
stack = np.vstack([np.load(f) for f in files])
pi_estimate = stack.mean() * 4
print(f"π ≈ {pi_estimate}")
if __name__ == "__main__":
generate_jobs = [generate(i) for i in range(5)]
estimate_job = estimate().after(*generate_jobs)
dawgz.schedule(estimate_job, backend="async")
$ python examples/pi.py
Task 1
Task 2
Task 3
Task 4
Task 5
π ≈ 3.14936
Alternatively, on a Slurm HPC cluster, changing the backend to "slurm" results in the following job queue.
$ python examples/pi.py
$ squeue -u username
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1868833 all 0005_estimate username PD 0:00 1 (Dependency)
1868832 all 0004_generate username PD 0:00 1 (Resources)
1868831 all 0003_generate username PD 0:00 1 (Resources)
1868830 all 0002_generate username PD 0:00 1 (Resources)
1868828 all 0000_generate username R 0:02 1 node-x
1868829 all 0001_generate username R 0:01 1 node-y
In addition to the Python interface, dawgz provides a simple command-line interface to list the scheduled workflows, the jobs of a workflow or the output of a job.
$ dawgz
╭────┬───────┬──────────────────────────┬─────────────────────┬─────────┬──────┬────────╮
│ │ Name │ ID │ Date │ Backend │ Jobs │ Errors │
├────┼───────┼──────────────────────────┼─────────────────────┼─────────┼──────┼────────┤
│ 0 │ pi.py │ handsome_jicama_bfc5a3e4 │ 2022-02-28 16:37:58 │ async │ 6 │ 0 │
│ 1 │ pi.py │ crowded_machine_23bdd047 │ 2022-02-28 17:18:33 │ slurm │ 6 │ 0 │
╰────┴───────┴──────────────────────────┴─────────────────────┴─────────┴──────┴────────╯
$ dawgz 1
╭────┬──────────┬───────────┬─────────╮
│ │ Job │ State │ ID |
├────┼──────────┼───────────┼─────────┤
│ 0 │ generate │ COMPLETED │ 1868828 |
│ 1 │ generate │ RUNNING │ 1868829 |
│ 2 │ generate │ RUNNING │ 1868830 |
│ 3 │ generate │ PENDING │ 1868831 |
│ 4 │ generate │ PENDING │ 1868832 |
│ 5 │ estimate │ PENDING │ 1868833 |
╰────┴──────────┴───────────┴─────────╯
$ dawgz 1 2
╭────┬──────────┬───────────┬────────╮
│ │ Job │ State │ Logs │
├────┼──────────┼───────────┼────────┤
│ 2 │ generate │ COMPLETED │ Task 3 │
╰────┴──────────┴───────────┴────────╯
$ dawgz 1 2 --raw
Task 3
$ dawgz 1 2 --source
╭────┬──────────┬───────────┬────────────────────────────────────────────╮
│ │ Job │ State │ Source │
├────┼──────────┼───────────┼────────────────────────────────────────────┤
│ 2 │ generate │ COMPLETED │ @dawgz.job(cpus=1, ram="2GB", time="5:00") │
│ │ │ │ def generate(i: int) -> None: │
│ │ │ │ print(f"Task {i + 1}") │
│ │ │ │ │
│ │ │ │ x = np.random.random(10000) │
│ │ │ │ y = np.random.random(10000) │
│ │ │ │ within_circle = x**2 + y**2 <= 1 │
│ │ │ │ │
│ │ │ │ np.save(f"pi_{i}.npy", within_circle) │
╰────┴──────────┴───────────┴────────────────────────────────────────────╯
By default, dawgz stores the workflow files and logs in a .dawgz directory within the current working directory. Therefore, workflows are only visible to the dawgz command when it is executed at the location they were scheduled from. Setting the DAWGZ_DIR environment variable to a global path, such as ~/.dawgz, stores all files at the same place regardless of the current working directory, enabling dawgz to be called from anywhere.
export DAWGZ_DIR=~/.dawgz
If uv is installed, uvx dawgz runs the CLI without creating or activating any virtual environment. This is especially convenient for coding agents, in combination with the --raw flag to inspect job logs.
$ uvx dawgz --help
See dawgz --help for the full option list.
Interface
-
dawgz.jobregisters a function as a job, with its settings (name, resources, ...). When the wrapped function is called, its context and arguments are captured in adawgz.Jobinstance for later execution. In the following example,ais a job with the name"A", a time limit of one hour, and running onteslaorquadropartitions.a_jobis an instance ofa, with inputs3and0.14, but hasn't been executed yet.@dawgz.job(name="A", time="01:00:00", partition="tesla,quadro") def a(n: int, x: float): ... a_job = a(3, 0.14)
If the decorator syntax doesn't suit your needs, you can equivalently use
dawgz.jobas a function.def a(n: int, x: float): ... a_job = dawgz.job(a, name="A", time="01:00:00", partition="tesla,quadro")(3, 0.14)
To declare that a job must wait for another one to complete, you can use the
dawgz.Job.aftermethod. By default, the job waits for its dependencies to complete with success. The desired completion status can be set to"success"(default),"failure"or"any".@dawgz.job def b(): ... b_job = b().after(a_job, status="failure")
If a job has several dependencies, the
dawgz.Job.waitformethod can be used to declare whether it should wait for"all"(default) or"any"of them to be satisfied before starting.@dawgz.job def c(): ... c_job = c().after(a_job, b_job).waitfor("any")
When running the same workflow multiple times, you may want to skip jobs that have already been executed. You can mark these jobs as completed with the
dawgz.Job.markmethod, and they will be automatically pruned out of the workflow. The completion status can be set to"pending"(default),"success","failure"or"cancelled".@dawgz.job def d(): ... d_job = d().mark("success")
Modifying global variables after a job has been created will not affect its execution. However, the content of Python modules is not captured, which means that modifying a module after a job has been submitted can affect its execution. If this becomes an issue for you, you can register the module such that it is pickled by value rather than by reference upon job instantiation.
import cloudpickle import my_module cloudpickle.register_pickle_by_value(my_module) @dawgz.job def dummy(): my_module.my_function() # does not break, even if `my_function` disappears
However, this only works for pure Python modules and will lead to larger pickle files.
-
dawgz.arraycreates a job array from a group of independent jobs. The primary use case of job arrays is to schedule a large number of small jobs while throttling the number of simultaneously running jobs without saturating the Slurm queue. The returned object is itself adawgz.Jobinstance and supports the methods presented above (after,waitfor, ...).@dawgz.job def e(i): ... e_jobs = [e(i) for i in range(42)] e_array = dawgz.array(*e_jobs, throttle=3) e_array.after(d_job) dawgz.schedule(e_array, name="the answer is 42", backend="slurm")
-
dawgz.scheduleschedules a set of jobs, as well as their pending dependencies. Three backends are currently supported:async,dummyandslurm.asyncwaits asynchronously for dependencies to complete before executing each job. The jobs are executed by the current Python interpreter.dummyis equivalent toasync, but instead of executing the jobs, prints their name before and after a short (random) sleep time. The main use ofdummyis debugging.slurmsubmits the jobs to the Slurm workload manager by automatically generatingsbatchsubmission scripts.
Metadata
Release files for dawgz 2.7.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dawgz-2.7.1.tar.gz | 32.1 kB | Without provenance | Details for dawgz-2.7.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dawgz-2.7.1-py3-none-any.whl | Python 3 | none | any | Details for dawgz-2.7.1-py3-none-any.whl |
Total release size: 56.7 kB
Release files / dawgz-2.7.1.tar.gz
| Download URL | dawgz-2.7.1.tar.gz |
|---|---|
| Size | 32.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9a4d01580f2436b106fcccb2f607bcd1d16e0865db2fb0ed369a5bcd5a8b826f
|
|
BLAKE2b-256 checksum How to use checksums |
722b1a97667f3723bd11fe8be6d3f7013ab90562050852300acd8962151e3779
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / dawgz-2.7.1-py3-none-any.whl
| Download URL | dawgz-2.7.1-py3-none-any.whl |
|---|---|
| Size | 24.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bfaefbd294543c9942ddbffb30b575fbb237a4f94e60307ec8fdffe5e30db369
|
|
BLAKE2b-256 checksum How to use checksums |
30fe075104362d3fd7fd4f309cb452845ba0388240630824404f4ca7e5e2d8b4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for dawgz 2.7.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.