daguflow
Write dagu workflows in Python, the way Airflow's TaskFlow does it:
@dag and @task decorators, and context managers for sub-DAGs and loops. A @task is a Python function
that runs when the step runs. Its arguments arrive as JSON and its return value becomes the step's output,
so passing one task's result to another is the dependency. daguflow build writes the YAML dagu schedules.
Each file is checked against the schema of the dagu version daguflow pins (v2.24.0), so a typo fails at
build time, not when dagu loads the file.
$ pip install daguflow # or: uv add daguflow
from typing import Annotated
from daguflow import ITEM, Cron, Range, SubDAG, dag, task
HUB = "http://192.168.1.60:8080"
@task
def list_lights(hub: str) -> list:
import json, urllib.request
with urllib.request.urlopen(f"{hub}/lights", timeout=10) as resp:
return [light["name"] for light in json.load(resp) if light["reachable"]]
@task
def dim(hub: str, light: str, level: int) -> dict:
import json, urllib.request
with urllib.request.urlopen(f"{hub}/lights/{light}", timeout=10) as resp:
now = json.load(resp)["brightness"]
if now <= level:
return {"light": light, "dimmed": False, "brightness": now}
req = urllib.request.Request(f"{hub}/lights/{light}", method="PUT",
data=json.dumps({"brightness": level}).encode(),
headers={"Content-Type": "application/json"})
urllib.request.urlopen(req, timeout=10).close()
return {"light": light, "dimmed": True, "from": now, "brightness": level}
@dag(schedule=Cron("30 22 * * *", tz="America/Los_Angeles"), labels=["lights"])
def dim_each(level: Annotated[int, Range(0, 100)] = 20):
"""Lower any light brighter than `level`, four at a time."""
found = list_lights(HUB)
with SubDAG("one", items=found.output, max_concurrent=4,
params={"light": ITEM, "level": level}) as one:
dim(HUB, one.params.light, one.params.level)
$ daguflow build workflows -o dags/ # one YAML file per DAG
$ daguflow build workflows -o dags/ --check # exit 1 with a diff if dags/ is stale
How it maps to dagu
| Python | dagu YAML |
|---|---|
@dag function / with DAG("name"): |
one file, <name>.yaml (the filename is the name) |
| function parameters | params:: plain defaults give - value: 50; Literal, Annotated[.., Range/Pattern/Length/Doc], Param(...) or no default give rich definitions dagu enforces at start |
| first docstring line / the rest | description: / the file's header comment |
@task call, @task.bash call, actions.x.y(...).step(), shell(...), step(...), target.run(...) |
a step |
a >> b, [a, b] >> c, chain(a, b, c) |
depends: |
| reading another step's output | depends: added for you (dagu won't infer it) |
with SubDAG("x"): |
a --- local DAG in the same file and a dag.run step; items= adds parallel: |
with ForEach(items): |
an inline foreach: step |
call_dag("other") |
action: dag.run against another file |
with on_failure(): (and on_success/exit/abort/init/wait) |
handler_on: |
Cron("0 9 * * *", tz=...) |
CRON_TZ=... 0 9 * * * |
Every other dagu key is passed by its own name, e.g. Step(..., repeat_policy={...}) or
DAG(..., max_active_steps=4), and checked against the schema. A few Python conveniences translate:
timeout=: seconds or atimedelta, becomestimeout_sec.retries=withretry_interval=(default 5s): dagu requires an interval for anyretry_policy.env=: a dict, becomes dagu's ordered list.tags=: becomeslabels.
References and quoting
Refs are build-time stand-ins for dagu's ${...}. Format them in f-strings:
| Python | renders |
|---|---|
f"{effect}" |
${effect} |
f"{effect:q}" |
"${effect}" |
f"{value:bare}" |
$value |
step.output["field"] |
${STEP.field} |
env.HOME |
${env.HOME} |
ctx.run.id |
${context.run.id} |
ITEM |
${ITEM} |
var("NAME") |
${NAME} |
These matter in the places a step is a shell command: @task.bash, shell() and SSHTarget.run(). A
@task never needs them: its arguments arrive as values.
Use {x:q} for free text in SSH commands. For an SSH target, dagu replaces a double-quoted "${x}" with the
value re-quoted as one shell word, so Rock 'n' Roll (Live) & More arrives as one argument. Unquoted, the
remote shell splits it. Single-quoted, dagu leaves it unexpanded. Measured on dagu 2.16.2 and 2.24.0
(tests/test_ssh_integration.py).
SSH with: fields expand only scoped references. Write user=env.PI_USER, never "${PI_USER}".
@task and @task.bash
@taskis Python that runs when the step runs. The function's own source ships to wherever the step runs: the dagu worker (run:), an SSH host (runs_on=SSHTarget(...)), or a container (runs_on=container(image)).- Arguments go in as JSON. The return value comes back as JSON on stdout and becomes the step's output.
- The function's own
prints go to stderr, so they can't corrupt that output. - Passing another step (or its output) as an argument makes it upstream.
- Decorator kwargs are step options (
output=,retries=,timeout=,continue_on=, ...), andtask.override(name=...)changes them for one call. Calling a task twice names the second stepname__1, as in Airflow. deps=["httpx"]runs it throughuv run --with.- The function must be self-contained: imports inside the body, no closures, no module globals. Build fails and names any outside names it uses. Pass configuration as arguments, or import your own package where the step runs.
- Python must exist where the step runs. dagu's official image has none: build
FROMit withpython3added, run the step in a container, or send it to a host that has Python. - How run-time values arrive:
- Locally, through step
env:, so values are never spliced into code. - Over SSH, each on its own file descriptor as a quoted heredoc, so quotes,
$and JSON arrive byte for byte. A value containing the lineDAGUFLOW_EOFends its heredoc early, and the rest runs as shell on the host. So build accepts only params dagu constrains: anenum, a numeric type, or apatternthat admits no newline. Step outputs, env and unconstrained params are refused;unsafe_args=Trueoverrides, for values whose every source you trust.
- Locally, through step
@task.bashis for steps whose job really is a shell command. The function runs at build time and returns the command string (or an argv list); anything else is refused.runs_on=works as for@task.
Actions
daguflow.actions has a typed, keyword-only builder for every builtin action at the pinned dagu version,
generated from its schema (tools/gen_actions.py):
from daguflow import actions
actions.http.request(method="GET", url="https://example.com", timeout=10).step("fetch", output="PAGE")
actions.jq.filter(filter=".items[]", data=page.output).step("items")
actions.custom("dagucloud/duckdb@v1", query="select 1").step("duck")
- Required keys are enforced, and keys dagu would reject raise
TypeError. - Python keywords take a trailing underscore:
from_=,import_=. SSHTarget(...)(alsoactions.ssh.Target) writes SSH settings once. Use.run(cmd)for a step, as@task(runs_on=...), or as a DAG'sssh=default.- A DAG-level
ssh=moves every step without an action onto that host, plainrun:steps included.
CLI
daguflow build TARGET... -o DIRwrites one YAML per DAG. TARGET is a module, a package directory or a.pyfile.--checkwrites nothing and exits 1 with a diff when DIR is stale.--prunedeletes files daguflow generated that no DAG produces any more. Hand-written files are left alone.
daguflow validate FILE...checks the schema plus dagu's document rules.--dagu-bin dagualso runsdagu validate.--server URLalso asks a running dagu's/dags/validate.
daguflow run DAG --server URL -p k=v --waitstarts a run and prints its outputs.--inline MODULEsends the DAG's YAML without a file on the server.daguflow schema shownames the pinned schema and its cache;daguflow schema fetch [--out PATH]downloads it ahead of an offline build.daguflow schema update --version vX.Y.Z, in a daguflow checkout, repins another dagu release. Then runmise run genand the tests.
The dagu schema
daguflow validates against the schema of one dagu release, pinned by commit and sha256, but does not ship
it: dagu is GPL-3.0-or-later and daguflow is MIT. The first build downloads it from dagu's repository,
checks the hash and caches it in $XDG_CACHE_HOME/daguflow (~/.cache/daguflow). Offline, run
daguflow schema fetch while you still have a network, or set DAGUFLOW_SCHEMA to a copy; either way the
hash is checked.
Client
from daguflow.client import Client
with Client("http://127.0.0.1:8080") as dagu:
run_id = dagu.start("set_light_effect", params={"effect": "Good Vibes"})
run = dagu.wait("set_light_effect", run_id)
print(run.status, dagu.outputs("set_light_effect", run_id, decode=True))
Covered calls:
validate,run_inline,start,enqueue,wait,status,outputs,step_log,run_log,stop,retry.list_dags,list_runs,sub_runs,suspend,create_dag/update_spec/delete_dag.request()for anything else.
API quirks the client absorbs:
- Params go as a JSON object string.
- The run-detail wrapper key differs by endpoint (
dagRunDetailsvsdagRun). - Outputs come back as strings under camelCased keys (
output: RESULT→result).
Development
mise run test runs everything in a throwaway isb sandbox:
- the unit tests;
- golden YAML for
examples/; dagu validateon every golden file, using the pinned release binary;- the examples running on a real dagu server, and SSH steps against a throwaway sshd.
Without isb: uv run --extra test pytest. The integration tier runs when DAGUFLOW_DAGU_BIN points at
the pinned dagu binary (tools/fetch-dagu.sh downloads it).
mise run gen regenerates the action builders. A test fails when they drift from the schema.
License
MIT. dagu's schema, which daguflow downloads at build time, is GPL-3.0-or-later and stays dagu's.
Metadata
Release files for daguflow 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| daguflow-0.2.0.tar.gz | 44.4 kB | Without provenance | Details for daguflow-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| daguflow-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for daguflow-0.2.0-py3-none-any.whl |
Total release size: 95.4 kB
Release files / daguflow-0.2.0.tar.gz
| Download URL | daguflow-0.2.0.tar.gz |
|---|---|
| Size | 44.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a94754dc249d3404a6b8d83b28a424fb5bbb8a3fc8d5cce3d3b738912b4d4b81
|
|
BLAKE2b-256 checksum How to use checksums |
0d5cd6178f949c0e952fcaaf5f683df0df82fb8af018627cd0554786360e73e1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.4 {"installer":{"name":"uv","version":"0.13.4","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / daguflow-0.2.0-py3-none-any.whl
| Download URL | daguflow-0.2.0-py3-none-any.whl |
|---|---|
| Size | 51.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9b298056922f2088e7469776f976c9f44b884fa2404e0f3d818e4401f97b9319
|
|
BLAKE2b-256 checksum How to use checksums |
243a86932577b69e578870440ca21f3e3f8c01d8a00815ad4ad9caea48c8a874
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.4 {"installer":{"name":"uv","version":"0.13.4","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for daguflow 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.