Skip to main content

daguflow

Write dagu workflows in Python, the way Airflow's TaskFlow does it: @dag and @task decorators, and context managers for sub-DAGs and loops. A @task is a Python function that runs when the step runs. Its arguments arrive as JSON and its return value becomes the step's output, so passing one task's result to another is the dependency. daguflow build writes the YAML dagu schedules. Each file is checked against the schema of the dagu version daguflow pins (v2.24.0), so a typo fails at build time, not when dagu loads the file.

$ pip install daguflow        # or: uv add daguflow

Every morning, fetch Hacker News' top stories, eight at a time, and rank the ones worth reading:

from daguflow import Cron, dag, task


@task
def top_ids(api: str, limit: int) -> list:
    import json, urllib.request
    with urllib.request.urlopen(f"{api}/topstories.json", timeout=10) as r:
        return json.load(r)[:limit]


@task(retries=3)
def story(api: str, story_id: int) -> dict:
    import json, urllib.request
    with urllib.request.urlopen(f"{api}/item/{story_id}.json", timeout=10) as r:
        s = json.load(r)
    return {"title": s["title"], "score": s["score"], "comments": s.get("descendants", 0)}


@task
def digest(stories: list, min_score: int) -> str:
    hot = sorted((s for s in stories if s["score"] >= min_score), key=lambda s: -s["score"])
    return "\n".join(f"{s['score']:>5}  {s['title']}" for s in hot)


@dag(schedule=Cron("0 8 * * *"))
def hn_digest(limit: int = 30, min_score: int = 100, api: str = "https://hacker-news.firebaseio.com/v0"):
    """This morning's Hacker News, best first."""
    ids = top_ids(api, limit)
    stories = story.partial(api=api).expand(story_id=ids, max_concurrent=8)
    digest(stories, min_score)

story.partial(api=api).expand(story_id=ids) runs story once per ID, eight at a time, and digest receives every story as a list. Each run is its own process, so one slow or failed fetch retries alone and shows in dagu's UI.

$ daguflow build workflows -o dags/          # one YAML file per DAG
$ daguflow build workflows -o dags/ --check  # exit 1 with a diff if dags/ is stale

How it maps to dagu

Python dagu YAML
@dag function / with DAG("name"): one file, <name>.yaml (the filename is the name)
function parameters params:: plain defaults give - value: 50; Literal, Annotated[.., Range/Pattern/Length/Doc], Param(...) or no default give rich definitions dagu enforces at start
first docstring line / the rest description: / the file's header comment
@task call, @task.bash call, actions.x.y(...).step(), shell(...), step(...), target.run(...) a step
a >> b, [a, b] >> c, chain(a, b, c) depends:
reading another step's output depends: added for you (dagu won't infer it)
task.expand(x=...), task.partial(...).expand(...), task.expand_kwargs(...) a parallel dag.run step over a one-step local DAG
with SubDAG("x"): a --- local DAG in the same file and a dag.run step; items= with as_= adds parallel:, and sub.results gives a @task every child's result as a list
with ForEach(items): an inline foreach: step
call_dag("other") action: dag.run against another file
with on_failure(): (and on_success/exit/abort/init/wait) handler_on:
Cron("0 9 * * *", tz=...) CRON_TZ=... 0 9 * * *

Every other dagu key is passed by its own name, e.g. Step(..., repeat_policy={...}) or DAG(..., max_active_steps=4), and checked against the schema. A few Python conveniences translate:

  • timeout=: seconds or a timedelta, becomes timeout_sec.
  • retries= with retry_interval= (default 5s): dagu requires an interval for any retry_policy.
  • env=: a dict, becomes dagu's ordered list.
  • tags=: becomes labels.

References and quoting

Refs are build-time stand-ins for dagu's ${...}. Format them in f-strings:

Python renders
f"{effect}" ${effect}
f"{effect:q}" "${effect}"
f"{value:bare}" $value
step.output["field"] ${STEP.field}
env.HOME ${env.HOME}
ctx.run.id ${context.run.id}
var("NAME") ${NAME}

These matter in the places a step is a shell command: @task.bash, shell() and SSHTarget.run(). A @task never needs them: its arguments arrive as values.

Use {x:q} for free text in SSH commands. For an SSH target, dagu replaces a double-quoted "${x}" with the value re-quoted as one shell word, so Rock 'n' Roll (Live) & More arrives as one argument. Unquoted, the remote shell splits it. Single-quoted, dagu leaves it unexpanded. Measured on dagu 2.16.2 and 2.24.0 (tests/test_ssh_integration.py).

SSH with: fields expand only scoped references. Write user=env.PI_USER, never "${PI_USER}".

@task and @task.bash

  • @task is Python that runs when the step runs. The function's own source ships to wherever the step runs: the dagu worker (run:), an SSH host (runs_on=SSHTarget(...)), or a container (runs_on=container(image)).
    • Arguments go in as JSON. The return value comes back as JSON on stdout and becomes the step's output.
    • The function's own prints go to stderr, so they can't corrupt that output.
    • Passing another step (or its output) as an argument makes it upstream.
    • Decorator kwargs are step options (output=, retries=, timeout=, continue_on=, ...), and task.override(name=...) changes them for one call. Calling a task twice names the second step name__1, as in Airflow.
    • deps=["httpx"] runs it through uv run --with.
    • The function must be self-contained: imports inside the body, no closures, no module globals. Build fails and names any outside names it uses. Pass configuration as arguments, or import your own package where the step runs.
    • Python must exist where the step runs. dagu's official image has none: build FROM it with python3 added, run the step in a container, or send it to a host that has Python.
    • How run-time values arrive:
      • Locally, through step env:, so values are never spliced into code.
      • Over SSH, each on its own file descriptor as a quoted heredoc, so quotes, $ and JSON arrive byte for byte. A value containing the line DAGUFLOW_EOF ends its heredoc early, and the rest runs as shell on the host. So build accepts only params dagu constrains: an enum, a numeric type, or a pattern that admits no newline. Step outputs, env and unconstrained params are refused; unsafe_args=True overrides, for values whose every source you trust.
  • @task.bash is for steps whose job really is a shell command. The function runs at build time and returns the command string (or an argv list); anything else is refused. runs_on= works as for @task.

Mapping: .expand()

As in Airflow, .expand() runs a task once per item, in parallel, and the next task gets the results as a list, in item order:

@task
def resize(path: str, width: int) -> str: ...

@task
def manifest(paths: list) -> dict: ...

@dag
def thumbnails(width: int = 320):
    paths = list_images("s3://photos/new")                       # a @task returning a list
    small = resize.partial(width=width).expand(path=paths, max_concurrent=4)
    manifest(small)
  • expand(arg=...) takes a list, or a step whose output is a JSON list.
  • Several mapped arguments run every combination: cell.expand(row=["a", "b"], col=[1, 2]) is four runs. They take literal lists.
  • partial(...) fixes the arguments every run shares. Constants are built into the child; DAG params and step outputs are passed to it.
  • expand_kwargs(dicts) runs once per dict of keyword arguments, from a literal list or from a step. A key a dict leaves out takes the function's default.
  • max_concurrent= caps the parallel runs (dagu's default is 10, and its limit 1,000 items).
  • dagu runs identical items once, so expand(x=[1, 1, 2]) is two runs and two results.
  • Underneath, each .expand() is a one-step local sub-DAG run by a parallel dag.run step. For a child with several steps, write the SubDAG yourself: with SubDAG("each", items=paths, as_="path") as each:, use each.params.path inside, and pass each.results on.

Actions

daguflow.actions has a typed, keyword-only builder for every builtin action at the pinned dagu version, generated from its schema (tools/gen_actions.py):

from daguflow import actions
actions.http.request(method="GET", url="https://example.com", timeout=10).step("fetch", output="PAGE")
actions.jq.filter(filter=".items[]", data=page.output).step("items")
actions.custom("dagucloud/duckdb@v1", query="select 1").step("duck")
  • Required keys are enforced, and keys dagu would reject raise TypeError.
  • Python keywords take a trailing underscore: from_=, import_=.
  • SSHTarget(...) (also actions.ssh.Target) writes SSH settings once. Use .run(cmd) for a step, as @task(runs_on=...), or as a DAG's ssh= default.
  • A DAG-level ssh= moves every step without an action onto that host, plain run: steps included.

CLI

  • daguflow build TARGET... -o DIR writes one YAML per DAG. TARGET is a module, a package directory or a .py file.
    • --check writes nothing and exits 1 with a diff when DIR is stale.
    • --prune deletes files daguflow generated that no DAG produces any more. Hand-written files are left alone.
  • daguflow validate FILE... checks the schema plus dagu's document rules.
    • --dagu-bin dagu also runs dagu validate.
    • --server URL also asks a running dagu's /dags/validate.
  • daguflow run DAG --server URL -p k=v --wait starts a run and prints its outputs. --inline MODULE sends the DAG's YAML without a file on the server.
  • daguflow schema show names the pinned schema and its cache; daguflow schema fetch [--out PATH] downloads it ahead of an offline build.
  • daguflow schema update --version vX.Y.Z, in a daguflow checkout, repins another dagu release. Then run mise run gen and the tests.

The dagu schema

daguflow validates against the schema of one dagu release, pinned by commit and sha256. The first build downloads it from dagu's repository and caches it in $XDG_CACHE_HOME/daguflow (~/.cache/daguflow). To build offline, run daguflow schema fetch once while online, or set DAGUFLOW_SCHEMA to a copy; either way the hash is checked.

Client

from daguflow.client import Client
with Client("http://127.0.0.1:8080") as dagu:
    run_id = dagu.start("set_light_effect", params={"effect": "Good Vibes"})
    run = dagu.wait("set_light_effect", run_id)
    print(run.status, dagu.outputs("set_light_effect", run_id, decode=True))

Covered calls:

  • validate, run_inline, start, enqueue, wait, status, outputs, step_log, run_log, stop, retry.
  • list_dags, list_runs, sub_runs, suspend, create_dag / update_spec / delete_dag.
  • request() for anything else.

API quirks the client absorbs:

  • Params go as a JSON object string.
  • The run-detail wrapper key differs by endpoint (dagRunDetails vs dagRun).
  • Outputs come back as strings under camelCased keys (output: RESULT → result).

Development

mise run test runs everything in a throwaway isb sandbox:

  • the unit tests;
  • golden YAML for examples/;
  • dagu validate on every golden file, using the pinned release binary;
  • the examples running on a real dagu server, and SSH steps against a throwaway sshd.

Without isb: uv run --extra test pytest. The integration tier runs when DAGUFLOW_DAGU_BIN points at the pinned dagu binary (tools/fetch-dagu.sh downloads it).

mise run gen regenerates the action builders. A test fails when they drift from the schema.

License

MIT.

Metadata

Release files for daguflow 0.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for daguflow 0.4.0
File Size Uploaded Provenance Actions
daguflow-0.4.0.tar.gz 47.8 kB Without provenance Details for daguflow-0.4.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for daguflow 0.4.0
File Interpreter ABI Platform Actions
daguflow-0.4.0-py3-none-any.whl Python 3 none any Details for daguflow-0.4.0-py3-none-any.whl

Total release size: 102.4 kB

Release files / daguflow-0.4.0.tar.gz

Download URL daguflow-0.4.0.tar.gz
Size 47.8 kB
Tags Source
SHA-256 checksum
How to use checksums
6feb41d8325b91ff17f65bc6e1e9ddb19ee66617996261b798c1e1b738c340e8
BLAKE2b-256 checksum
How to use checksums
167163ffd5b832200d1c34f9be7fff385442a3f59438abd7de5a13b5591a7938
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.4 {"installer":{"name":"uv","version":"0.13.4","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / daguflow-0.4.0-py3-none-any.whl

Download URL daguflow-0.4.0-py3-none-any.whl
Size 54.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c46feb46cc71ba16d05c8a245758363561f4ec5314c6b7239bdc8f3040d9167a
BLAKE2b-256 checksum
How to use checksums
7a63dc84d48fd0c6c4ff7f1ca525ab640b5704ce2aeef8d46cb5b024099687b6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.4 {"installer":{"name":"uv","version":"0.13.4","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for daguflow 0.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page