Skip to main content

adag

pipeline status PyPI version Python versions License: MIT

A DAG engine for Python data and ML pipelines with a content-addressed cache. A step's cache key is a hash of its fields, its inputs, and its source code. Change any of them and that step and everything downstream recompute. Nothing else does.

Install

pip install adag

Requires Python 3.12 or newer. CI runs the tests on 3.12 and 3.13.

Quickstart

Each step is a frozen dataclass. Call a step to wire it into the graph, then run the graph with a Pipeline.

from dataclasses import dataclass
from pathlib import Path

from pydantic import BaseModel

from adag import InputParam, Pipeline, Step


class Number(BaseModel):
    value: int


@dataclass(frozen=True)
class Square(Step[Number]):
    def run(self, x: int) -> Number:
        print("computing square")
        return Number(value=x * x)


@dataclass(frozen=True)
class AddOne(Step[Number]):
    def run(self, n: Number) -> Number:
        return Number(value=n.value + 1)


x = InputParam[int]("x")
total = AddOne()(n=Square()(x=x))
pipeline = Pipeline(total, cache_dir=Path("cache"))

print(pipeline.execute(x=4))  # computes both steps
print(pipeline.execute(x=4))  # served from the cache

The second run reads both results from cache/ and does not call Square.run again:

computing square
value=17
value=17

(output trimmed: structlog lines removed)

Why adag

Use it for data and ML pipelines where steps are slow and you rerun the graph often. How it compares to Luigi, Airflow, Prefect, Dagster, Hamilton and redun: Why adag.

Key features

  • Content-addressed cache: a code edit invalidates only the steps it touches.
  • Config fields stay out of the cache key (worker counts, clients, timeouts).
  • Sync and async steps (Step, AsyncStep), with a parallel executor.
  • BatchStep caches each item on its own, so a rerun resumes after a failure.
  • FanOutStep runs a sub-graph per key.
  • Cache stores for step results and BatchStep items: filesystem, memory, SQLite, SQLAlchemy (adag[sql]), Redis (adag[redis]). Pass one with Pipeline(..., store=).
  • explain_key lists every part of a cache key, so you can see why a step ran again.
  • save_result and load_result cache a result in your own format (JSONL, a directory).
  • Save a pipeline to disk and load it back with identical hashes.
  • SVG rendering of the graph, and an adag run pipeline.py CLI.
  • Run fan-out branches in worker processes or on a job scheduler (AWS Batch, SLURM, Kubernetes): write submit, status and cancel; adag does retries, timeouts and cancel.

Contributing

Developer setup, the dev loop, and the release process live in CONTRIBUTING.md. In short: uv sync && make install-hooks, then make check runs the same gate CI runs.

License

MIT © 2026 Jorge Cardona (see LICENSE).

Metadata

Release files for adag 0.3.12

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for adag 0.3.12
File Size Uploaded Provenance Actions
adag-0.3.12.tar.gz 168.1 kB Without provenance Details for adag-0.3.12.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for adag 0.3.12
File Interpreter ABI Platform Actions
adag-0.3.12-py3-none-any.whl Python 3 none any Details for adag-0.3.12-py3-none-any.whl

Total release size: 280.6 kB

Release files / adag-0.3.12.tar.gz

Download URL adag-0.3.12.tar.gz
Size 168.1 kB
Tags Source
SHA-256 checksum
How to use checksums
944be424014fc41ec4b31186ec5331f7804977e626910a2ff299985dd14eddb8
BLAKE2b-256 checksum
How to use checksums
82b0848ff5dcf503a78a3beb29386fc356142c91ef36ae3136f4a592ddb002ad
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / adag-0.3.12-py3-none-any.whl

Download URL adag-0.3.12-py3-none-any.whl
Size 112.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
40443dcff68dbdebe597aca10ad8b63d4c85d0ccd11d03951cfd9141540d1b65
BLAKE2b-256 checksum
How to use checksums
d325d22b4eca8ffa6cf85f58fc8623b890741867642224121eed4a0a70c1aacd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for adag 0.3.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.3.12 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page