Skip to main content

adag

pipeline status PyPI version Python versions License: MIT

A DAG engine for Python data and ML pipelines with a content-addressed cache. A step's cache key is a hash of its fields, its inputs, and its source code. Change any of them and that step and everything downstream recompute. Nothing else does.

Install

pip install adag

Requires Python 3.12 or newer. CI runs the tests on 3.12 and 3.13.

Quickstart

Each step is a frozen dataclass. Call a step to wire it into the graph, then run the graph with a Pipeline.

from dataclasses import dataclass
from pathlib import Path

from pydantic import BaseModel

from adag import InputParam, Pipeline, Step


class Number(BaseModel):
    value: int


@dataclass(frozen=True)
class Square(Step[Number]):
    def run(self, x: int) -> Number:
        print("computing square")
        return Number(value=x * x)


@dataclass(frozen=True)
class AddOne(Step[Number]):
    def run(self, n: Number) -> Number:
        return Number(value=n.value + 1)


x = InputParam[int]("x")
total = AddOne()(n=Square()(x=x))
pipeline = Pipeline(total, cache_dir=Path("cache"))

print(pipeline.execute(x=4))  # computes both steps
print(pipeline.execute(x=4))  # served from the cache

The second run reads both results from cache/ and does not call Square.run again:

computing square
value=17
value=17

(output trimmed: structlog lines removed)

Why adag

Use it for data and ML pipelines where steps are slow and you rerun the graph often. How it compares to Luigi, Airflow, Prefect, Dagster, Hamilton and redun: Why adag.

Key features

  • Content-addressed cache: a code edit invalidates only the steps it touches.
  • Config fields stay out of the cache key (worker counts, clients, timeouts).
  • Sync and async steps (Step, AsyncStep), with a parallel executor.
  • BatchStep caches each item on its own, so a rerun resumes after a failure.
  • FanOutStep runs a sub-graph per key.
  • Cache stores for step results and BatchStep items: filesystem, memory, SQLite, SQLAlchemy (adag[sql]), Redis (adag[redis]). Pass one with Pipeline(..., store=).
  • explain_key lists every part of a cache key, so you can see why a step ran again.
  • save_result and load_result cache a result in your own format (JSONL, a directory).
  • Save a pipeline to disk and load it back with identical hashes.
  • SVG rendering of the graph, and an adag run pipeline.py CLI.
  • Run fan-out branches in worker processes or on a job scheduler (AWS Batch, SLURM, Kubernetes): write submit, status and cancel; adag does retries, timeouts and cancel.

Contributing

Developer setup, the dev loop, and the release process live in CONTRIBUTING.md. In short: uv sync && make install-hooks, then make check runs the same gate CI runs.

License

MIT © 2026 Jorge Cardona (see LICENSE).

Metadata

Release files for adag 0.3.9

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for adag 0.3.9
File Size Uploaded Provenance Actions
adag-0.3.9.tar.gz 163.8 kB Without provenance Details for adag-0.3.9.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for adag 0.3.9
File Interpreter ABI Platform Actions
adag-0.3.9-py3-none-any.whl Python 3 none any Details for adag-0.3.9-py3-none-any.whl

Total release size: 275.3 kB

Release files / adag-0.3.9.tar.gz

Download URL adag-0.3.9.tar.gz
Size 163.8 kB
Tags Source
SHA-256 checksum
How to use checksums
cc4fbb0469ef86c9c9d4e109a95eabf7fbbc37c1ae1606917fe3edbd6fa148b2
BLAKE2b-256 checksum
How to use checksums
3ef49391ba0b7ec6baa12494d18c741c89c5bc8a1dff7e6dba557284f335ecda
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / adag-0.3.9-py3-none-any.whl

Download URL adag-0.3.9-py3-none-any.whl
Size 111.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
076c70a9200176c0a4f0ba744dfde9850ee352ff4c74962a12ae3f3910ab71ec
BLAKE2b-256 checksum
How to use checksums
4495ce2b3fa4f9ab2691a03e177b543bc4193b0ed8710110117e578e0ff78aa0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for adag 0.3.9

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.3.9 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page