pydiverse.pipedag
A pipeline orchestration library executing tasks within one python session. It takes care of SQL table (de)materialization, caching and cache invalidation. Blob storage is supported as well for example for storing model files.
This is an early stage version 0.x, however, it is already used in real projects. We are happy to receive your feedback as issues on the GitHub repo. Feel free to also comment on existing issues to extend them to your needs or to add solution ideas.
Usage
pydiverse.pipedag can either be installed via pypi with pip install pydiverse-pipedag pydot or via
conda-forge with conda install pydiverse-pipedag pydot -c conda-forge. Our recommendation would be
to use pixi which is also based on conda-forge:
mkdir my_project
pixi init
pixi add pydiverse-pipedag pydot
With pixi, you run python like this:
pixi run python -c 'import pydiverse.pipedag'
or this:
pixi run python my_script.py
Example
A flow can look like this (i.e. put this in a file named run_pipeline.py):
import tempfile
import pandas as pd
import sqlalchemy as sa
from pydiverse.pipedag import Flow, Stage, Table, materialize
from pydiverse.pipedag.context import StageLockContext
from pydiverse.pipedag.core.config import create_basic_pipedag_config
from pydiverse.common.util.structlog import setup_logging
@materialize(lazy=True)
def lazy_task_1():
return sa.select(
sa.literal(1).label("x"),
sa.literal(2).label("y"),
)
@materialize(lazy=True, input_type=sa.Table)
def lazy_task_2(input1: sa.Alias, input2: sa.Alias):
query = sa.select(
(input1.c.x * 5).label("x5"),
input2.c.a,
).select_from(input1.outerjoin(input2, input2.c.x == input1.c.x))
return Table(query, name="task_2_out", primary_key=["a"])
@materialize(lazy=True, input_type=sa.Table)
def lazy_task_3(input1: sa.Alias):
return sa.text(f"SELECT * FROM {input1.original.schema}.{input1.original.name}")
@materialize(lazy=True, input_type=sa.Table)
def lazy_task_4(input1: sa.Alias):
return sa.text(f"SELECT * FROM {input1.original.schema}.{input1.original.name}")
@materialize(nout=2, version="1.0.0")
def eager_inputs():
dfA = pd.DataFrame(
{
"a": [0, 1, 2, 4],
"b": [9, 8, 7, 6],
}
)
dfB = pd.DataFrame(
{
"a": [2, 1, 0, 1],
"x": [1, 1, 2, 2],
}
)
return Table(dfA, "dfA"), Table(dfB, "dfB_%%")
@materialize(version="1.0.0", input_type=pd.DataFrame)
def eager_task(tbl1: pd.DataFrame, tbl2: pd.DataFrame):
return tbl1.merge(tbl2, on="x")
def main():
with tempfile.TemporaryDirectory() as temp_dir:
cfg = create_basic_pipedag_config(
f"duckdb:///{temp_dir}/db.duckdb",
disable_stage_locking=True, # This is special for duckdb
# Attention: If uncommented, stage and task names might be sent to the following URL.
# You can self-host kroki if you like:
# https://docs.kroki.io/kroki/setup/install/
# You need to install optional dependency 'pydot' for any visualization
# URL to appear.
# kroki_url="https://kroki.io",
).get("default")
with cfg:
with Flow() as f:
with Stage("stage_1"):
lazy_1 = lazy_task_1()
a, b = eager_inputs()
with Stage("stage_2"):
lazy_2 = lazy_task_2(lazy_1, b)
lazy_3 = lazy_task_3(lazy_2)
eager = eager_task(lazy_1, b)
with Stage("stage_3"):
lazy_4 = lazy_task_4(lazy_2)
_ = lazy_3, lazy_4, eager # unused terminal output tables
# Run flow
result = f.run()
assert result.successful
# Run in a different way for testing
with StageLockContext():
result = f.run()
assert result.successful
assert result.get(lazy_1, as_type=pd.DataFrame)["x"][0] == 1
if __name__ == "__main__":
setup_logging() # you can set up the logging and/or structlog libraries as you wish
main()
The with tempfile.TemporaryDirectory() is only needed to have an OS independent temporary directory available.
You can also get rid of it like this:
def main():
cfg = create_basic_pipedag_config(
"duckdb:////tmp/pipedag/{instance_id}/db.duckdb",
disable_stage_locking=True, # This is special for duckdb
).get("default")
...
For a more sophisticated setup with a pipedag.yaml configuration file and with a separate database
(i.e. containerized Postgres), please look here.
Troubleshooting
Installing mssql odbc driver for linux
Installing with
instructions here
worked.
But odbcinst -j revealed that it installed the configuration in /etc/unixODBC/*. But conda installed pyodbc brings
its own odbcinst executable and that shows odbc config files are expected in /etc/*. Symlinks were enough to fix the
problem. Try python -c 'import pyodbc;print(pyodbc.drivers())' and see whether you get more than an empty list.
Furthermore, make sure you use 127.0.0.1 instead of localhost. It seems that /etc/hosts is ignored.
Incompatibility with specific pydiverse.transform Versions
pydiverse.pipedag currently doesn't support pydiverse.transform Versions (0.2.0, 0.2.1, 0.2.2), due to major
differences to pdt 0.2.3 and pdt <0.2.
However, it does still work with pdt <0.2.
3 maintainers
Metadata
3 maintainers
Release files for pydiverse-pipedag 0.12.15
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| pydiverse_pipedag-0.12.15.tar.gz | 3.2 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| pydiverse_pipedag-0.12.15-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 3.5 MB
Release files / pydiverse_pipedag-0.12.15.tar.gz
| Download URL | pydiverse_pipedag-0.12.15.tar.gz |
|---|---|
| Size | 3.2 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
675c20c9f7f270cd7c5cfa7bee93a493eaf50198a77b1329234d93481e4de77b
|
|
BLAKE2b-256 checksum How to use checksums |
6e4e37fe97ed5dbd7bfc03237f67fd19b40490927fb83ced95d33d19546219cc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency logRelease files / pydiverse_pipedag-0.12.15-py3-none-any.whl
| Download URL | pydiverse_pipedag-0.12.15-py3-none-any.whl |
|---|---|
| Size | 237.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2f2a068a2b904bf711ce3dcc9e6bf75110d20c60ba14d7102ba819ce53f69508
|
|
BLAKE2b-256 checksum How to use checksums |
2b2f38031228c32d8305109e2a81aaeb3e1ff4b76614aea5d9aaee7fbc93d01f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency log
