pydiverse.transform

Pipe based dataframe manipulation library that can also transform data on SQL databases

This is an early stage version 0.x, however, it is already used in real projects. We are happy to receive your feedback as issues on the GitHub repo. Feel free to also comment on existing issues to extend them to your needs or to add solution ideas.

Usage

pydiverse.transform can either be installed via pypi with pip install pydiverse-transform or via conda-forge with conda install pydiverse-transform -c conda-forge. Our recommendation would be to use pixi which is also based on conda-forge:

mkdir my_project
pixi init
pixi add pydiverse-transform

With pixi, you run python like this:

pixi run python -c 'import pydiverse.transform'

or this:

pixi run python my_script.py

Example

This code illustrates how to use pydiverse.transform with pandas and SQL:

from pydiverse.transform import Table
from pydiverse.transform.lazy import SQLTableImpl
from pydiverse.transform.eager import PandasTableImpl
from pydiverse.transform.core.verbs import *
import pandas as pd
import sqlalchemy as sqa


def main():
    dfA = pd.DataFrame(
        {
            "x": [1],
            "y": [2],
        }
    )
    dfB = pd.DataFrame(
        {
            "a": [2, 1, 0, 1],
            "x": [1, 1, 2, 2],
        }
    )

    input1 = Table(PandasTableImpl("dfA", dfA))
    input2 = Table(PandasTableImpl("dfB", dfB))

    transform = (
        input1
        >> left_join(input2 >> select(), input1.x == input2.x)
        >> mutate(x5=input1.x * 5, a=input2.a)
    )
    out1 = transform >> collect()
    print("\nPandas based result:")
    print(out1)

    engine = sqa.create_engine("sqlite:///:memory:")
    dfA.to_sql("dfA", engine, index=False, if_exists="replace")
    dfB.to_sql("dfB", engine, index=False, if_exists="replace")
    input1 = Table(SQLTableImpl(engine, "dfA"))
    input2 = Table(SQLTableImpl(engine, "dfB"))
    transform = (
        input1
        >> left_join(input2 >> select(), input1.x == input2.x)
        >> mutate(x5=input1.x * 5, a=input2.a)
    )
    out2 = transform >> collect()
    print("\nSQL query:")
    print(transform >> build_query())
    print("\nSQL based result:")
    print(out2)

    out1 = out1.sort_values("a").reset_index(drop=True)
    out2 = out2.sort_values("a").reset_index(drop=True)
    assert len(out1.compare(out2)) == 0


if __name__ == "__main__":
    main()

Metadata

Release files for pydiverse-transform 0.7.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pydiverse-transform 0.7.2
File	Size	Uploaded
pydiverse_transform-0.7.2.tar.gz	328.2 kB	Jul 10, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for pydiverse-transform 0.7.2
File	Interpreter	ABI	Platform	Reset
pydiverse_transform-0.7.2-py3-none-any.whl 112.6 kB Jul 10, 2026	Python 3	none	any	Details

Total release size: 440.9 kB

Release files / pydiverse_transform-0.7.2.tar.gz

Download URL	pydiverse_transform-0.7.2.tar.gz
Size	328.2 kB
Tags	Source
SHA-256 checksum How to use checksums	`13dafcd422da9c48417badc7550beca8dc921e5898b7b13836e7e6d3f3c61b2b`
BLAKE2b-256 checksum How to use checksums	`524658b0bbc82ef1d7c5d96717b5cfddfdff89354a57053d638991e3281d10d3`
Upload date	Jul 10, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/6.1.0 CPython/3.13.7`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/pydiverse/pydiverse.transform
Publishing commit: github.com/pydiverse/pydiverse.transform/tree/3ad7a62d32251d1a2caffdfb17b2fd4e719ca4e6

Workflow

Publishing configuration: github.com/pydiverse/pydiverse.transform/blob/3ad7a62d32251d1a2caffdfb17b2fd4e719ca4e6/.github/workflows/release.yml
Publishing logs: github.com/pydiverse/pydiverse.transform/actions/runs/29098289526/attempts/1

Artifact

Subject: pydiverse_transform-0.7.2.tar.gz
SHA-256 checksum: 13dafcd422da9c48417badc7550beca8dc921e5898b7b13836e7e6d3f3c61b2b Verifying attestations

Release files / pydiverse_transform-0.7.2-py3-none-any.whl

Download URL	pydiverse_transform-0.7.2-py3-none-any.whl
Size	112.6 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`717ab58eb0c2418cbd21fc6cb1a8c282a235ef553377b8ca62f79ff66f5fef9b`
BLAKE2b-256 checksum How to use checksums	`4e8f37fa7f5fe2aff72d7a06ae63e44a6423f16ef1c0df6ccf70ea56980de885`
Upload date	Jul 10, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/6.1.0 CPython/3.13.7`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/pydiverse/pydiverse.transform
Publishing commit: github.com/pydiverse/pydiverse.transform/tree/3ad7a62d32251d1a2caffdfb17b2fd4e719ca4e6

Workflow

Publishing configuration: github.com/pydiverse/pydiverse.transform/blob/3ad7a62d32251d1a2caffdfb17b2fd4e719ca4e6/.github/workflows/release.yml
Publishing logs: github.com/pydiverse/pydiverse.transform/actions/runs/29098289526/attempts/1

Artifact

Subject: pydiverse_transform-0.7.2-py3-none-any.whl
SHA-256 checksum: 717ab58eb0c2418cbd21fc6cb1a8c282a235ef553377b8ca62f79ff66f5fef9b Verifying attestations