6 projects
pydiverse-pipedag
A pipeline orchestration library executing tasks within one python session. It takes care of SQL table (de)materialization, caching and cache invalidation. Blob storage is supported as well for example for storing model files.
pydiverse-common
Common functionality shared between pydiverse libraries
pydiverse-transform
Pipe based dataframe manipulation library that can also transform data on SQL databases
pydiverse-colspec
Validate column specifications and constraints for SQL tables and polars data frames.
pydiverse.pipetest
An adaption layer for pydiverse.pipedag that simplyfies execution of pipedag steps as unit tests with cache invalidation awareness.
pydiverse
A collection of nicely interoperable libraries for data pipeline orchestration allowing for both SQL target and in-memory operation.
