Note: Disdat 1.0 no longer contains the instrumented form of Luigi. Disdat-Luigi now resides here. Want to build versioned pipelines? pip install disdat-luigi Want to just use the Disdat API? pip install disdat
Disdat is a Python (3.9+) package for data versioning that allows data scientists to create, share, and track data products. Disdat organizes data into bundles, collections of literal values and files – bundles are the unit at which data is versioned and shared. Disdat provides an API for creating, finding, and publishing bundles to cloud storage (e.g., AWS S3).
Disdat-Luigi uses this API to instrument Spotify’s Luigi, so you can build pipelines that automatically create bundles, making it easy to share the latest outputs with other users and pipelines. Instead of lengthy email conversations with multiple file attachments, searching through Slack for the most recent S3 file path, users can instead dsdt pull awesome_data to get the latest ‘awesome_data.’
Disdat’s bundle API and pipelines provide:
Simplified pipelines – Users implement two functions per task: requires and run.
Enhanced re-execution logic – Disdat re-runs processing steps when code or data changes.
Data versioning/lineage – Disdat records code and data versions for each output data set.
Share data sets – Users may push and pull data to remote contexts hosted in AWS S3.
Auto-docking – Disdat dockerizes pipelines so that they can run locally or execute on the cloud.
Find our latest documentation on gitbook here!
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file disdat-1.2.0.tar.gz.
File metadata
- Download URL: disdat-1.2.0.tar.gz
- Upload date:
- Size: 726.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.13.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
4a6f4dce186e2f2cc98252cc32162bdff05e252dc82e4098188a52a5c147c796
|
|
| MD5 |
07eaab28f6dfee3cf2a28666f47bbae2
|
|
| BLAKE2b-256 |
34397f2a9f617469f31af805720f7afbd9654163734c3d109dbb55ae41b93669
|
File details
Details for the file disdat-1.2.0-py3-none-any.whl.
File metadata
- Download URL: disdat-1.2.0-py3-none-any.whl
- Upload date:
- Size: 92.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.13.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2f40fd1fc9ede845b773ca5824ade21a42cba7fcb662c0aca90b019084c62c1b
|
|
| MD5 |
d4f7669fc6362558ad4f1eb87fe4e422
|
|
| BLAKE2b-256 |
0bd6f746f3bfc580330b3a776af8319902b2c27d5b4e6e13231ae486b59e5b2f
|
