Abstract Dataloader: Dataloader Not Included
What is the Abstract Dataloader?
The abstract dataloader (ADL) is a minimalist specification for creating composable and interoperable dataloaders and data transformations, along with abstract template implementations and reusable generic components.
The ADL's specifications and bundled implementations lean heavily on generic type annotations in order to enable type checking using static type checkers such as mypy or pyright and runtime (dynamic) type checkers such as beartype and typeguard.
For detailed documentation, please see the project site.
Why Abstract?
Loading, preprocessing, and training models on time-series data is ubiquitous in machine learning for cyber-physical systems. However, unlike mainstream machine learning research, which has largely standardized around "canonical modalities" in computer vision (RGB images) and natural language processing (ordinary unstructured text), each new setting, dataset, and modality comes with a new set of tasks, questions, challenges - and data types which must be loaded and processed.
This poses a substantial software engineering challenge. With many different modalities, processing algorithms which operate on the power set of those different modalities, and downstream tasks which also each depend on some subset of modalities, two undesirable potential outcomes emerge:
- Data loading and processing components fragment into an exponential number of incompatible chunks, each of which encapsulates its required loading and processing functionality in a slightly different way. The barrier this presents to rapid prototyping needs no further explanation.
- The various software components coalesce into a monolith which nominally supports the power set of all functionality. However, in addition to the compatibility issues that come with bundling heterogeneous requirements such as managing "non-dependencies" (i.e. dependencies which are required by the monolith, but not a particular task), this also presents a hidden challenge in that by support exponentially many possible configurations, such an architecture is also exponentially hard to debug and verify.
However, we do not believe that these outcomes are a foregone conclusion. In particular, we believe that it's possible to write "one true dataloader" which can scale while maintaining intercompability by not writing a common dataloader at all -- but rather a common specification for writing dataloaders. We call this the "abstract dataloader".
Setup
While it is not necessary to install the abstract_dataloader in order to take advantage of ADL-compliant components, installing this library provides access to Protocol types which describe each interface, as well as generic components which may be useful for working with ADL-compliant components.
pip install abstract-dataloader
Metadata
Release files for abstract_dataloader 0.4.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| abstract_dataloader-0.4.7.tar.gz | 1.7 MB | Without provenance | Details for abstract_dataloader-0.4.7.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| abstract_dataloader-0.4.7-py3-none-any.whl | Python 3 | none | any | Details for abstract_dataloader-0.4.7-py3-none-any.whl |
Total release size: 1.8 MB
Release files / abstract_dataloader-0.4.7.tar.gz
| Download URL | abstract_dataloader-0.4.7.tar.gz |
|---|---|
| Size | 1.7 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e07f1d1997be4f1041902f5321d017174e4680043ee020408a7cfa8ff7d8ea80
|
|
BLAKE2b-256 checksum How to use checksums |
dde469f44bd0700939c219148adc7de043494e17f58afa34c54bbfe0c76a28a0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.6.16
|
Release files / abstract_dataloader-0.4.7-py3-none-any.whl
| Download URL | abstract_dataloader-0.4.7-py3-none-any.whl |
|---|---|
| Size | 39.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c397a1df7a329c6e62cd43a9c7fb51a8dd004452bd65b16b1819564a493321aa
|
|
BLAKE2b-256 checksum How to use checksums |
f27a74d821df7fc1321f4029d407c76a9702ceb8fcc365fe995c24b266f2bd5d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.6.16
|
Security for abstract_dataloader 0.4.7
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.