Pytorch Datastream
This is a simple library for creating readable dataset pipelines and reusing best practices for issues such as imbalanced datasets. There are just two components to keep track of: Dataset and Datastream.
Dataset is a simple mapping between an index and an example. It provides pipelining of functions in a readable syntax originally adapted from tensorflow 2's tf.data.Dataset.
Datastream combines a Dataset and a sampler into a stream of examples. It provides a simple solution to oversampling / stratification, weighted sampling, and finally converting to a torch.utils.data.DataLoader.
See the documentation for more information.
Install
poetry add pytorch-datastream
Or, for the old-timers:
pip install pytorch-datastream
Usage
The list below is meant to showcase functions that are useful in most standard and non-standard cases. It is not meant to be an exhaustive list. See the documentation for a more extensive list on API and usage.
Dataset.from_subscriptable
Dataset.from_dataframe
Dataset
.map
.subset
.split
.cache
.with_columns
Datastream.merge
Datastream.zip
Datastream
.map
.data*loader
.zip_index
.update_weights*
.update*example_weight*
.weight
.state_dict
.load_state_dict
Simple image dataset example
Here's a basic example of loading images from a directory:
from datastream import Dataset
from pathlib import Path
from PIL import Image
# Assuming images are in a directory structure like:
# images/
# class1/
# image1.jpg
# image2.jpg
# class2/
# image3.jpg
# image4.jpg
image_dir = Path("images")
image_paths = list(image_dir.glob("\*_/_.jpg"))
dataset = (
Dataset.from_paths(
image_paths,
pattern=r".\*/(?P<class_name>\w+)/(?P<image_name>\w+).jpg"
)
.map(lambda row: dict(
image=Image.open(row["path"]),
class_name=row["class_name"],
image_name=row["image_name"],
))
)
# Access an item from the dataset
first_item = dataset[0]
print(f"Class: {first_item['class_name']}, Image name: {first_item['image_name']}")
Merge / stratify / oversample datastreams
The fruit datastreams given below repeatedly yields the string of its fruit type.
>>> datastream = Datastream.merge([
>>> ... (apple_datastream, 2),
>>> ... (pear_datastream, 1),
>>> ... (banana_datastream, 1),
>>> ... ])
>>> next(iter(datastream.data_loader(batch_size=8)))
>>> ['apple', 'apple', 'pear', 'banana', 'apple', 'apple', 'pear', 'banana']
>>>
Zip independently sampled datastreams
The fruit datastreams given below repeatedly yields the string of its fruit type.
>>> datastream = Datastream.zip([
>>> ... apple_datastream,
>>> ... Datastream.merge([pear_datastream, banana_datastream]),
>>> ... ])
>>> next(iter(datastream.data_loader(batch_size=4)))
>>> [('apple', 'pear'), ('apple', 'banana'), ('apple', 'pear'), ('apple', 'banana')]
>>>
More usage examples
See the documentation for more usage examples.
1 maintainer
Metadata
1 maintainer
Release files for pytorch-datastream 0.4.13
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| pytorch_datastream-0.4.13.tar.gz | 18.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| pytorch_datastream-0.4.13-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 43.3 kB
Release files / pytorch_datastream-0.4.13.tar.gz
| Download URL | pytorch_datastream-0.4.13.tar.gz |
|---|---|
| Size | 18.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
79c2722a9f9efa91190193f314921ed04d04dc38ccd1003c8087f251725e1e7e
|
|
BLAKE2b-256 checksum How to use checksums |
c291ca75d94e186d366fbc46079005d3709ff3ef1f053b95d37aaced58898b01
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.27 {"installer":{"name":"uv","version":"0.11.27","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / pytorch_datastream-0.4.13-py3-none-any.whl
| Download URL | pytorch_datastream-0.4.13-py3-none-any.whl |
|---|---|
| Size | 25.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
91713e70b87c0b9e8b1e7c9269c80247e4d5043f9bf680734c8c68ae2366f2d5
|
|
BLAKE2b-256 checksum How to use checksums |
0a6670d3ab699f1f3b8e6caa3ed7949edec28b2b1a8c0d76353cea006483c9a0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.27 {"installer":{"name":"uv","version":"0.11.27","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
