Dataset Grouper - Scalable Dataset Pipelines for Group-Structured Learning
Dataset Grouper is a library for creating, writing, and iterating over datasets with group-level structure. It is primarily intended for creating large-scale datasets for federated learning research.
Installation
We recommend installing via PyPI. Please check the PyPI page for up-to-date version requirements, including python version requirements.
pip install --upgrade pip
pip install dataset-grouper
Getting Started
Below is a simple starting example, that partitions MNIST across 10 clients by label.
First we import some necessary packages, including Apache Beam, which will be used to run the Dataset Grouper's pipelines.
import apache_beam as beam
import dataset_grouper as dsgp
import tensorflow_datasets as tfds
Next, we download and prepare the MNIST dataset.
dataset_builder = tfds.builder('mnist')
dataset_builder.download_and_prepare(...)
We now write a function that assigns each MNIST example a client identifier
(generally a bytes object). In this case, we will partition examples according
to their label, but you can use much more interesting partition functions as
well.
def label_partition(x):
label = x['label'].numpy()
return str(label).encode('utf-8')
Finally, we build a pipeline that will partition MNIST according to this function, and run it using Beam's Direct Runner.
mnist_pipeline = dsgp.tfds_to_tfrecords(
dataset_builder=dataset_builder,
split='test',
get_key_fn=label_partition,
file_path_prefix=...
)
with beam.Pipeline() as root:
mnist_pipeline(root)
This will save a version of MNIST that has been partitioned according to labels to a TFRecord format. We can also load it to iterate over client datasets.
partitioned_dataset = dsgp.PartitionedDataset(
file_pattern=...,
tfds_features='mnist')
for group_dataset in partitioned_dataset.build_group_stream():
pass
Generally, PartitionedDataset.build_group_stream() is a tf.data.Dataset that
yields datasets, each of which is contains all the examples held by one group.
If you'd like to use these datasets with NumPy, you can simply do:
group_dataset_numpy = group_dataset.as_numpy_iterator()
What Else?
The example above is primarily for educational purposes. MNIST is a relatively small dataset, and can generally fit entirely into memory. For more interesting examples, check out the examples folder.
Dataset Grouper is intended more for large-scale datasets, especially those datasets that do not fit into memory. For these datasets, we recommend using more sophisticated Beam runners, in order to partition the data in a distributed fashion.
Disclaimers
This is not an officially supported Google product.
This is a utility library that downloads and prepares public datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have license to use the dataset. It is your responsibility to determine whether you have permission to use the dataset under the dataset's license.
If you're interested in learning more about responsible AI practices, please see Google AI's Responsible AI Practices.
Dataset Grouper is Apache 2.0 licensed. See the LICENSE file.
Metadata
Release files for dataset-grouper 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dataset_grouper-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for dataset_grouper-0.3.0-py3-none-any.whl |
Release files / dataset_grouper-0.3.0-py3-none-any.whl
| Download URL | dataset_grouper-0.3.0-py3-none-any.whl |
|---|---|
| Size | 22.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
eb6fede98872cef9ed08364039f18ee5f2afd9cca1e68615cf75b2c723394f10
|
|
BLAKE2b-256 checksum How to use checksums |
55fddd4d64a3f81d903a2d8cf6c21603ba30cb4d7f77d2ea32669fc28be086e6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.11.8
|
Security for dataset-grouper 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.