What is Squirrel Datasets Core?

squirrel-datasets-core is an extension of the Squirrel library. squirrel-datasets-core is a hub where the user can 1) explore existing datasets registered in the data mesh by other users and 2) preprocess their datasets and share them with other users. As an end user, you will be able to load many publically available datasets with ease and speed with the help of squirrel, or load and preprocess your own datasets with the tools we provide here.

For preprocessing, we currently support Spark as the main tool to carry out the task.

If you have any questions or would like to contribute, join our Slack community!

Installation

Install squirrel-core and squirrel-datasets-core with pip. Note that you can install with different dependencies based on your requirements for squirrel drivers. For using the torchvision driver call:

pip install "squirrel-core[torch]"
pip install "squirrel-datasets-core[torchvision]"

For using the hub driver call:

pip install "squirrel-datasets-core[hub]"

For using the spark preprocessing pipelines call:

pip install "squirrel-datasets-core[preprocessing]"

If you would like to get Squirrel's full functionality, install squirrel-core and squirrel-datasets-core with all their dependencies.

pip install "squirrel-core[all]"
pip install "squirrel-datasets-core[all]"

Documentation

Visit our documentation on Readthedocs.

Contributing

squirrel-datasets-core is open source and community contributions are welcome!

Contributing

squirrel-datasets-core is open source and community contributions are welcome!

Check out the contribution guide to learn how to get involved. Please follow our recommendations for best practices and code style.

The humans behind Squirrel

We are Merantix Momentum, a team of ~30 machine learning engineers, developing machine learning solutions for industry and research. Each project comes with its own challenges, data types and learnings, but one issue we always faced was scalable data loading, transforming and sharing. We were looking for a solution that would allow us to load the data in a fast and cost-efficient way, while keeping the flexibility to work with any possible dataset and integrate with any API. That's why we build Squirrel – and we hope you'll find it as useful as we do! By the way, we are hiring!

Citation

If you use Squirrel Datasets in your research, please cite Squirrel using:

@article{2022squirrelcore,
  title={Squirrel: A Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way.},
  author={Squirrel Developer Team},
  journal={GitHub. Note: https://github.com/merantix-momentum/squirrel-core},
  year={2022}
}