vl-datasets 0.0.3

Clean datasets for computer vision.

Open, Clean Datasets for Computer Vision

🔥 We use fastdup - a free tool to clean all datasets shared in this repo.
Explore the docs »
Report Issues · Read Blog · Get In Touch · About Us

What?

This repo shares clean version of publicly available computer vision datasets.

Why?

Even with the success of generative models, data quality remains an issue that's mainly overlooked. Training models will erroneours data impacts model accuracy, incurs costs in time, storage and computational resources.

How?

In this repo we share clean version of various computer vision datasets.

The datasets are cleaned using a free tool we released - fastdup.

We hope this effort will also help the community train better models and mitigate various model biases.

The cleaned image dataset should be free from most if not all of the following issues:

• Duplicates.
• Broken images.
• Outliers.
• Dark/Bright/Blurry images.

Datasets

Here are some of the datasets we are currently working on.

Dataset	Issues
Food-101	Duplicates - 0.24% (12,345) Outliers - 0.85% (456) Broken - 0.85% (456) Blur - 0.85% (456) Dark - 0.85% (456) Bright - 0.85% (456)
Oxford Pets	Duplicates - 0.24% (12,345) Outliers - 0.85% (456) Broken - 0.85% (456) Blur - 0.85% (456) Dark - 0.85% (456) Bright - 0.85% (456)
Imagenette	Duplicates - 0.24% (12,345) Outliers - 0.85% (456) Broken - 0.85% (456) Blur - 0.85% (456) Dark - 0.85% (456) Bright - 0.85% (456)
Laion 1B	Duplicates - 0.24% (12,345) Outliers - 0.85% (456) Broken - 0.85% (456) Blur - 0.85% (456) Dark - 0.85% (456) Bright - 0.85% (456)
Imagenet-21k	Duplicates - 0.24% (12,345) Outliers - 0.85% (456) Broken - 0.85% (456) Blur - 0.85% (456) Dark - 0.85% (456) Bright - 0.85% (456)
Imagenet-1k	Duplicates - 0.24% (12,345) Outliers - 0.85% (456) Broken - 0.85% (456) Blur - 0.85% (456) Dark - 0.85% (456) Bright - 0.85% (456)
KITTI	Duplicates - 0.24% (12,345) Outliers - 0.85% (456) Broken - 0.85% (456) Blur - 0.85% (456) Dark - 0.85% (456) Bright - 0.85% (456)
DeepFashion	Duplicates - 0.24% (12,345) Outliers - 0.85% (456) Broken - 0.85% (456) Blur - 0.85% (456) Dark - 0.85% (456) Bright - 0.85% (456)
Places365-standard	Duplicates - 0.24% (12,345) Outliers - 0.85% (456) Broken - 0.85% (456) Blur - 0.85% (456) Dark - 0.85% (456) Bright - 0.85% (456)
CelebA-HQ	Duplicates - 0.24% (12,345) Outliers - 0.85% (456) Broken - 0.85% (456) Blur - 0.85% (456) Dark - 0.85% (456) Bright - 0.85% (456)
ADE20K	Duplicates - 0.24% (12,345) Outliers - 0.85% (456) Broken - 0.85% (456) Blur - 0.85% (456) Dark - 0.85% (456) Bright - 0.85% (456)
COCO	Duplicates - 0.24% (12,345) Outliers - 0.85% (456) Broken - 0.85% (456) Blur - 0.85% (456) Dark - 0.85% (456) Bright - 0.85% (456)

Setting Up

Prerequisites

Supported Python versions:

Supported operating systems:

Installation

Option 1 - Install vl_datasets package from PyPI.

pip install vl-datasets

Option 2 - Install the bleeding edge version on GitHub

pip install git+https://github.com/visual-layer/vl-datasets.git@master --upgrade

Getting Started

Import the clean version of dataset.

from vl_datasets import CleanFood101

Load the dataset into a PyTorch DataLoader.

train_dataset = CleanFood101('./', split='train', exclude_csv='food_101_vl-datasets_analysis.csv', transform=train_transform)
valid_dataset = CleanFood101('./', split='test', exclude_csv='food_101_vl-datasets_analysis.csv', transform=valid_transform)

Now you can use the dataset in a PyTorch training loop. Refer to our sample training notebooks for details.

Sample training notebooks:

• PyTorch
• fast.ai

Disclaimer

You are bound to the usage license of the original dataset. It is your responsibility to determine whether you have permission to use the dataset under the dataset's license. We provide no warranty or guarantee of accuracy or completeness.