🧼🔎 SelfClean
A holistic self-supervised data cleaning strategy to detect off-topic samples, near duplicates, and label errors.
Publications: SelfClean Paper (NeurIPS24) | Data Cleaning Protocol Paper (ML4H23@NeurIPS)
NOTE: Make sure to have git-lfs installed before pulling the repository to ensure the pre-trained models are pulled correctly (git-lfs install instructions).
This project is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International license.
Installation
Install SelfClean via PyPI:
# upgrade pip to its latest version
pip install -U pip
# install selfclean
pip install selfclean
# Alternatively, use explicit python version (XX)
python3.XX -m pip install selfclean
Getting Started
You can run SelfClean in a few lines of code:
from selfclean import SelfClean
selfclean = SelfClean(
# displays the top-7 images from each error type
# per default this option is disabled
plot_top_N=7,
)
# run on pytorch dataset
issues = selfclean.run_on_dataset(
dataset=copy.copy(dataset),
)
# run on image folder
issues = selfclean.run_on_image_folder(
input_path="path/to/images",
)
# get the data quality issue rankings
df_near_duplicates = issues.get_issues("near_duplicates", return_as_df=True)
df_off_topic_samples = issues.get_issues("off_topic_samples", return_as_df=True)
df_label_errors = issues.get_issues("label_errors", return_as_df=True)
Examples:
In examples/, we've provided some example notebooks where you will learn how to analyze and clean datasets using SelfClean.
These examples analyze different benchmark datasets such as:
- Imagenette 🖼️ (Open in NBViewer | GitHub | Colab)
- Oxford-IIIT Pet 🐶 (Open in NBViewer | GitHub | Colab)
Also, check out our Kaggle notebook to see an illustration of how to get a gold medal for cleaning a competition dataset.
More Ressources
- Recording of the Computer Vision Meetup: Intrinsic Self-Supervision for Data Quality Audits
- Are We Measuring What We Think We Are? The Perils of Contaminated Benchmark Datasets
- Detecting Near Duplicates, Label Errors and Outliers in Image Datasets
Reference
If you find this repository useful for your research, please cite the following work.
@article{groger_selfclean_2024,
title = {{Intrinsic Self-Supervision for Data Quality Audits}},
shorttitle = {{SelfClean}},
author = {Gr\"oger, Fabian and Lionetti, Simone and Gottfrois, Philippe and Gonzalez-Jimenez, Alvaro and Amruthalingam, Ludovic and Consortium, Labelling and Groh, Matthew and Navarini, Alexander A. and Pouly, Marc},
year = 2024,
month = 12,
journal = {Advances in Neural Information Processing Systems (NeurIPS)},
}
Development Environment
Run make for a list of possible targets.
Run these commands to install the requirements for the development environment:
make init
make install
To run linters on all files:
pre-commit run --all-files
We use the following packages for code and test conventions:
blackfor code styleisortfor import sortingpytestfor running tests
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file selfclean-0.0.39.tar.gz.
File metadata
- Download URL: selfclean-0.0.39.tar.gz
- Upload date:
- Size: 112.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.2.0 CPython/3.9.25
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
273a53a8886b92b8e77d18461fd18492ffd368bc75d92773a85680b30432369f
|
|
| MD5 |
b0d8f597bac2b22948e9616dc26f3f58
|
|
| BLAKE2b-256 |
3184f4e6bb241121f7253249390cd2f194e125d56006f0aed982ab8bd7667712
|
File details
Details for the file selfclean-0.0.39-py3-none-any.whl.
File metadata
- Download URL: selfclean-0.0.39-py3-none-any.whl
- Upload date:
- Size: 169.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.2.0 CPython/3.9.25
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
71695670fca35fd5f6bc69d12c5a635ba16588dd5fc59b2d1ba59c6f2fb7e8d2
|
|
| MD5 |
aecb95e8b6e3d75e0f02fa3e24f7c538
|
|
| BLAKE2b-256 |
6abdb0461c4d684eca1e1a8a304a6bbbc14abd37a88d79d388e8215f4c02baed
|
