any-gold

Have you ever been in a situation where you wanted to experiment with a new dataset and wasted a few hours of your time before even having access to the data? We did, and we truly believe that it should not be like that anymore.

Any Gold is thus a comprehensive collection of custom PyTorch Dataset implementations for publicly available datasets across various modalities.

Purpose

The goal of this repository is to provide custom PyTorch Dataset classes that are compatible with PyTorch's DataLoader to facilitate experimentation with publicly available datasets. Each dataset implementation includes automated download functionality to locally cache the data before use. Instead of spending time to access the data, you can focus on experimenting with it.

Features

• PyTorch Integration: All datasets implement the PyTorch Dataset interface
• Automatic Downloads: Built-in functionality to download and cache datasets locally
• Multimodal Support: Datasets spanning various data types and domains
• Consistent API: Uniform interface across different dataset implementations
• Minimal Dependencies: Core dependencies are managed with uv

Available Datasets

Image Datasets

• PlantSeg: Large-scale in-the-wild dataset for plant disease segmentation (Paper, Zenodo)
• MVTecADDataset: Anomaly detection dataset for industrial inspection (Paper, Hugging Face)
• KPITask1PatchLevel: A dataset for kidney disease segmentation (Paper, Synapse)
• DeepGlobeRoadExtraction: Road extraction from satellite images (Paper, Kaggle)
• ISIC2018SkinLesionDataset: A dataset for skin lesion segmentation(Paper, Hugging Face)
• PascalVOC2012Segmentation: A dataset for semantic segmentation (Website, Kaggle)

Usage

import any_gold as ag
from torch.utils.data import DataLoader

# Initialize dataset (downloads data if not already present)
dataset = ag.AnyDataset()

# Create DataLoader
dataloader = DataLoader(dataset, batch_size=32, shuffle=True)

# Iterate through batches
for batch in dataloader:
  # Your training/evaluation code here
  pass

Contributing

Process

Contributions are welcome! To contribute to this project:

1. Fork the repository on GitHub
2. Clone your fork: git clone https://github.com/yourusername/any-gold.git
3. Create a new branch for your feature: git checkout -b feature-name
4. Install development dependencies (see below)
5. Set up pre-commit hooks: uv run pre-commit install
6. Implement a new class that inherits from AnyDataset
7. Include download functionality for the dataset
8. Add appropriate documentation and tests (pytest) for your dataset class
9. Ensure code passes all pre-commit checks
10. Submit a pull request to the main repository

We use pre-commit hooks to maintain code quality:

• Ruff for linting and formatting
• MyPy for type checking

Installation

Dependencies in this repository are managed with uv, a fast Python package installer and resolver. The dependencies are defined in the pyproject.toml file.

# Clone the repository
git clone https://github.com/yourusername/any-gold.git
cd any-gold

# Install dependencies with uv
uv sync --all-extras
source .venv/bin/activate

Release Process

To release a new version of the any-gold package:

1. Create a new branch for the release: git checkout -b release-vX.Y.Z
2. Update the version vX.Y.Z in pyproject.toml
3. Run uv sync to update the lock file
4. Commit the changes with a message like release vX.Y.Z
5. Merge the branch into main
6. trigger a new release on GitHub with the tag vX.Y.Z