scrapeddit 0.0.4

Simple test package

Scrapeddit

Overview

Scrapeddit is a Python class designed for scraping images from Reddit subreddits and creating PyTorch datasets. It facilitates the collection of image data from various subreddits, allowing for easy integration into machine learning pipelines or data analysis projects.

Key Features

• Reddit Scraping: Automatically retrieves image URLs from specified subreddits using the PRAW library.
• Flexible Configuration: Users can customize parameters such as subreddit names, post limits, sorting methods, and content safety filters.
• Data Transformation: Supports image transformation and resizing to fit specific requirements.
• Error Handling: Handles invalid subreddits, restricted subreddits, and failed image fetching gracefully, ensuring smooth data collection.
• Data Visualization: Provides visualization tools to understand the distribution of data sources across different subreddits.

Usage

1. Initialization: Instantiate the ScrapeditDataset class with a list of subreddit names and optional parameters for customization.
2. Data Loading: Access the dataset like any other PyTorch dataset, allowing for seamless integration into machine learning workflows.
3. Data Analysis: Use the provided visualization functions to gain insights into the distribution of data sources and explore the collected dataset.
4. Model Training: Utilize the ScrapeditDataset as a DataLoader for training machine learning models. Integrate it with PyTorch's DataLoader for efficient batch processing and model training.

Requirements

• Python 3.x
• PRAW
• pandas
• requests
• matplotlib
• Pillow
• torch
• torchvision
• tqdm

Acknowledgements

• PRAW: The Python Reddit API Wrapper
• tqdm: A Fast, Extensible Progress Bar for Python and CLI
• Pillow: The Python Imaging Library

License

This project is licensed under the MIT License. See the LICENSE file for details.