yDataPrep 0.0.1.2.1

prepare your dataset for finetuning LLMs

Dataset Preparation for Transformers Fine-tuning

The Dataset Prep Transformers package simplifies the process of preparing datasets for fine-tuning or training various large language models available in the Hugging Face Transformers library. Whether you're using a model from the Hugging Face repository or have your own dataset, this package streamlines the data integration for a seamless training experience.

Features

• Easily integrate your dataset with Hugging Face Transformers models for training or fine-tuning.
• Specify the model repository ID and dataset from the Hugging Face library to automatically fetch and configure the data.
• Seamlessly incorporate your custom dataset by providing it as input to the package.
• new: custom map function => map_function = your_function (having the facility of tokenization)

Installation

You can install the package using pip:

pip install yDataPrep