Skip to main content

prepare your dataset for finetuning LLMs

Project description

Dataset Preparation for Transformers Fine-tuning

License

The Dataset Prep Transformers package simplifies the process of preparing datasets for fine-tuning or training various large language models available in the Hugging Face Transformers library. Whether you're using a model from the Hugging Face repository or have your own dataset, this package streamlines the data integration for a seamless training experience.

Features

  • Easily integrate your dataset with Hugging Face Transformers models for training or fine-tuning.
  • Specify the model repository ID and dataset from the Hugging Face library to automatically fetch and configure the data.
  • Seamlessly incorporate your custom dataset by providing it as input to the package.
  • new: custom map function => map_function = your_function (having the facility of tokenization)

Installation

You can install the package using pip:

pip install yDataPrep

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

yDataPrep-0.0.1.2.1.tar.gz (3.1 kB view hashes)

Uploaded Source

Built Distribution

yDataPrep-0.0.1.2.1-py3-none-any.whl (3.3 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page