prepare your dataset for finetuning LLMs
Project description
Dataset Preparation for Transformers Fine-tuning
The Dataset Prep Transformers package simplifies the process of preparing datasets for fine-tuning or training various large language models available in the Hugging Face Transformers library. Whether you're using a model from the Hugging Face repository or have your own dataset, this package streamlines the data integration for a seamless training experience.
Features
- Easily integrate your dataset with Hugging Face Transformers models for training or fine-tuning.
- Specify the model repository ID and dataset from the Hugging Face library to automatically fetch and configure the data.
- Seamlessly incorporate your custom dataset by providing it as input to the package.
- new: custom map function => map_function = your_function (having the facility of tokenization)
Installation
You can install the package using pip:
pip install yDataPrep
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
yDataPrep-0.0.1.2.1.tar.gz
(3.1 kB
view hashes)
Built Distribution
Close
Hashes for yDataPrep-0.0.1.2.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 524c3c7bd430aab544da8608bac69ba75b0733f7f15fb3a1fa17e917aad53f14 |
|
MD5 | 7fc986502237bf811b358395b9af06ac |
|
BLAKE2b-256 | 7b668b1105e53188384e3f967c0d62580bbce53e6f0da27f9426187f99227d69 |