atom - Pytorch
Project description
Atom
Atom is a finetuned LLAMA to create better LLMS through Pytorch Data!
Installation
You can install the package using pip
git clone https://github.com/jquesnelle/yarn
cd Atom
pip install -e .
Training
To train the models, run accelerate config
and enable DeepSpeed acceleration. deepspeed/zero3.json
was the configuration file used for training.
# ./train.sh
The tokenized training data is available on Hugging Face and was derived from the pg19 dataset.
Evaluation
To reproduce the evaluations, install lm-evaluation-harness with pip install git+https://github.com/EleutherAI/lm-evaluation-harness
and then run the two provided scripts.
# ./eval.sh
# ./eval-harness.sh
Citation
@misc{peng2023yarn,
title={YaRN: Efficient Context Window Extension of Large Language Models},
author={Bowen Peng and Jeffrey Quesnelle and Honglu Fan and Enrico Shippole},
year={2023},
eprint={2309.00071},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
atom_torch-0.0.1.tar.gz
(32.2 kB
view hashes)
Built Distribution
atom_torch-0.0.1-py3-none-any.whl
(33.6 kB
view hashes)
Close
Hashes for atom_torch-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4e06992c0ce58addf145e21cfcada5a71d81209c9357114f139f54a774c096cf |
|
MD5 | 9d76128087a2c64d53d7bb1e735d7f17 |
|
BLAKE2b-256 | 13df2ff3e95b88d78ca197c45eb0c3a02c06bc27a9676868c3da44601ebd2926 |