Atom
Atom is a finetuned LLAMA to create better LLMS through Pytorch Data!
Installation
You can install the package using pip
git clone https://github.com/jquesnelle/yarn
cd Atom
pip install -e .
Training
To train the models, run accelerate config and enable DeepSpeed acceleration. deepspeed/zero3.json was the configuration file used for training.
# ./train.sh
The tokenized training data is available on Hugging Face and was derived from the pg19 dataset.
Evaluation
To reproduce the evaluations, install lm-evaluation-harness with pip install git+https://github.com/EleutherAI/lm-evaluation-harness and then run the two provided scripts.
# ./eval.sh
# ./eval-harness.sh
Citation
@misc{peng2023yarn,
title={YaRN: Efficient Context Window Extension of Large Language Models},
author={Bowen Peng and Jeffrey Quesnelle and Honglu Fan and Enrico Shippole},
year={2023},
eprint={2309.00071},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
1 maintainer
Metadata
1 maintainer
Release files for atom-torch 0.0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| atom_torch-0.0.2.tar.gz | 32.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| atom_torch-0.0.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 65.8 kB
Release files / atom_torch-0.0.2.tar.gz
| Download URL | atom_torch-0.0.2.tar.gz |
|---|---|
| Size | 32.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6483a3a259491dbd88348fc14a657adf7c53e4f387bd3ecb7de4d21192a0ab76
|
|
BLAKE2b-256 checksum How to use checksums |
c31fa948e5d9423c4b929246d79e49e91cc29132ca714a2cff2baeda30be09d6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.3.2 CPython/3.11.0 Darwin/22.4.0
|
Release files / atom_torch-0.0.2-py3-none-any.whl
| Download URL | atom_torch-0.0.2-py3-none-any.whl |
|---|---|
| Size | 33.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2ebe961645855f9a97a204bc4b0bf3ccdb1a3a643a37ec39edb4f90702c79800
|
|
BLAKE2b-256 checksum How to use checksums |
80f46c4c5f1a17aa98a8040e7b40bf27b944c75eab06d3650f7113b3692607c1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.3.2 CPython/3.11.0 Darwin/22.4.0
|
