GPT training in Lightning
Project description
lightning-GPT
lightning-GPT is a minimal wrapper around Andrej Karpathy's minGPT and nanoGPT in Lightning.
It is aimed at providing a minimal Lightning layer on top of minGPT and nanoGPT, while leveraging the full breadth of Lightning.
There are currently a few options:
MinGPT
: the GPT model from minGPT vanilla (set--implementation=mingpt
)NanoGPT
: the GPT model from nanoGPT vanilla (set--implementation=nanogpt
)DeepSpeedMinGPT
: the GPT model from minGPT made DeepSpeed-ready (set--strategy=deepspeed
)DeepSpeedNanoGPT
: the GPT model from nanoGPT made DeepSpeed-ready (set--strategy=deepspeed
)FSDPMinGPT
: the GPT model from minGPT made FSDP (native)-ready (set--strategy=fsdp-gpt
)FSDPNanoGPT
: the GPT model from nanoGPT made FSDP (native)-ready (set--strategy=fsdp-gpt
)
minGPT and nanoGPT are vendored with the repo in the mingpt
and nanogpt
directories respectively. Find the respective LICENSE there.
Thanks to:
- @karpathy for the original minGPT and nanoGPT implementation
- @williamFalcon for the first Lightning port
- @SeanNaren for the DeepSpeed pieces
Installation from source
To clone the repository, please clone the repo with
git clone https://github.com/Lightning-AI/lightning-GPT
git submodule update --init --recursive
After this you can proceed with the following steps.
MinGPT
First install the dependencies
pip install -r requirements.txt
then
python train.py
See
python train.py --help
for the available flags.
NanoGPT
First install the dependencies.
pip install -r requirements.txt
pip install -r requirements/nanogpt.txt
then
python train.py
See
python train.py --help
for the available flags.
DeepSpeed
Install the extra-dependencies:
pip install -r requirements/deepspeed.txt
and pass the strategy
flag to the script
python train.py --implementation mingpt --strategy deepspeed
or
python train.py --implementation nanogpt --strategy deepspeed
FSDP native
Pass the strategy
flag to the script
python train.py --implementation mingpt --strategy fsdp_native
or
python train.py --implementation nanogpt --strategy fsdp_native
PyTorch 2.0
To run on dynamo/inductor from the PyTorch 2.0 compiler stack, run
python train.py --compile dynamo
Note that you will need a recent torch
nightly (1.14.x) for torch.compile
to be available.
Credits
- https://github.com/karpathy/nanoGPT
- https://github.com/karpathy/minGPT
- https://github.com/SeanNaren/minGPT
- https://pytorch-lightning.readthedocs.io/en/stable/advanced/model_parallel.html
License
Apache 2.0 license https://opensource.org/licenses/Apache-2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for lightning_gpt-0.1.0rc0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 47ef2010f9b164f4140573a951d9d7258015decfca08f1f984803be4a76e4614 |
|
MD5 | 5a057556ce90415530bb7cfa749d1256 |
|
BLAKE2b-256 | b9364dfe05581cafccb2a2d535efe1a89a37537a0844b56b27635ea45101273c |