lightning-GPT
lightning-GPT is a minimal wrapper around Andrej Karpathy's minGPT and nanoGPT in Lightning.
It is aimed at providing a minimal Lightning layer on top of minGPT and nanoGPT, while leveraging the full breadth of Lightning.
There are currently a few options:
MinGPT: the GPT model from minGPT vanilla (set--implementation=mingpt)NanoGPT: the GPT model from nanoGPT vanilla (set--implementation=nanogpt)DeepSpeedMinGPT: the GPT model from minGPT made DeepSpeed-ready (set--strategy=deepspeed)DeepSpeedNanoGPT: the GPT model from nanoGPT made DeepSpeed-ready (set--strategy=deepspeed)FSDPMinGPT: the GPT model from minGPT made FSDP (native)-ready (set--strategy=fsdp-gpt)FSDPNanoGPT: the GPT model from nanoGPT made FSDP (native)-ready (set--strategy=fsdp-gpt)
minGPT and nanoGPT are vendored with the repo in the mingpt and nanogpt directories respectively. Find the respective LICENSE there.
Thanks to:
- @karpathy for the original minGPT and nanoGPT implementation
- @williamFalcon for the first Lightning port
- @SeanNaren for the DeepSpeed pieces
Installation
There are two main ways to install this package.
Installation from source is preferred if you need the latest version with yet unreleased changes, want to use the provided benchmarking or training suites or need to adjust the package.
Installation from PyPI is preferred if you just want to use a stable version of the package without any modifications.
Installation from PyPI
To install the package, simply run
pip install lightning-gpt
Installation from source
To clone the repository, please clone the repo with
git clone https://github.com/Lightning-AI/lightning-GPT && cd lightning-GPT
git submodule update --init --recursive
and install with
pip install -e .
After this you can proceed with the following steps.
MinGPT
First install the dependencies
pip install -r requirements.txt
then
python train.py
See
python train.py --help
for the available flags.
NanoGPT
First install the dependencies.
pip install -r requirements.txt
pip install -r requirements/nanogpt.txt
then
python train.py
See
python train.py --help
for the available flags.
DeepSpeed
Install the extra-dependencies:
pip install -r requirements/deepspeed.txt
and pass the strategy flag to the script
python train.py --implementation mingpt --strategy deepspeed
or
python train.py --implementation nanogpt --strategy deepspeed
FSDP native
Pass the strategy flag to the script
python train.py --implementation mingpt --strategy fsdp_native
or
python train.py --implementation nanogpt --strategy fsdp_native
PyTorch 2.0
To run on dynamo/inductor from the PyTorch 2.0 compiler stack, run
python train.py --compile dynamo
Note that you will need a recent torch nightly (1.14.x) for torch.compile
to be available.
Credits
- https://github.com/karpathy/nanoGPT
- https://github.com/karpathy/minGPT
- https://github.com/SeanNaren/minGPT
- https://pytorch-lightning.readthedocs.io/en/stable/advanced/model_parallel.html
License
Apache 2.0 license https://opensource.org/licenses/Apache-2.0
Metadata
Release files for lightning-gpt 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| lightning_gpt-0.1.1.tar.gz | 595.9 kB | Without provenance | Details for lightning_gpt-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| lightning_gpt-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for lightning_gpt-0.1.1-py3-none-any.whl |
Total release size: 1.1 MB
Release files / lightning_gpt-0.1.1.tar.gz
| Download URL | lightning_gpt-0.1.1.tar.gz |
|---|---|
| Size | 595.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
514800fdce06a9a01fe29095fd2a60075010eda1ff5e6e4e8a563b8e6a2cd3a1
|
|
BLAKE2b-256 checksum How to use checksums |
35fdeaf9c5d7b3934e4ee504f5cf778ada80c5e1c242b6092ffabf2ba90ce2cc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.6
|
Release files / lightning_gpt-0.1.1-py3-none-any.whl
| Download URL | lightning_gpt-0.1.1-py3-none-any.whl |
|---|---|
| Size | 466.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
aa450e87f4716145d8a97fbbb838dc5fc380647e728e3acb218d7b3042285487
|
|
BLAKE2b-256 checksum How to use checksums |
5c2a7ea9993d758899cd33d4b5faa73eff933f7034dc2b1ce4b5a7e5e099ca86
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.6
|
Security for lightning-gpt 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.