Lightning ⚡ Bagua
Deep Learning Training Acceleration with Bagua and Lightning AI
Bagua is a deep learning training acceleration framework which supports multiple advanced distributed training algorithms including:
- Gradient AllReduce for centralized synchronous communication, where gradients are averaged among all workers.
- Decentralized SGD for decentralized synchronous communication, where each worker exchanges data with one or a few specific workers.
- ByteGrad and QAdam for low precision communication, where data is compressed into low precision before communication.
- Asynchronous Model Average for asynchronous communication, where workers are not required to be synchronized in the same iteration in a lock-step style.
By default, Bagua uses Gradient AllReduce algorithm, which is also the algorithm implemented in DDP, but Bagua can usually produce a higher training throughput due to its backend written in Rust.
Installation
pip install -U lightning-bagua
Usage
Simply set the strategy argument in the Trainer:
from lightning import Trainer
# train on 4 GPUs (using Bagua mode)
trainer = Trainer(strategy="bagua", accelerator="gpu", devices=4)
See Bagua Tutorials for more details on installation and advanced features.
Release files for lightning-bagua 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| lightning-bagua-0.1.0.tar.gz | 14.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| lightning_bagua-0.1.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 27.4 kB
Release files / lightning-bagua-0.1.0.tar.gz
| Download URL | lightning-bagua-0.1.0.tar.gz |
|---|---|
| Size | 14.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3ca4f1659816b6d62cc604081fa0b4210a0b80de45cc604c9bdcb2c906cb4d4c
|
|
BLAKE2b-256 checksum How to use checksums |
61c8232a596b7809ef6f3957c3e7027fa2af848d0b12507374db06bdf1b2d4ef
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / lightning_bagua-0.1.0-py3-none-any.whl
| Download URL | lightning_bagua-0.1.0-py3-none-any.whl |
|---|---|
| Size | 13.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8cb1e0156be950c1859061c9586d606f9543c96c5e0bb8dd128c7075a4233295
|
|
BLAKE2b-256 checksum How to use checksums |
6f4a33799300a1fa158e4545a6639325b91a26ddef5d336fa514089915905fd9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
