Lightning + Hivemind
Collaborative Training tries to solve the need for top-tier multi-GPU servers by allowing you to train across unreliable machines, such as local machines or even preemptible cloud compute across the internet.
Under the hood, we use Hivemind which provides de-centralized training across the internet.
To use Collaborative Training, you need to first this extension.
pip install -U lightning-Hivemind
The HivemindStrategy accumulates gradients from all processes that are collaborating until they reach a target_batch_size. By default, we use the batch size
of the first batch to determine what each local machine batch contributes towards the target_batch_size. Once the target_batch_size is reached, an optimizer step
is made on all processes.
When using HivemindStrategy note that you cannot use gradient accumulation (accumulate_grad_batches). This is because Hivemind manages accumulation internally.
from lightning import Trainer
from lightning_hivemind.strategy import HivemindStrategy
trainer = Trainer(strategy=HivemindStrategy(target_batch_size=8192), accelerator="gpu", devices=1)
Followed by:
python train.py
# Other machines can connect running the same command:
# INITIAL_PEERS=... python train.py
# or passing the peers to the strategy:"
# HivemindStrategy(initial_peers=...)"
A helper message is printed once your training begins, which shows you how to start training on other machines using the same code.
Release files for lightning-Hivemind 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| lightning-Hivemind-0.1.0.tar.gz | 13.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| lightning_Hivemind-0.1.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 25.8 kB
Release files / lightning-Hivemind-0.1.0.tar.gz
| Download URL | lightning-Hivemind-0.1.0.tar.gz |
|---|---|
| Size | 13.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
71a85bec32d45229f8352ab78320c06aad10d1a2b368068e87117666e39ad651
|
|
BLAKE2b-256 checksum How to use checksums |
0621e6108d9f3fcafd8dafee72f2fcf540d8eaeeace37c32bfba8fdb2a68ed7b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / lightning_Hivemind-0.1.0-py3-none-any.whl
| Download URL | lightning_Hivemind-0.1.0-py3-none-any.whl |
|---|---|
| Size | 12.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e479ae2ec144a93cc67a06005f6fdec4ecc420819a3f47068a0aef267d3ce3e2
|
|
BLAKE2b-256 checksum How to use checksums |
fed5d92720a8b60213b8ae0e5b204215e33bb96a14cc152a9b5edb60cfffe8b1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
