TorchClippedOptimizers
torch-clip a library to improve optimization methods by clipping off heavy-tailed gradient. This makes it possible to increase the accuracy and speed of convergence during the training of neural networks on a specific number of tasks.
Installation
you can install our library using pip:
pip install torch-clip
numpy~=1.20.0
torch~=1.11.0+cu113
matplotlib~=3.4.3
tqdm~=4.62.3
What do you need us for?
In the last few years, for various neural network training models (for example, BERT + CoLA), it has been found that in the case of "large stochastic gradients", it is advantageous to use special clipping (clipping/normalization) of the batched gradient. Since all modern machine learning, one way or another, ultimately boils down to stochastic optimization problems, the question of exactly how to "clip" large values of batched gradients plays a key role in the development of effective numerical training methods for a large class of models. This repository implements optimizers for the pytorch library with different clipping methods.
Our repository
The source code and research results can be found at the link: https://github.com/EugGolovanov/TorchClippedOptimizers
Use example
You can use our optimizers as well as all the standard optimizers from the pytorch library
from torch_clip.optimizers import ClippedSGD
optimizer = ClippedSGD(model.parameters(), lr=5e-2, momentum=0.9, clipping_type="layer_wise", clipping_level=1)
loss = my_loss_function
for epoch in range(EPOCHS):
for i, data in enumerate(train_loader, 0):
outputs = net(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
optimizer.zero_grad()
Use example (with restarts)
from torch_clip.optimizers import ClippedSGD
from torch_clip.restarter import Restarter
from torch_clip.optimizers_collector import OptimizerProperties, ModelProperties, RestartProperties
loss = my_loss_function
model = my_model_object
optimizer_props = OptimizerProperties(ClippedSGD, lr=5e-2, momentum=0.9,
clipping_type="layer_wise", clipping_level=1)
restarter = Restarter(optimizer_properties=optimizer_props, first_restart_steps_cnt=50,
restart_coeff=1.25, max_steps_cnt=2000)
optimizer = optimizer_props.optimizer_class(model.parameters(), **optimizer_props.optimizer_kwargs)
for epoch in range(EPOCHS):
for i, data in enumerate(train_loader, 0):
outputs = model(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
optimizer.zero_grad()
restarter.add_coords(model.parameters())
optimizers = restarter.make_restart(net, optimizer)
Metadata
Release files for torch-clip 0.0.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| torch_clip-0.0.6.tar.gz | 12.1 kB | Without provenance | Details for torch_clip-0.0.6.tar.gz |
Release files / torch_clip-0.0.6.tar.gz
| Download URL | torch_clip-0.0.6.tar.gz |
|---|---|
| Size | 12.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
34ab255b33e4c5371bebe6b582b89be6633097e3ec47d1781c4ea121ca8d1ef7
|
|
BLAKE2b-256 checksum How to use checksums |
595fc10d9df2ce115a106b0a879e43d2ad40e67b34b1552b108031d873f2407d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.9.7
|
Security for torch-clip 0.0.6
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.