Cosine Annealing Linear Warmup
Project description
Cosine Annealing Scheduler with Linear Warmup
Implementation of a Cosine Annealing Scheduler with Linear Warmup and Restarts in PyTorch.
It has support for multiple parameters groups and minimum target learning rates.
Also works with the Lightning Modules!
Installation
pip install 'git+https://github.com/santurini/cosine-annealing-linear-warmup'
Usage
It is important to specify the parameters groups in the optimizer instantiation as the learning rates are directly inferred from the wrapped optimizer.
Example: Multiple groups
from cosine_warmup import CosineAnnealingLinearWarmup
optimizer = torch.optim.Adam([
{"params": first_group_params, "lr": 1e-3},
{"params": second_group_params, "lr": 1e-4},
]
)
scheduler = CosineAnnealingLinearWarmup(
optimizer = optimizer,
min_lrs = [ 1e-5, 1e-6 ],
first_cycle_steps = 1000,
warmup_steps = 500,
gamma = 0.9
)
# this is equivalent to
scheduler = CosineAnnealingLinearWarmup(
optimizer = optimizer,
min_lrs_pow = 2,
first_cycle_steps = 1000,
warmup_steps = 500,
gamma = 0.9
)
Example: Single groups
from cosine_linear_warmup import CosineAnnealingLinearWarmup
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
scheduler = CosineAnnealingLinearWarmup(
optimizer = optimizer,
min_lrs = [ 1e-5 ],
first_cycle_steps = 1000,
warmup_steps = 500,
gamma = 0.9
)
# this is equivalent to
scheduler = CosineAnnealingLinearWarmup(
optimizer = optimizer,
min_lrs_pow = 2,
first_cycle_steps = 1000,
warmup_steps = 500,
gamma = 0.9
)
Visual Example
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
cosine-warmup-0.0.0.tar.gz
(7.5 kB
view details)
Built Distribution
File details
Details for the file cosine-warmup-0.0.0.tar.gz
.
File metadata
- Download URL: cosine-warmup-0.0.0.tar.gz
- Upload date:
- Size: 7.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.16
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | c7aa9c4483888a6f47471912292cbeaf96389747fea55cc8d794db865f1f336f |
|
MD5 | 3590d40610a0e81bbe85ee3f59aef98a |
|
BLAKE2b-256 | b94d2b586fc3c15276b57b96ba3368f4d8dde054d8ecb4a3603aaa1f5be365f4 |
File details
Details for the file cosine_warmup-0.0.0-py3-none-any.whl
.
File metadata
- Download URL: cosine_warmup-0.0.0-py3-none-any.whl
- Upload date:
- Size: 8.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.16
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8e3d4c0fa0057368282fd1c44d8cba636bb31a124ed56da38624ffe87b747748 |
|
MD5 | e05c757a461a0e38a7afba7e98c2d47a |
|
BLAKE2b-256 | 11bfd5115a46a9546f79c7aad2f234eeab876721c2a93395623385fb40e584e9 |