Skip to main content

Cosine Annealing Linear Warmup

Project description

Cosine Annealing Scheduler with Linear Warmup

Implementation of a Cosine Annealing Scheduler with Linear Warmup and Restarts in PyTorch.
It has support for multiple parameters groups and minimum target learning rates.
Also works with the Lightning Modules!

Installation

pip install 'git+https://github.com/santurini/cosine-annealing-linear-warmup'

Usage

It is important to specify the parameters groups in the optimizer instantiation as the learning rates are directly inferred from the wrapped optimizer.

Example: Multiple groups

from cosine_warmup import CosineAnnealingLinearWarmup

optimizer = torch.optim.Adam([
    {"params": first_group_params, "lr": 1e-3},
    {"params": second_group_params, "lr": 1e-4},
    ]
)

scheduler = CosineAnnealingLinearWarmup(
    optimizer = optimizer,
    min_lrs = [ 1e-5, 1e-6 ],
    first_cycle_steps = 1000,
    warmup_steps = 500,
    gamma = 0.9
    )
    
# this is equivalent to

scheduler = CosineAnnealingLinearWarmup(
    optimizer = optimizer,
    min_lrs_pow = 2,
    first_cycle_steps = 1000,
    warmup_steps = 500,
    gamma = 0.9
    )

Example: Single groups

from cosine_linear_warmup import CosineAnnealingLinearWarmup

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

scheduler = CosineAnnealingLinearWarmup(
    optimizer = optimizer,
    min_lrs = [ 1e-5 ],
    first_cycle_steps = 1000,
    warmup_steps = 500,
    gamma = 0.9
    )
    
# this is equivalent to

scheduler = CosineAnnealingLinearWarmup(
    optimizer = optimizer,
    min_lrs_pow = 2,
    first_cycle_steps = 1000,
    warmup_steps = 500,
    gamma = 0.9
    )

Visual Example

Unknown-2

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cosine-warmup-0.0.0.tar.gz (7.5 kB view details)

Uploaded Source

Built Distribution

cosine_warmup-0.0.0-py3-none-any.whl (8.1 kB view details)

Uploaded Python 3

File details

Details for the file cosine-warmup-0.0.0.tar.gz.

File metadata

  • Download URL: cosine-warmup-0.0.0.tar.gz
  • Upload date:
  • Size: 7.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.16

File hashes

Hashes for cosine-warmup-0.0.0.tar.gz
Algorithm Hash digest
SHA256 c7aa9c4483888a6f47471912292cbeaf96389747fea55cc8d794db865f1f336f
MD5 3590d40610a0e81bbe85ee3f59aef98a
BLAKE2b-256 b94d2b586fc3c15276b57b96ba3368f4d8dde054d8ecb4a3603aaa1f5be365f4

See more details on using hashes here.

File details

Details for the file cosine_warmup-0.0.0-py3-none-any.whl.

File metadata

File hashes

Hashes for cosine_warmup-0.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 8e3d4c0fa0057368282fd1c44d8cba636bb31a124ed56da38624ffe87b747748
MD5 e05c757a461a0e38a7afba7e98c2d47a
BLAKE2b-256 11bfd5115a46a9546f79c7aad2f234eeab876721c2a93395623385fb40e584e9

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page