AdamR

Adam with weight Recovery optimizer

TL;DR

AdamW tends to decay parameters towards zero, which makes the model "forget" the pretrained parameters during finetuning. Instead, AdamWR tries to recover parameters towards pretrained values during finetuning.

Have a try

Just like other PyTorch optimizers,

from adamr import AdamR
from xxx import SomeModel, SomeData, SomeDevice, SomeLoss

model = SomeModel()
dataloader = SomeData()
model.to(SomeDevice)

adamr = AdamR(
   model.parameters(),
   lr=1e-5,
   betas=(0.9, 0.998), # Adam's beta parameters
   eps=1e-8,
   weight_recovery=0.1
   )

loss_fn = SomeLoss()

for x, y in dataloader:
   adamwr.zero_grad()
   y_bar = model(x)
   loss = loss_fn(y_bar, y)
   loss.backward()
   adamr.step()

Algorithm

TODO: improve the readability

Here is a paper snippet:

Metadata

Release files for adamr 0.0.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for adamr 0.0.2
File	Size	Uploaded	Provenance	Actions
adamr-0.0.2.tar.gz	316.8 kB	May 13, 2023	Without provenance	Details for adamr-0.0.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for adamr 0.0.2
File	Interpreter	ABI	Platform	Actions Reset
adamr-0.0.2-py3-none-any.whl 10.8 kB May 13, 2023 Without provenance	Python 3	none	any	Details for adamr-0.0.2-py3-none-any.whl

Total release size: 327.6 kB

Release files / adamr-0.0.2.tar.gz

Download URL	adamr-0.0.2.tar.gz
Size	316.8 kB
Tags	Source
SHA-256 checksum How to use checksums	`68eef97e26377a0d5791c2a587b30abc1a01f6de8ce9436e6f60d3db4c67a5a7`
BLAKE2b-256 checksum How to use checksums	`70c7cfa5365a9de0267cb210c756cf2707198cc30f99751bbbe22693e4845ed5`
Upload date	May 13, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.10.9`

Release files / adamr-0.0.2-py3-none-any.whl

Download URL	adamr-0.0.2-py3-none-any.whl
Size	10.8 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`f773611fb0058eb288a9751aba442daa9422d46e63411d93c0caa01d9c5e0ea5`
BLAKE2b-256 checksum How to use checksums	`d872f92944fbd66d7c4d71cd3d51e1192051f4600ee03ddc9b19a637ad3a9d62`
Upload date	May 13, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.10.9`

Security for adamr 0.0.2

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.0.2 This release

May 13, 2023 2 release files No provenance

0.0.1