AdamR
Adam with weight Recovery optimizer
TL;DR
AdamW tends to decay parameters towards zero, which makes the model "forget" the pretrained parameters during finetuning. Instead, AdamWR tries to recover parameters towards pretrained values during finetuning.
Have a try
Just like other PyTorch optimizers,
from adamr import AdamR
from xxx import SomeModel, SomeData, SomeDevice, SomeLoss
model = SomeModel()
dataloader = SomeData()
model.to(SomeDevice)
adamr = AdamR(
model.parameters(),
lr=1e-5,
betas=(0.9, 0.998), # Adam's beta parameters
eps=1e-8,
weight_recovery=0.1
)
loss_fn = SomeLoss()
for x, y in dataloader:
adamwr.zero_grad()
y_bar = model(x)
loss = loss_fn(y_bar, y)
loss.backward()
adamr.step()
Algorithm
TODO: improve the readability
Here is a paper snippet:
1 maintainer
Metadata
1 maintainer
Release files for adamr 0.0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| adamr-0.0.2.tar.gz | 316.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| adamr-0.0.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 327.6 kB
Release files / adamr-0.0.2.tar.gz
| Download URL | adamr-0.0.2.tar.gz |
|---|---|
| Size | 316.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
68eef97e26377a0d5791c2a587b30abc1a01f6de8ce9436e6f60d3db4c67a5a7
|
|
BLAKE2b-256 checksum How to use checksums |
70c7cfa5365a9de0267cb210c756cf2707198cc30f99751bbbe22693e4845ed5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.9
|
Release files / adamr-0.0.2-py3-none-any.whl
| Download URL | adamr-0.0.2-py3-none-any.whl |
|---|---|
| Size | 10.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f773611fb0058eb288a9751aba442daa9422d46e63411d93c0caa01d9c5e0ea5
|
|
BLAKE2b-256 checksum How to use checksums |
d872f92944fbd66d7c4d71cd3d51e1192051f4600ee03ddc9b19a637ad3a9d62
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.9
|
