Skip to main content

AFOR

AFOR is a tensor-wise adaptive forgetting optimizer for PyTorch. It adapts the second-moment decay coefficient from local gradient residuals, direction consistency, and online Z-score normalization.

Installation

pip install afor-optimizer

Usage

import torch
from AFOR import afor

model = torch.nn.Linear(10, 2)
optimizer = afor(
    model.parameters(),
    lr=1e-3,
    betas=(0.9, 0.999),
    beta2_min=0.99,
    dir_weight=1.0,
    weight_decay=1e-4,
)

inputs = torch.randn(16, 10)
targets = torch.randint(0, 2, (16,))
loss = torch.nn.functional.cross_entropy(model(inputs), targets)
loss.backward()
optimizer.step()
optimizer.zero_grad()

Main Parameters

  • lr: learning rate.
  • betas: first-moment and initial second-moment coefficients.
  • beta2_min: lower bound for the adaptive second-moment coefficient.
  • dir_weight: weight of gradient-direction consistency.
  • eps: numerical stability term.
  • weight_decay: decoupled weight decay.
  • fast_ref: use the larger fast/slow noise estimate as the noise reference.

AFOR currently supports dense gradients only.

License

Released under the MIT License. See LICENSE.

Metadata

Release files for afor-optimizer 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for afor-optimizer 0.1.0
File Size Uploaded Provenance Actions
afor_optimizer-0.1.0.tar.gz 4.6 kB Without provenance Details for afor_optimizer-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for afor-optimizer 0.1.0
File Interpreter ABI Platform Actions
afor_optimizer-0.1.0-py3-none-any.whl Python 3 none any Details for afor_optimizer-0.1.0-py3-none-any.whl

Total release size: 9.4 kB

Release files / afor_optimizer-0.1.0.tar.gz

Download URL afor_optimizer-0.1.0.tar.gz
Size 4.6 kB
Tags Source
SHA-256 checksum
How to use checksums
a519d28b1a67d620ee8055997d0325e56184065c99a5d2b17cc47917c26f513c
BLAKE2b-256 checksum
How to use checksums
a8be8993b2fdf6459403b3983f76fc5d765c09a45c12c49f059ecc3891d76902
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.18

Release files / afor_optimizer-0.1.0-py3-none-any.whl

Download URL afor_optimizer-0.1.0-py3-none-any.whl
Size 4.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
cb7cdd866e2e943f9315ca651786df82faf136255a743b288771436639c980bf
BLAKE2b-256 checksum
How to use checksums
f056b94910837c9aa693b5b0f6c64d7a7b16550f6bf421e078e13768e40b5973
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.18

Security for afor-optimizer 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page