Advanced Optimizers (AIO)

A comprehensive, all-in-one collection of state-of-the-art optimization algorithms for deep learning. Designed for maximum efficiency, minimal memory footprint, and superior performance across diverse model architectures and training scenarios.

PyPI version Python versions License

📦 Installation

pip install adv_optm

Requires PyTorch 2.3+ for torch.compile support.

What's New

🌟 Version 2.5.x: The Massive Refactor

This major update introduces a complete architectural refactor of the library:

🆕 New Optimizers & Scaling

  • SinkSGD_adv: Added a powerful new optimizer to the lineup.
  • Spectral Scaling: Now available across all optimizers, achieving width/rank invariant updates for highly stable training.

💾 Memory & State Precision Control

  • Granular State Precision (state_precision): Drastically reduce memory overhead with new optimizer state modes:
    • factored (Rank-2 factored mode)
    • fp32 (Full precision)
    • bf16_sr & int8_sr (BF16/Int8 with Stochastic Rounding)
  • Factored Second Moment (factored_2nd): Available for all Adam variants. Works seamlessly alongside any state_precision setting to further slash memory usage.

⚙️ Advanced Dynamics & Momentum

  • Variance Normalized Momentum (normed_momentum): Applies optimizer normalization before momentum (Normalization then Momentum/NtM). Available for AdamW_adv, SignSGD_adv, and SinkSGD_adv.
  • Universal Nesterov Momentum: Replaced the hard-to-tune Simplified_AdEMAMix with Nesterov momentum (nesterov) and a dedicated coefficient (nesterov_coef) across all optimizers.
  • Preconditioning & Signs:
    • Added Variance/Confidence Preconditioning (snr_cond) for SignSGD_adv and SinkSGD_adv (requires normed_momentum). Read the technical reports: AASS & sink-v.
    • Added Adaptive Stochastic Sign with $L_\infty$ preconditioning (stochastic_sign) for SignSGD_Adv and Lion_adv.
  • Improved CANS (accelerated_ns): Enhanced for Muon variants by integrating a dynamic lower bound.
  • New OrthoGrad modes (orthogonal_gradient): Standard OrthoGrad flattened and a new matrix-wise mode iterative.

⚓ Weight Decay Innovations

  • Centered Weight Decay (centered_wd): Pulls weights toward their pre-train state (anchor). To save memory, anchor precision (centered_wd_mode) can be set to full, float8, int8, or int4.
  • Fisher Weight Decay (fisher_wd): Now available for Adam variants based on the FAdam paper.
  • Geometric Weight Decay: Added specifically for SinkSGD_adv and SignSGD_adv.

(Note: Lion_Prodigy_adv, Simplified_AdEMAMix, and heuristic cautious/grams modes have been deprecated in favor of these superior, theoretically-grounded features).

Click to see older release notes (v1.2.x - v2.1.x)

Version 2.1.x

  • New Optimizer: Added Signum (SignSGD with momentum) to the SignSGD_adv family.

Version 2.0.x

  • ⚡ torch.compile Support: Fully implemented for all advanced optimizers. Enable via compiled_optimizer=True to heavily fuse and optimize the optimizer step path.
  • 📉 1-Bit Factored Mode: Vastly improved implementation via nnmf_factor=True.
  • 🛠️ Broad performance and stability improvements across all optimizers.

Version 1.2.x

  • Advanced Muon Variants: Brought the groundbreaking Muon optimizer into the fold, enriched with features from recent literature.
Optimizer Description
Muon_adv Advanced Muon implementation featuring CANS, NorMuon, Low-Rank Orthogonalization, and more.
AdaMuon_adv Combines Muon's geometry with Adam-like adaptive scaling and sign-based orthogonalization.
  • Prodigy Speedup: Prodigy variants are now 50% faster by eliminating unnecessary CUDA syncs (Shoutout to @dxqb!).
  • Stochastic Rounding for BF16: Parameter updates and weight decay now accumulate in float32 and round once at the end.
  • Cautious Weight Decay: Implemented for all advanced optimizers (Paper).
  • Fused Operations: Transitioned to fused and in-place operations wherever possible.

💡 Core Innovations

(Documentation expanding on the theory and usage of these features is coming soon!)

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for Korata_hiu from gravatar.com Korata_hiu

Credits

Author: Koratahiu

License

Apache Software License (Apache 2.0)

Requires

Python >=3.8

Tags

llm fine-tuning memory-efficient low-rank compression pytorch optimizer adam

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

adv_optm-2.5.13.tar.gz (60.9 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

adv_optm-2.5.13-py3-none-any.whl (85.7 kB view details)

Uploaded Python 3

File details

Details for the file adv_optm-2.5.13.tar.gz.

File metadata

  • Download URL: adv_optm-2.5.13.tar.gz
  • Upload date:
  • Size: 60.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.2.0 CPython/3.12.10

File hashes

Hashes for adv_optm-2.5.13.tar.gz
Algorithm Hash digest
SHA256 c6f3044a521482e892d44172961c90b07fafa172c7632ef054d7e033ba140fed
MD5 df3ec0744d069a8c72b50c8ba2bedc03
BLAKE2b-256 6dfae741bedb23b4e41d36703b6bca14b278b0a0dbc5890fb5ce1bb779caad00

See more details on using hashes here.

File details

Details for the file adv_optm-2.5.13-py3-none-any.whl.

File metadata

  • Download URL: adv_optm-2.5.13-py3-none-any.whl
  • Upload date:
  • Size: 85.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.2.0 CPython/3.12.10

File hashes

Hashes for adv_optm-2.5.13-py3-none-any.whl
Algorithm Hash digest
SHA256 8d5dae165cd4181098bfa46c92ac2f318e3caaf9342ce869035de52a9fe5de11
MD5 afb6ef0b7a9eaf13f4cdadb652eca652
BLAKE2b-256 ab99d54c423c50324e4171a911cecacc342797c4e3d7033b84bb1d5a16fb502e

See more details on using hashes here.

Release history Release notifications | RSS feed

Pre-release

2.6.1.dev3

2 files

Pre-release

2.6.1.dev2

2 files

Pre-release

2.6.1.dev1

2 files

Pre-release

2.6.dev1

2 files

This release

2.5.13 This release

2 files

2.5.12

2 files

2.5.11

2 files

2.5.10

2 files

2.5.9

2 files

2.5.8

2 files

2.5.7

2 files

2.5.6

2 files

2.5.5

2 files

2.5.4

2 files

2.5.3

2 files

2.5.2

2 files

2.5.1

2 files

2.5

2 files

Pre-release

2.4.dev25

2 files

Pre-release

2.4.dev24

2 files

Pre-release

2.4.dev23

2 files

Pre-release

2.4.dev22

2 files

Pre-release

2.4.dev21

2 files

Pre-release

2.4.dev20

2 files

Pre-release

2.4.dev19

2 files

Pre-release

2.4.dev18

2 files

Pre-release

2.4.dev17

2 files

Pre-release

2.4.dev16

2 files

Pre-release

2.4.dev15

2 files

Pre-release

2.4.dev14

2 files

Pre-release

2.4.dev13

2 files

Pre-release

2.4.dev12

2 files

Pre-release

2.4.dev11

2 files

Pre-release

2.4.dev10

2 files

Pre-release

2.4.dev9

2 files

Pre-release

2.4.dev8

2 files

Pre-release

2.4.dev7

2 files

Pre-release

2.4.dev6

2 files

Pre-release

2.4.dev5

2 files

Pre-release

2.4.dev4

2 files

Pre-release

2.4.dev2

2 files

Pre-release

2.4.dev1

2 files

Pre-release

2.3.dev3

2 files

Pre-release

2.3.dev2

2 files

Pre-release

2.3.dev1

2 files

2.2.3

2 files

2.2.2

2 files

2.2.1

2 files

Pre-release

2.2.1.dev2

2 files

Pre-release

2.2.1.dev1

2 files

2.2.0

2 files

Pre-release

2.2.dev5

2 files

Pre-release

2.2.dev4

2 files

Pre-release

2.2.dev3

2 files

Pre-release

2.2.dev2

2 files

Pre-release

2.2.dev1

2 files

2.1.0

2 files

Pre-release

2.1.dev5

2 files

Pre-release

2.1.dev3

2 files

Pre-release

2.1.dev2

2 files

Pre-release

2.1.dev1

2 files

2.0.1

2 files

2.0.0

2 files

1.4.1

2 files

1.4.0

2 files

1.3.4

2 files

1.3.3

2 files

1.3.2

2 files

1.3.1

2 files

1.3.0

2 files

Pre-release

1.3.dev1

2 files

1.2.13

2 files

1.2.12

2 files

1.2.11

2 files

1.2.10

2 files

1.2.9

2 files

1.2.8

2 files

1.2.7

2 files

1.2.6

2 files

1.2.5

2 files

1.2.4

2 files

1.2.3

2 files

1.2.2

2 files

1.2.1

2 files

1.2.0

2 files

Pre-release

1.2.dev20

2 files

Pre-release

1.2.dev19

2 files

Pre-release

1.2.dev18

2 files

Pre-release

1.2.dev17

2 files

Pre-release

1.2.dev16

2 files

Pre-release

1.2.dev15

2 files

Pre-release

1.2.dev14

2 files

Pre-release

1.2.dev13

2 files

Pre-release

1.2.dev12

2 files

Pre-release

1.2.dev11

2 files

Pre-release

1.2.dev10

2 files

Pre-release

1.2.dev9

2 files

Pre-release

1.2.dev8

2 files

Pre-release

1.2.dev7

2 files

Pre-release

1.2.dev6

2 files

Pre-release

1.2.dev5

2 files

Pre-release

1.2.dev4

2 files

Pre-release

1.2.dev3

2 files

Pre-release

1.2.dev2

2 files

Pre-release

1.2.dev1

2 files

1.1.4

2 files

1.1.3

2 files

1.1.2

2 files

1.1.1

2 files

1.1.0

2 files

Pre-release

1.1.0.dev5

2 files

Pre-release

1.1.0.dev4

2 files

Pre-release

1.1.0.dev3

2 files

Pre-release

1.1.0.dev2

2 files

Pre-release

1.1.0.dev1

2 files

1.0.6

2 files

1.0.5

2 files

1.0.4

2 files

1.0.3

2 files

1.0.1

2 files

1.0.0

2 files

0.1.9

2 files

0.1.8

2 files

0.1.7

2 files

0.1.6

2 files

0.1.5

2 files

0.1.4

2 files

0.1.3

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page