LuminaV Optimizer

We Were Too Broke for AdamW So We Trapped Gradients in a Hyperbolic Straitjacket and Hired a Traffic Cop to Slap Them

Official Repository & Research Paper

• Official Repository & Issue Tracker: https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper
• Primary Research Paper Archive: https://huggingface.co/cloverx-id/XoneLM-1.0-Paper
• Software DOI: 10.57967/hf/10365
• Paper DOI: 10.57967/hf/10270

What's New in v1.1.5 (Latest Release)

The v1.1.5 release introduces the Zero-VRAM FP16 Numerical Safety Shield, resolving early pretraining instability and eliminating NaN collapses in pure half-precision training without allocating 4-byte master weights:

• Dynamic Epsilon Floor for FP16: Automatically floors effective epsilon to max(eps, 1e-4) for torch.float16 parameters, preventing the bias-corrected scale factor c2 from underflowing into IEEE-754 subnormal/zero limits (~5.96e-8) and eliminating step-1 0.0 / 0.0 = NaN errors on zero-gradient parameters (e.g., unselected vocabulary tokens, padding, or dropout paths).
• On-Chip Second-Moment Clamping: Added an upper-bound register ceiling (65,000.0) to the centered innovation variance in Triton kernels and PyTorch loops before storing into FP16 pointers. This prevents unclipped initial gradient spikes (|g| > 256) from exceeding the FP16 maximum dynamic range (65,504) and permanently saturating state buffers to +inf.
• Zero-Sigma Division Guard: Embedded an explicit sigma + 1e-6 protection across all Triton kernel passes and PyTorch fallbacks to ensure division safety during cold-start iterations.
• Native Hardware Capability Detection: Integrated _supports_native_bf16() to automatically detect Nvidia Ampere SM80+ architectures for optimal native execution.
• Empirically Proven on Tesla T4: Empirically verified by pretraining SmolLM-135M from scratch on Wikimedia Wikipedia in pure FP16 on an Nvidia Tesla T4 GPU for 100 consecutive steps with zero NaN occurrences—even with external gradient clipping completely disabled.

Overview

LuminaV is a master-free, memory-efficient adaptive optimizer engineered specifically for deep learning workloads running directly in low precision (FP16 / BF16) without maintaining redundant 4-byte FP32 master weights.

By combining Centered Innovation Variance, Hyperbolic Tangent (tanh) Coordinate Bounding, a Directional Traffic-Cop Mask, and On-Chip Bitwise Stochastic Rounding, LuminaV eliminates the standard 16-byte-per-parameter memory tax imposed by AdamW while preventing weight stagnation and numerical explosions.

Key Features

1. Zero Master-Weight Copies: Directly mutates parameter weights in native FP16 or BF16, eliminating the redundant 4-byte FP32 master weight allocation.
2. On-Chip Bitwise Stochastic Rounding (SR): Implements in-register bitcast hashing in Triton to provide unbiased stochastic rounding, preventing weight stagnation during fine-grained updates or learning rate decay.
3. Hyperbolic tanh Bounding Envelope: Maps normalized momentum through a (-1.0, 1.0) transfer function, guaranteeing coordinate updates cannot explode beyond the step learning rate.
4. The Traffic-Cop Directional Gate: Dynamically eliminates coordinate updates whenever historical momentum conflicts with the incoming mini-batch gradient direction (u_t · g_t <= 0).
5. Centered Innovation Variance: Tracks centered innovation dispersion (g_t - m_t)^2 rather than uncentered raw second moments, suppressing variance inflation during confident descent.
6. Dual Execution Engine: Fully accelerated custom OpenAI Triton kernels for CUDA devices, paired with vectorized C++ torch._foreach multi-tensor fallbacks.

Installation

Install directly via PyPI:

pip install luminav

For GPU acceleration via OpenAI Triton:

pip install luminav[triton]

Or install in editable mode from source:

git clone https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper
cd LuminaV-Optimizer-Paper
pip install -e .

Quickstart

import torch
import torch.nn as nn
from luminav import LuminaV

# 1. Instantiate model directly in native half precision (e.g. BF16 or FP16)
device = "cuda" if torch.cuda.is_available() else "cpu"
model = nn.Linear(1024, 1024).to(device=device, dtype=torch.float16)

# 2. Initialize LuminaV
optimizer = LuminaV(
    model.parameters(),
    lr=8e-4,
    betas=(0.9, 0.999),
    eps=1e-8,
    weight_decay=0.08,
    tau=0.8,
    buffer=2,                   # 2 = Dual-Buffer (Standard), 1 = Single-Buffer (Low VRAM)
    stochastic_rounding=True,
    execution="auto"
)

# 3. Standard training loop
data = torch.randn(32, 1024, device=device, dtype=torch.float16)
target = torch.randn(32, 1024, device=device, dtype=torch.float16)
criterion = nn.MSELoss()

optimizer.zero_grad(set_to_none=True)
output = model(data)
loss = criterion(output, target)
loss.backward()

optimizer.step()

Parameter Reference

Citation

If you utilize LuminaV in your research or applications, please cite both the foundational paper and this software implementation:

# 1. To cite the official research paper & theoretical mechanics
@misc{luminamoon2026luminav_paper,
  author       = {{Silver Moon (cloverxion)}},
  organization = {Lumina Moon},
  title        = {{LuminaV: We Were Too Broke for AdamW So We Trapped Gradients in a Hyperbolic Straitjacket and Hired a Traffic Cop to Slap Them}},
  year         = {2026},
  publisher    = {Hugging Face},
  doi          = {10.57967/hf/10270},
  url          = {https://huggingface.co/cloverx-id/XoneLM-1.0-Paper}
}

# 2. To cite this software implementation & standalone codebase
@software{luminamoon2026luminav_code,
  author       = {{Silver Moon (cloverxion)}},
  organization = {Lumina Moon},
  title        = {{LuminaV Optimizer: Official PyTorch Implementation}},
  year         = {2026},
  publisher    = {Hugging Face},
  version      = {1.1.5},
  doi          = {10.57967/hf/10365},
  url          = {https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper}
}

License

Apache License 2.0. See LICENSE for full terms.