luminav 1.2.1

---

license: apache-2.0 language:

• en tags:
• pytorch
• optimizer
• custom-optimizer
• triton
• memory-efficient
• stochastic-rounding
• low-vram
• bf16
• fp16
• adamw
• cuda
• llm
• transformer
• deep-learning
• research
• 3am-engineering

---

---

Notice: Official Upstream Repository

This repository (cloverx-id/LuminaV-Optimizer-Paper) is the official standalone and living development repository for the LuminaV optimizer family.

While LuminaV was originally conceived and validated as the core engine for the XoneLM-1.0 language model series, all subsequent optimizer upgrades, low-precision Triton kernels, PyTorch standards compliance, and bug fixes are actively maintained and released directly in this repository.

---

What's New in v1.2.1 (Latest Release)

The v1.2.1 release delivers critical GPU pipeline acceleration, eliminates host-device blocking synchronization, and ensures 100% compliance with modern PyTorch compilation and strict determinism:

• Zero-Sync In-Kernel Parameter Norm Reduction: Fused the parameter Euclidean norm squared ∑(p_i²) calculation directly into Pass 1 Triton reduction kernels (_lumina_v2_pass1_kernel and _lumina_v1_pass1_kernel). Pass 2 and Pass 3 load and resolve norm_p entirely within GPU SRAM/registers. Completely eradicated p_contig.float().norm().item(), saving 200–400 synchronous PCIe roundtrips and pipeline stalls per training step on deep Transformer models.

• CUDA Graphs & torch.compile Compatibility: Eradicated all host-blocking .item() calls, resolving fatal TorchDynamo graph breaks and making the optimizer step fully captureable via CUDA Graphs (torch.accelerator.Graph).

• Stateless CPU Golden-Ratio PRNG Hashing: Replaced dynamic GPU RNG tensor allocation with a pure CPU integer multiplicative hash using Knuth's 32-bit golden ratio constant: (step * 0x9E3779B9 + i * 10007) & 0x7FFFFFFF. Generates uncorrelated 31-bit integer seeds per layer in nanoseconds without launching GPU kernels.

• Strict Deterministic Compliance (torch.use_deterministic_algorithms): Injected dedicated per-device generator caches (self._generators) into fallback stochastic rounding (_sr_update). Prevents stochastic rounding from mutating PyTorch's default CUDA RNG state, ensuring bit-for-bit reproducibility for RLHF, Dropout, and DataLoader shuffling.

• Consolidated Batch Scratch Buffer Reset: Replaced hundreds of per-slice memset micro-calls with a single consolidated scratch.zero_() reset per step, removing up to 400–800 driver launch overheads per iteration.

  For the full version history and detailed patch notes, see CHANGELOG.md.)

---

Overview

LuminaV is a master-free, memory-efficient adaptive optimizer engineered specifically for deep learning workloads running directly in low precision (FP16 / BF16) without maintaining redundant 4-byte FP32 master weights.

By combining Centered Innovation Variance, Hyperbolic Tangent (tanh) Coordinate Bounding, a Directional Traffic-Cop Mask, and On-Chip Bitwise Stochastic Rounding, LuminaV eliminates the standard 16-byte-per-parameter memory tax imposed by AdamW while avoiding weight freezing and gradient shocks.

---

Key Features

1. Zero Master-Weight Copies: Directly mutates parameter weights in native FP16 or BF16, eliminating the 4-byte FP32 master weight allocation.
2. On-Chip Bitwise Stochastic Rounding (SR): Implements in-register bitcast hashing in Triton to provide unbiased stochastic rounding, preventing weight stagnation during fine-grained updates or learning rate decay.
3. Hyperbolic tanh Bounding Envelope: Maps normalized momentum through a (-1.0, 1.0) transfer function, guaranteeing coordinate updates cannot explode beyond the step learning rate.
4. The Traffic-Cop Directional Gate: Dynamically eliminates coordinate updates whenever historical momentum conflicts with the incoming mini-batch gradient direction (u_t · g_t ≤ 0).
5. Centered Innovation Variance: Tracks centered innovation dispersion (g_t - m_t)² rather than uncentered raw second moments, suppressing variance inflation during confident descent.
6. Automatic FP16 Cliff Governor: Built-in asymptotic boundary governor that dampens steps near the IEEE-754 FP16 overflow limit (> 65,504), enabling stable pure FP16 training without external schedulers or clipping.
7. Dual Execution Engine: Fully accelerated custom OpenAI Triton kernels for CUDA devices, paired with vectorized C++ torch._foreach multi-tensor fallbacks.

---

Installation

From PyPI (Recommended)

pip install luminav

For GPU acceleration via OpenAI Triton:

pip install luminav[triton]

From Source (Editable Mode)

git clone https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper
cd LuminaV-Optimizer-Paper
pip install -e .

Direct File Drop-in

Alternatively, you can copy luminav.py directly into your working project directory without packaging overhead:

wget https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper/raw/main/luminav.py

---

Quickstart

Standard Instantiation

import torch
from luminav import LuminaV

# Instantiate your model in native low precision (e.g. BF16 or FP16)
model = YourModel().to(device="cuda", dtype=torch.float16)

# Initialize LuminaV v1.2.1
optimizer = LuminaV(
    model.parameters(),
    lr=8e-4,                    # or 8e-5 / 8e-6 for fine-tuning
    betas=(0.9, 0.999),
    eps=1e-8,
    weight_decay=0.08,
    tau=0.8,
    alpha_ss=0.5,
    cautious=True,
    cautious_clamp_min=0.5,     # Exact power-of-two ceiling (2.00x)
    buffer=2,                   # 2 = Dual-Buffer (Standard), 1 = Single-Buffer (Extreme Low VRAM)
    stochastic_rounding=True,
    bound=True,                 # Smooth asymptotic step bounding
    bound_type="radial",        # "radial" (preserves 100% angular direction) or "coordinate"
    bound_ratio=0.03,
    execution="auto"
)

# Standard training step
optimizer.zero_grad(set_to_none=True)
loss = model(inputs, targets)
loss.backward()
optimizer.step()

Loading from config.json

import json
import torch
from luminav import LuminaV

with open("config.json", "r") as f:
    config = json.load(f)

# Initialize with verified default configuration
optimizer = LuminaV(model.parameters(), **config["default_params"])

---

Parameter Reference

Parameter	Type	Default	Description
params	iterable	Required	Iterable of parameters to optimize or dicts defining parameter groups.
lr	float	8e-4	Learning rate (η).
betas	Tuple[float, float]	(0.9, 0.999)	Coefficients (β₁, β₂) for running momentum and centered innovation variance.
eps	float	1e-8	Numerical stability term (ε). Automatically floored to 1e-4 in FP16 to prevent subnormal underflow.
weight_decay	float	8e-2	Decoupled weight decay coefficient (λ).
tau	float	0.8	Analytical bias correction temperature parameter (τ).
alpha_ss	float	0.5	Softsign dampening factor (α_ss) used in single-buffer mode (buffer=1).
cautious	bool	True	If True, enables Traffic-Cop directional verification masking.
cautious_clamp_min	float	0.5	Safety floor density clamp (γ_min) enforcing a power-of-two maximum energy scaling ceiling (2.00x, 2¹) and preventing division by zero.
buffer	int	2	Buffer mode: 2 (Dual-buffer tracking m_t and v_t) or 1 (Single-buffer scalar RMS tracking).
stochastic_rounding	bool	True	Enables bitwise stochastic rounding on native FP16/BF16 weights.
bound	bool	True	If True, enables smooth asymptotic parameter bounding to prevent divergence in deep networks.
bound_type	str	"radial"	Asymptotic bounding formulation: "radial" (direction-preserving squashing using tanh(r)/r) or "coordinate" (elementwise squashing).
bound_ratio	float	0.03	Maximum allowed step displacement ratio relative to parameter norm or magnitude (R = bound_ratio * ‖p‖).
execution	str	"auto"	Execution engine: "auto", "triton", "foreach", or "single".

---

Operational Modes

LuminaV-2 (Dual-Buffer Default: buffer=2)

Maintains first moment m_t and centered innovation variance v_t:

$$ m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t $$

$$ v_t = \beta_2 v_{t-1} + (1 - \beta_2)(g_t - m_t)^2 $$

Updates are bounded through the hyperbolic tangent envelope:

$$ u_t = \tanh\left(\frac{\tilde{m}_t}{\sigma_t}\right) $$

LuminaV-1 (Single-Buffer Extreme-Poverty Mode: buffer=1)

Collapses variance tracking into a scalar Root-Mean-Square (RMS) across the entire tensor, saving 50% optimizer state memory by maintaining only a single state buffer (m_t):

$$ \text{RMS}(\tilde{m}t) = \sqrt{\frac{1}{N} \sum{i=1}^N \tilde{m}_{t,i}^2 + \epsilon} $$

$$ u_t = \tanh\left(\frac{z}{1 + \alpha_{ss}|z|}\right), \quad z = \frac{\tilde{m}_t}{\tau \cdot \text{RMS}(\tilde{m}_t) + \epsilon(1 - \beta_1^t)\tau} $$

---

Contributors & Acknowledgements

LuminaV is developed and maintained by Silver Moon (@cloverxion) and the Lumina Moon community contributors.

For the complete list of individuals who have contributed code, mathematical analyses, and experimental validation, please refer to CONTRIBUTORS.md.

---

Citation

If you utilize LuminaV in your research or applications, please cite both the foundational paper and this software implementation:

# 1. To cite the official research paper & theoretical mechanics
@misc{luminamoon2026luminav_paper,
  author       = {{Silver Moon (cloverxion)}},
  organization = {Lumina Moon},
  title        = {{LuminaV: We Were Too Broke for AdamW So We Trapped Gradients in a Hyperbolic Straitjacket and Hired a Traffic Cop to Slap Them}},
  year         = {2026},
  publisher    = {Hugging Face},
  doi          = {10.57967/hf/10270},
  url          = {https://huggingface.co/cloverx-id/XoneLM-1.0-Paper}
}

# 2. To cite this software implementation & standalone codebase
@software{luminamoon2026luminav_code,
  author       = {{Silver Moon (cloverxion) and Lumina Moon Contributors}},
  organization = {Lumina Moon},
  title        = {{LuminaV Optimizer: Official PyTorch Implementation}},
  year         = {2026},
  publisher    = {Hugging Face / PyPI},
  version      = {1.2.1},
  doi          = {10.57967/hf/10365},
  url          = {https://huggingface.co/cloverx-id/LuminaV-Optimizer-Paper}
}

---

License

Apache License 2.0. See LICENSE for full terms.