custom_float_lib

A from-scratch IEEE-754-style floating point simulator for exploring low-precision number formats used in LLM quantization (FP8, FP6, FP4, bfloat16, TF32, and the OCP Microscaling family) - built to understand why these formats work the way they do, not just to call a library that already does it.

What this is

Every format in this library is implemented at the bit level: sign, exponent, mantissa, biasing, subnormals, infinities, and NaN - built up from IEEE-754 binary64 decoding rather than wrapping an existing quantization library. The goal was to actually understand quantization mechanics (guard/round/sticky rounding, round-to-nearest-even tie breaking, exponent overflow, subnormal underflow) well enough to explain every number it produces, then use that understanding to quantize a real language model (GPT-2) and measure the real accuracy/size tradeoff.

Features

• 10 named format presets covering general-purpose training formats (bf16, fp16, tf32) and the OCP Microscaling family used on current accelerator hardware (MXFP8, MXFP6, MXFP4 / NVFP4), plus additional FP8 splits from quantization research
• Three rounding modes: round-to-nearest-even (default), truncate, and stochastic rounding (with seeded reproducibility)
• Vectorized batch quantization (quantize_array) using real numpy bit operations - not a Python loop - validated bit-for-bit identical to the scalar path across thousands of constructed test cases
• Per-tensor scaling (quantize_array_scaled) to keep narrow formats (FP4/FP6) from overflowing to inf on real weight distributions
• A working end-to-end LLM quantization pipeline: load a real HuggingFace model, quantize its weights through any preset, and measure the perplexity impact against the fp32 baseline

Format presets

Quickstart

from custom_float_lib import CustomFloat
import numpy as np

# Scalar
cf = CustomFloat.mxfp8_e4m3()
cf.quantize(3.14159265)  # -> 3.25

# Vectorized (real model tensors, not just scalars)
weights = np.random.randn(4096, 4096)
quantized = cf.quantize_array(weights)

# Rounding modes
cf = CustomFloat.mxfp8_e4m3(rounding="stochastic", seed=42)  # unbiased in expectation
cf = CustomFloat.mxfp8_e4m3(rounding="truncate")              # cheap, deterministic, biased

# Per-tensor scaling - required for narrow formats (FP4/FP6) to avoid
# overflowing to inf on real weight magnitudes
quantized, scale = cf.quantize_array_scaled(weights)

Quantizing a real model

from transformers import GPT2LMHeadModel
from custom_float_lib import CustomFloat
from quantize_model import quantize_model_copy, evaluate_loss, perplexity_from_loss

model = GPT2LMHeadModel.from_pretrained("gpt2")
cf = CustomFloat.mxfp8_e4m3()
quantized_model = quantize_model_copy(model, cf, verbose=True)

Validated results (GPT-2, 124M params)

Perplexity on a fixed eval sample, fp32 baseline = 97.37:

8-bit formats are near-lossless, matching published results. The narrow 4-bit/6-bit formats degrade sharply under per-tensor scaling - this is a known, expected limitation (not a bug): real MXFP4/NVFP4 deployments always use per-block scaling plus additional calibration, which is the natural next experiment on top of this codebase.

On-disk size was independently verified, not just estimated: casting fp8_e4m3fn-quantized weights into PyTorch's native torch.float8_e4m3fn dtype produced a real 124.86 MB file against a 497.76 MB fp32 baseline (74.9% reduction), matching the theoretical bits-per-value estimate (124.80 MB) to within 0.06 MB.

Validation methodology

Every arithmetic path was cross-checked before being trusted on a real model:

• Scalar vs. vectorized outputs compared bit-for-bit across special values, random normals, deliberately constructed subnormal-triggering and overflow-triggering values, and constructed RNE tie-boundary cases, across all 10 presets
• Stochastic rounding validated for seeded reproducibility (both scalar and vectorized paths) and empirically confirmed unbiased: mean of 20,000 draws landed ~73x closer to the true value than RNE's single deterministic estimate, on a coarse format
• Root-caused every anomaly found (FP64 exponent field width bug, reserved-exponent collision on overflow, subnormal boundary handling) back to the bit-level cause rather than patching symptoms

Known limitations / roadmap

See TODO.md for the full list. Notably:

• Overflow currently saturates to infinity; a saturate-to-max-finite mode (matching real hardware formats like e4m3fn) is not yet implemented
• Scaling is per-tensor only; per-block scaling (what real MX hardware uses) is not yet implemented - this is the most impactful next step for narrow-format accuracy
• No native low-bit storage for formats without a PyTorch dtype (mxfp6, mxfp4) - quantization here simulates precision loss but does not shrink files on disk for those formats
• Not yet packaged for PyPI (no pyproject.toml, no formal test suite)

Project structure

custom_float_lib/
├── __init__.py       # public API re-exports
├── core.py            # CustomFloat class, scalar quantize/encode/decode
├── presets.py          # named format presets
└── vectorized.py       # numpy-vectorized batch quantization
quantize_model.py        # torch model quantization + eval helpers