mantissa

A low-precision neural-network compute core in C, with a Python binding.

mantissa is a from-scratch AI compute core: dense layers and 2-D convolution, forward and backward, with weights and activations stored in a narrow float format of your choice — bfloat16, fp16, FP8 (E4M3/E5M2), FP4 (E2M1), or a custom 1·7·24 type — while every accumulator stays float32. That is the standard mixed-precision recipe, narrow storage / wide accumulate, and it is the whole design: on a dense layer the budget is memory bandwidth, so halving the stored size of a weight is worth nearly as much as halving the runtime.

It is dependency-free C11 with hand-written NEON and runtime-dispatched AVX2 kernels in one portable binary, a thread pool, and a ctypes Python layer. It runs on the CPU today.

Roadmap (not yet implemented): a GPU backend, and the sequence models the same primitive already implies — word2vec, GRU/LSTM, and transformer-shaped work. See DESIGN.md §3 for why tk_linear_forward is the shared building block of all of them.

Every performance number in this repository is measured, and the counter-stories are written down too: PERFORMANCE.md says where narrow storage does not pay, and DESIGN.md lists the optimizations that were tried and rejected with the measurement that killed them.

Started by Tekin Ertekin (2024); later refactored with Claude Code — see AUTHORS.md.

Version history and per-release benchmark scores: RELEASES.md.

Low-precision, in numbers

mantissa keeps weights and activations in a narrow storage format and accumulates in float32 — narrow storage, wide accumulate. To show what that buys, a linear MNIST classifier (784→10) is trained once in float32, then its weights are post-training-quantized into each storage format via Mantissa.prepare() (rebuild the core with make DTYPE=N). Same weights, same test set — only the storage type changes:

fp8 holds float32 accuracy at 4× less memory (−0.1 pt); fp16/bf16 are effectively lossless at 2×. fp4 is a step too far for naïve post-training quantization — 4-bit needs per-block scaling or quantization-aware training (a good first contribution — see CONTRIBUTING.md).

Run it yourself: make DTYPE=4 lib && python examples/mnist_demo.py trains a classifier on MNIST and reports accuracy, throughput, and footprint at the chosen storage precision (needs only numpy + matplotlib).

_{The linear model tops out near 81 %; the point is the relative cost of each format, not the absolute number. Convolutions currently run in float32, so this low-precision story is about the dense/linear engine.}

How it gets its speed: see docs/PERFORMANCE.md — the SIMD kernels (NEON + runtime-dispatched AVX2), im2col+GEMM convolution, and threading, each with measured numbers.

The mantissa family

Part of the mantissa family: a low-precision engine written in C, with small Python packages built on top. Each package sits under the one it depends on — ⭐ marks where you are, and every other name links to its repo.

• ⭐ mantissa — low-precision neural-network engine in C (the core) (you are here)
  • mantissa-perceptron — perceptron & ADALINE, the linear classics
  • mantissa-nn — shared neural-net primitives (layers, engine binding)
    • mantissa-cnn — convolutional networks for images
      • mantissa-auto-encoder — autoencoders for denoising & super-resolution
      • mantissa-interpret — CNN interpretability (occlusion, saliency, Grad-CAM)
      • mantissa-embed — CNN metric learning (image embeddings for similarity & retrieval)
    • mantissa-mlp — multilayer perceptrons, fully-connected nets

How it works

mantissa is a shared library with a plain C ABI, so any language that can call C — Python (via ctypes), C/C++, Rust, Go — drives the same engine. The caller passes ordinary float32 arrays in and gets float32 back; inside, weights are kept in a narrow type to save memory.

A call is one of three kinds:

• Build — quantize weights from float32 into the compact storage type (once).
• Forward pass (tk_linear_forward) — compute y = activation(W·x + b). Called forward because data flows forward through the network, input → output (also feed-forward). There is no "fast" in it — the opposite direction is the backward pass.
• Backward pass / training (tk_linear_backward → tk_sgd_step) — gradients flow backward, output → input, and the weights are updated.

Whichever it is, the compute core does the same underneath: read the narrow weights, accumulate in float32, apply an activation, return float32.

Performance at a glance

A 2048×2048 dense layer (4.2M parameters) on an Apple M-series laptop, default bfloat16 (make bench / make benchbp):

Forward and backward use explicit NEON kernels (bfloat16 leads — half the bytes, same FMAs) and a persistent thread pool that splits rows across cores for large layers. Numbers are for the default bfloat16 on a 10-core Apple laptop.

Memory is the headline at scale — it decides whether a model loads at all:

How it stays small and fast

1. Narrow weight storage — weights are the bulk of a model, so storing each in 2 bytes (or 1) instead of 4 is where the RAM/VRAM savings come from, and since a dense layer is memory-bandwidth bound, moving fewer bytes is also faster. The precision is a build-time dial (below).
2. float32 accumulation — compute always sums in float32 so accuracy holds across a layer's millions of terms (mixed precision; Micikevicius 2017).
3. Tuned kernels — register-blocked GEMV with explicit SIMD FMA kernels (NEON on arm64, AVX2 on x86-64) and a persistent thread pool across cores, branchless activations (sign-bit / fmax / copysign, chosen by benchmark), and stochastic rounding so training survives narrow weights. Details and measurements in docs/DESIGN.md.

The precision dial

Storage precision is one build-time knob (DTYPE); the default needs no config. It is how the core trades accuracy for memory — a means to the speed/RAM goal, not the goal itself.

*fp4 is stored one value per byte today (1 B); the "½" is the logical E2M1 width. Two-per-byte packing is a roadmap item gated on block-scaling — see docs/DESIGN.md.

Bit layouts, the tekin formats' design rationale, the hot/cold conversion split, and the current-research context (MX / NVFP4 / posit / IEEE P3109) all live in docs/DESIGN.md. Every value is stored in the selected type but computed in float32.

Accuracy cost, seen

The same value stored in each format and read back (make DTYPE=<n> test). This is the accuracy you trade away for the memory you save — float32 is the reference column:

fp4_e2m1 is coarser still: its 8 magnitudes are {0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6} — everything rounds onto that grid.

Full benchmarks

The numbers behind Performance at a glance, per dtype. make DTYPE=<n> bench — a 2048×2048 dense layer (4.2M params), plus the activation-dispatch micro-test. Apple M-series laptop, clang -O3; indicative, not absolute.

The forward pass is register-blocked (4 output rows share each x load and run 4 independent FMA chains), with explicit SIMD FMA kernels for float32 and bfloat16 — NEON on arm64, AVX2 on x86-64 (runtime-dispatched via __builtin_cpu_supports, scalar fallback on older CPUs). bfloat16 leads on both axes: it moves half the bytes of float32 for the same FMAs. tekin8 stays conversion- bound (E4M3→float unpack, no SIMD; native FP8 hardware removes that).

On top, a thread pool splits the output rows across cores for large layers, both forward and backward (the per-kernel numbers above are single-thread):

Scaling is sub-linear because GEMV does only ~2 FLOPs per byte, so it saturates memory bandwidth before compute — float32 (twice the bytes) barely gains, which is exactly why the narrow default matters. Small layers run serially (below a work threshold) so the pool never hurts the millions-of-small-calls case; set MANTISSA_THREADS to tune. Numbers are laptop-noisy.

Backward pass (make DTYPE=<n> benchbp, same 2048×2048 layer):

Activation dispatch (4M elements): a per-element switch beats a resolved function pointer ~7× for relu and ~1.5× for sigmoid on Apple Silicon — on the x86 CI runner the gap shrinks to ≤1.2× (parity on some cases), so the win is real but platform-dependent. The inline switch vectorizes (relu compiles to a single fmax per lane); an indirect call per element does not. So tk_activate keeps the switch, and the function-pointer API (tk_act_resolve) is reserved for genuinely pluggable dispatch. The step/sign/relu kernels themselves are branchless (sign-bit read, fmax, copysign), each picked by benchmark over the obvious comparison. Measure, don't assume.

Mixed architectures

Every layer configures itself independently — bias is a per-call NULL-able pointer, activation is a per-call argument:

tk_linear_forward(W1, x,  b1,   h1, 6, 4, TK_ACT_TANH);     /* bias + tanh    */
tk_linear_forward(W2, h1, NULL, h2, 5, 6, TK_ACT_RELU);     /* no bias + relu */
tk_linear_forward(W3, h2, b3,   y,  2, 5, TK_ACT_SIGMOID);  /* bias + sigmoid */

(Under the default narrow storage each float output is requantized to tk_scalar_t before it feeds the next layer — see examples/mlp_example.c; the snippet elides that for readability. It compiles as-is only under DTYPE=0.)

That is a full 3-layer MLP with three different bias/activation setups — exactly the heterogeneity a Transformer needs (bias-free attention projections, bias-using feed-forward). Run it: make mlp.

Training (back-propagation)

The reverse pass mirrors the forward core: tk_linear_backward computes the weight, bias, and input gradients for a dense layer; tk_sgd_step updates narrow-stored weights; tk_loss (MSE / BCE) seeds the gradient. No autograd graph — the caller drives the layers, keeping everything explicit and inspectable.

One training step is a loop: run forward, measure the loss against the target, propagate gradients backward, update the weights, repeat.

Correctness is proven by a gradient check (make testbp): analytic gradients vs central finite differences, matching to <1e-2 relative error for tanh, sigmoid, relu, and gelu.

make train learns XOR — the problem a single perceptron cannot solve — end to end, in the default bfloat16:

Training XOR  (dtype=bfloat16, stochastic_rounding=1)
  epoch    0  loss 0.31523
  epoch 1000  loss 0.00035
  epoch 4000  loss 0.00007
predictions:
  (0,0) -> 0.004   (0,1) -> 0.990   (1,0) -> 0.991   (1,1) -> 0.009

That it converges in bf16 is the point of stochastic rounding (config TK_USE_STOCHASTIC_ROUNDING): under plain round-to-nearest, a weight update smaller than the storage type's precision rounds to zero and training stalls; SR rounds up/down with probability proportional to distance, so tiny updates accumulate in expectation (Gupta et al., 2015; the technique behind FP8 training on Hopper/Blackwell). It needs no fp32 master copy of the weights.

Measured — same XOR run, 4000 epochs, round-to-nearest vs SR (make DTYPE=<n> benchbp):

In the 1-byte type, plain rounding never learns XOR; stochastic rounding does. That is the whole reason the technique exists.

Training config (all OFF by default)

Each sets a default; the runtime tk_optim / dropout calls override per layer. Back-propagation covers the dense layer (the shared primitive) and, since v0.2.1, the conv/pool family (conv.h, float32 — see the roadmap section); recurrent backward reuses the same pieces later.

Zero-config

Never touch config.h and you get Google's bfloat16 with bias enabled — the safe general-purpose default. Override only when you want to:

make DTYPE=4 test     # switch storage to tekin8

Install

pip install mantissa-core       # import name stays `mantissa`

The wheel ships a prebuilt libmantissa (default bfloat16) — no compiler or make needed. numpy is optional; install mantissa-core[numpy] for the zero-copy training path. Then:

from mantissa import Mantissa, STEP
tk = Mantissa()
print(tk.dtype)                # 'bfloat16'

The distribution is named mantissa-core on PyPI (the bare mantissa name is taken by an unrelated project); the Python import name is still mantissa.

Installing from an sdist (or pip install . from a checkout) compiles the C core on your machine, so a C toolchain (cc, make-grade compiler) is required for that path — wheel users need nothing. Override the storage dtype at build time with MANTISSA_DTYPE=<id> (same ids as DTYPE; default 2 = bfloat16):

pip install .                       # from a source checkout
MANTISSA_DTYPE=4 pip install .      # build the tekin8 (FP8) library instead

Download

Prebuilt shared libraries are attached to every release — built by CI for Linux, macOS, and Windows. Grab the one for your OS and load it straight from Python, no compiler or make needed:

import ctypes
lib = ctypes.CDLL("./libmantissa-linux-x86_64.so")   # the file you downloaded
lib.tk_dtype_name.restype = ctypes.c_char_p
print(lib.tk_dtype_name().decode())                  # -> e.g. "bfloat16"

Or use the ctypes wrapper in python/mantissa/ (point it at the downloaded file) — pass float32 numpy arrays (or array('f')) for a zero-copy, in-place training path, ~200× faster per step than plain lists (numpy optional). For an epoch loop, Mantissa.trainer() binds the buffers' pointers once and roughly halves the per-epoch call cost again (measured 9.8 → 4.8 µs at 1030×4 — the conversion, not the C, was the cost); see docs/USAGE.md. Runnable forward + back-prop examples for Python, C++, C#, Java, JavaScript, and Rust live in clients/ — all calling the same C ABI. To build from source instead, see below.

For a full, installable client with honest benchmarks, see the sister project mantissa-perceptron — a Python perceptron classifier built on this engine.

Quick start

make test                     # tests, default bfloat16
make example                  # C perceptron
make mlp                      # mixed 3-layer MLP
make DTYPE=0 bench            # benchmark, float32
make dist && python3 python/perceptron_example.py

The Python binding is dtype-agnostic: it calls the float32 entry point, so the same script runs against any storage type the library was built for. Full API and examples in docs/USAGE.md.

Numeric landscape & roadmap

The low-precision frontier moves fast; mantissa tracks it deliberately:

• Microscaling (MX) — OCP MX v1.0 (2023): blocks of 32 elements share one E8M0 scale, mitigating the tiny dynamic range of 4/6-bit elements. MXFP8, MXFP6, MXFP4.
• NVFP4 — NVIDIA Blackwell (2024 hardware): 16-element blocks with an FP8 E4M3 block scale; used to pretrain LLMs at 4 bits (arXiv:2509.25149, 2025).
• Posit / takum — tapered-precision alternatives to IEEE floats, strong for ≤8-bit inference on zero-centered weights.
• IEEE P3109 — an emerging standard for ML arithmetic formats.

mantissa implements the element formats (E4M3, E5M2, E2M1); block-level microscaling (a shared per-block scale) is the next planned step. The conv/pool primitives landed in v0.2.1 (conv.h): im2col + register-blocked GEMM convolution, max pooling with argmax scatter, a batched dense head, fused softmax-cross-entropy and plain-f32 SGD — the full CNN training family, gradient-checked (make testconv) and benched at LeNet-5/VGG shapes (make benchconv; VGG 64→64@3×3 forward 199 GFLOP/s threaded on M4). This family is deliberately pure float32: narrow-storage conv stays on the roadmap, gated on the same block-scaling work as the 4-bit packing. Recurrent backward will reuse the same gradient and optimizer pieces.

Project layout

include/   config, dtypes + conversions, activations, ops, loss, backprop, conv, pool
src/       implementations (conv.c: the float32 CNN family — conv2d, maxpool, softmax-xent)
tests/     forward round-trip checks (7 formats) + backprop & conv gradient checks
examples/  perceptron, mixed-MLP, XOR training
bench/     GEMV + activation-dispatch benchmark, conv shapes (benchconv)
python/    ctypes binding + Python perceptron & training examples
clients/   forward + back-prop demos in C++, C#, Java, JavaScript, Rust
docs/      DESIGN.md (numerics, optimization), USAGE.md (API + examples)

License

MIT — see LICENSE. © 2024 Tekin Ertekin.