NanoGEMM ⚡

🏆 Featured in The Daily Diff: Selected as a top AI & HPC story (Interest: 9/10, Depth: 9/10, Utility: 9/10). Ranked in GitHub Trending under Deep Learning topics.

NanoGEMM is a minimalist, bare-metal General Matrix Multiplication (GEMM) engine designed for sub-microsecond CPU inference and high-performance computing in Python.

Built with direct AVX2 / FMA (256-bit SIMD) and ARM NEON (128-bit SIMD) assembly-level register tiling and cache blocking, NanoGEMM eliminates the heavy function-call dispatch, thread-pool barriers, and memory-packing overhead of heavyweight BLAS libraries (OpenBLAS, MKL) for small-to-medium tensors.

🧩 Universal Compatibility Matrix

🚀 Performance Benchmarks

Measured on Intel/AMD x86-64 CPU (AVX2 + FMA) against NumPy 2.2.3 (single-precision float32):

⚡ Quantized INT8 SIMD GEMM Performance (matmul_int8)

Measured live on Google Colab CPU (AVX2) against NumPy (int8 x int8 -> int32):

🧠 Batched Matrix Multiplication (BMM) for Transformer Attention (bmm)

Measured live on Google Colab CPU (Multi-Head Attention $Q @ K^T$ shapes):

💡 Why is NanoGEMM faster on small/medium matrices?
Traditional BLAS engines incur 3–10 µs of fixed overhead per invocation due to dynamic runtime dispatch, argument sanitization, thread synchronization, and packing buffers. NanoGEMM utilizes a zero-allocation, direct register-tiled microkernel that executes in sub-microsecond time immediately upon invocation.

⚙️ Benchmark Environment & Test Configuration

📌 Single-Thread Design Note: NanoGEMM runs strictly on a single CPU core without thread pools, pthreads, or mutex barriers. For small matrices ($16 \times 16$ to $64 \times 64$), thread synchronization in OpenMP costs more cycles than the matrix multiply itself. NumPy/OpenBLAS incurs dispatch latency, whereas NanoGEMM enters CPU registers directly.

⚖️ Architectural Comparison: NanoGEMM vs OpenBLAS vs LIBXSMM

When evaluating CPU matrix multiplication engines, understanding the design trade-offs between compilation models, binary footprint, dispatch latency, and target matrix shapes is critical:

💡 Key Design Rationale:

1. NanoGEMM vs OpenBLAS / MKL on Small Tensors:
   Traditional BLAS libraries are optimized for massive matrices ($1000 \times 1000+$). On small matrices ($16 \times 16$ to $64 \times 64$), the fixed overhead of thread synchronization, buffer packing, and NumPy ufunc argument sanitization takes 3–5 µs before arithmetic even begins. NanoGEMM eliminates this glue overhead completely, achieving sub-microsecond latency in Python.
2. NanoGEMM vs LIBXSMM:
   LIBXSMM (Intel Labs) is the gold standard for small-matrix GEMM in pure C/C++ when runtime JIT code emission is permissible. NanoGEMM adopts a different engineering philosophy: Zero-JIT, zero dynamic allocations, zero external dependencies, and a ~100 KB footprint. NanoGEMM deploys seamlessly in hardened environments (where W^X / DEP security policies block runtime JIT emission) and installs instantly via standard Python wheels.
3. v0.3.4 Boundary Microkernel Acceleration & Colab Validation:
   In NanoGEMM v0.3.4, dedicated $4 \times 16$ and $2 \times 16$ boundary register tiles (plus $8$-wide column tiles) are introduced. Standard square power-of-two matrices ($8 \times 8$, $16 \times 16$, $32 \times 32$, $64 \times 64$) now compute 100% inside AVX2+FMA vector registers with zero scalar tail fallback, verified live in Google Colab.

🛠 Architectural Design

1. Register Tiling ($6 \times 16$ Microkernel)

• Register allocation: Utilizes 12 ymm registers (ymm0 – ymm11) as 256-bit floating-point accumulators storing a $6 \times 16$ tile of matrix $C$.
• Vector broadcast & FMA: Two ymm registers load vectors from $B$, while individual scalar elements of $A$ are broadcast across ymm using _mm256_set1_ps and accumulated via fused multiply-add (_mm256_fmadd_ps).
• Zero Spilling: Fits completely inside the 16 available x86-64 YMM registers without stack eviction.

2. Multi-Level Cache Blocking

• $L_1$ / $L_2$ Cache Tiling: Matrices are processed in cache blocks ($M_c = 64, N_c = 128, K_c = 128$) to maintain maximum L1/L2 data cache hit ratios and eliminate memory bus thrashing.
• Vectorized Edge Handling: Arbitrary matrix dimensions (non-multiples of 6 or 16) are processed using boundary SIMD edge loops without padding or buffer allocations.

       Matrix A (M x K)              Matrix B (K x N)
     [ . . . . . . . . ]           [ . . . ymm0 . . . ]
     [ . . . . . . . . ]           [ . . . ymm1 . . . ]
     [ a0 a1 a2 a3 . . ]     x     [ . . . . .  . . . ]
     [ . . . . . . . . ]           [ . . . . .  . . . ]
     [ . . . . . . . . ]
             │                             │
             └──────────────┬──────────────┘
                            ▼
                Matrix C (6 x 16 Tile)
             [ ymm0  ymm1  ] -> Row 0
             [ ymm2  ymm3  ] -> Row 1
             [ ymm4  ymm5  ] -> Row 2
             [ ymm6  ymm7  ] -> Row 3
             [ ymm8  ymm9  ] -> Row 4
             [ ymm10 ymm11 ] -> Row 5

3. ARM NEON ($4 \times 8$ Microkernel)

• Target Architecture: Apple Silicon (M1/M2/M3/M4) and Linux AArch64 (AWS Graviton, Ampere Altra, Neoverse).
• Register Tiling: Operates on 8 128-bit float32x4_t accumulator registers (c00–c31) computing a $4 \times 8$ tile of matrix $C$.
• Vector FMA: Inner loop uses vld1q_f32 vector loads and vmlaq_n_f32 fused multiply-accumulates. Standard power-of-two matrix sizes ($16\times 16$, $32\times 32$, $64\times 64$) tile evenly with zero scalar tail fallback.

📦 Installation & Quickstart

Installation via PyPI (Recommended)

pip install nanogemm
# or with uv
uv add nanogemm

Python Compatibility: Fully tested and verified across Python 3.9 through 3.15 (including 3.15.0rc2).

⚡ No-GIL & Free-Threaded Ready (PEP 703):

• Free-Threaded CPython: Explicitly declares Py_MOD_GIL_NOT_USED to run safely without re-enabling the GIL under python3.13t, 3.14t, and 3.15t.
• GIL-Releasing Operations: Automatically releases the GIL (Py_BEGIN_ALLOW_THREADS) during hardware SIMD execution, allowing Python threads (threading.Thread, ThreadPoolExecutor) to run matrix multiplications concurrently across multiple CPU cores without lock contention.

Build from Source

git clone https://github.com/eminsk/nanogemm.git
cd nanogemm
pip install -e .

Python Usage

import nanogemm as ng
import numpy as np

# Verify SIMD hardware acceleration
print("Active ISA:", ng.get_simd_isa())
# Output: Active ISA: AVX2+FMA (256-bit SIMD, 6x16 register tiling)

# Allocate input matrices
A = np.random.randn(32, 64).astype(np.float32)
B = np.random.randn(64, 128).astype(np.float32)

# Direct hardware-accelerated MatMul: C = A @ B (supports 2D, 3D, and 4D tensors)
C = ng.matmul(A, B)

# Pre-allocated zero-copy output buffer for maximum throughput:
out = np.empty((32, 128), dtype=np.float32)
ng.matmul(A, B, out=out)

# Batched Multi-Head Attention GEMM for Transformers (B x H x S x D):
Q = np.random.randn(8, 16, 32, 64).astype(np.float32)
K_t = np.random.randn(8, 16, 64, 32).astype(np.float32)
scores = ng.bmm(Q, K_t)  # Output: (8, 16, 32, 32) with zero Python loop overhead

# Quantized INT8 GEMM (int8 x int8 -> int32, up to 5.0x faster than NumPy):
A_i8 = np.random.randint(-128, 128, size=(32, 64), dtype=np.int8)
B_i8 = np.random.randint(-128, 128, size=(64, 128), dtype=np.int8)
C_i32 = ng.matmul_int8(A_i8, B_i8)

# End-to-end Quantized Linear Layer forward pass:
Y_float = ng.quantized_matmul(A_i8, B_i8, scale_a=0.05, scale_b=0.02)

# Standard BLAS SGEMM interface: C = alpha * (A @ B) + beta * C
res = ng.sgemm(A, B, alpha=2.0, beta=0.5, c=out)

🚀 Interactive Google Colab Demo

Run NanoGEMM benchmarks interactively in your browser with zero local setup:

The Interactive Benchmark Notebook demonstrates:

• Zero-Setup Installation & Hardware ISA Detection: Compiles native C/AVX2 on Google Colab CPU in seconds.
• 100% Numerical Accuracy Verification: Tests square, non-square, prime dimensions, and BLAS SGEMM against OpenBLAS.
• Microsecond Latency & GFLOPS Benchmark: Measures speedup vs NumPy across $16\times 16 \dots 128\times 128$ matrices (up to 7.26x faster on $24\times 24$).
• Interactive Performance Plotting: Side-by-side latency & speedup bar/line charts using Matplotlib with Edge AI sweet spot highlights.
• Real-Time Edge AI Loop: Demonstrates 100,000 iterations of zero-allocation in-place multiplication (out=C) for $24\times 24$ and $32\times 32$.
• Batched Matrix Multiplication (BMM): Multi-Head Attention benchmarks ($Q @ K^T$) across 8, 16, 32, and 64 heads.
• Quantized INT8 SIMD GEMM: Measures speedup vs NumPy integer multiplication across $16\times 16 \dots 128\times 128$ (up to 5.0x faster).

🧪 Testing & Verification

Run the comprehensive correctness test suite comparing NanoGEMM with NumPy reference outputs across random uniforms, normals, non-square dimensions, and prime shapes:

python tests/test_correctness.py

Run the official benchmark against your installed NumPy BLAS:

python benchmarks/bench_vs_numpy.py

⚡ Standalone Flat Assembler (FASM) 32-bit & 64-bit Engines

NanoGEMM includes native standalone assembly implementations written in Flat Assembler (FASM) for both 64-bit and 32-bit architectures in the asm/ directory:

• x86-64 Engine (asm/nanogemm64.dll, asm/test_nanogemm64.exe):
  • Microkernel with $4 \times 16$ and $4 \times 8$ register-tiled AVX2+FMA instructions (vfmadd231ps, vbroadcastss, vmovups).
  • Complies strictly with the Microsoft x64 ABI (preserving non-volatile registers RBX, RSI, RDI, R12–R15, XMM6–XMM15).
  • Peak throughput exceeding 38–40 GFLOPS on a single CPU core.
• x86 32-bit Engine (asm/nanogemm32.dll, asm/test_nanogemm32.exe):
  • Vectorized $4 \times 4$ SSE2 microkernel (movups, shufps, mulps, addps) using cdecl calling convention.
  • Compatible with all 32-bit x86 environments and 64-bit Windows via WoW64 with zero external dependencies.
  • Delivers 13–15 GFLOPS in pure 32-bit mode.

Building & Running FASM Tests

:: Build all 4 binaries and run native executable suites
cd asm
build.bat

:: Run Python verification and NumPy comparison suite
python tests/test_fasm.py

🌐 High-Performance Systems Ecosystem

NanoGEMM is developed by @eminsk as part of an engineering ecosystem focused on low-level hardware performance, assembly programming, and native desktop computing:

• ⚡ NanoVector — Minimalist, bare-metal vector search & episodic memory engine for AI agents in ~120KB with AVX2+FMA, ARM NEON, and FASM x64 (pip install nanovector).
• 🎥 screenvideo — Lightweight desktop screen recorder featuring WASAPI loopback audio and a standalone pure x64 Flat Assembler (FASM) native edition.
• 📊 xlsx_vievers — Desktop spreadsheet processor with 80+ formula functions, Chart Wizard, and hardware-accelerated SIMD SSE2 math engine.
• 📈 yfinance-ta-patterns — Candlestick pattern scanner and AI ranking suite powered by TA-Lib and quantitative backtesting.
• 🔍 StackOverflowAPI — Desktop client for Stack Overflow built with CustomTkinter and native FASM x64 search client.

📄 License

MIT License — Copyright (c) 2026 eminsk.