NanoGEMM ⚡

🤝 Community, Issues & Support

• 🐛 Found a bug, numeric issue, or SIMD edge case? Please Open an Issue — reports are tracked and resolved quickly!
• 💬 Questions, performance ideas, or new kernels? Join our GitHub Discussions.
• ⭐ Find NanoGEMM useful? Give it a star on GitHub — it helps more engineers and researchers discover it!
• ☕ Support development (USDT TRC-20): TDVbEdnpgNgoAhNcn1EwXxWHiR3RKLit5P (Verify on TronScan)

🏆 Featured in The Daily Diff: Selected as a top AI & HPC story (Interest: 9/10, Depth: 9/10, Utility: 9/10). Ranked in GitHub Trending under Deep Learning topics.

📦 Multi-Platform Installation

pip install nanogemm
# or with uv
uv add nanogemm

NanoGEMM is a minimalist, bare-metal General Matrix Multiplication (GEMM) engine designed for sub-microsecond CPU inference and high-performance computing in Python.

Built with direct AVX2 / FMA (256-bit SIMD) and ARM NEON (128-bit SIMD) assembly-level register tiling and cache blocking, NanoGEMM eliminates the heavy function-call dispatch, thread-pool barriers, and memory-packing overhead of heavyweight BLAS libraries (OpenBLAS, MKL) for small-to-medium tensors.

🧩 Universal Compatibility Matrix

🚀 Performance Benchmarks

Measured on Intel/AMD x86-64 CPU (AVX2 + FMA) against NumPy 2.2.3 (single-precision float32):

⚡ Quantized INT8 SIMD GEMM Performance (matmul_int8)

Measured live on Google Colab CPU (AVX2) against NumPy (int8 x int8 -> int32):

🧠 Batched Matrix Multiplication (BMM) for Transformer Attention (bmm)

Measured live on Google Colab CPU (Multi-Head Attention $Q @ K^T$ shapes):

💡 Why is NanoGEMM faster on small/medium matrices?
Traditional BLAS engines incur 3–10 µs of fixed overhead per invocation due to dynamic runtime dispatch, argument sanitization, thread synchronization, and packing buffers. NanoGEMM utilizes a zero-allocation, direct register-tiled microkernel that executes in sub-microsecond time immediately upon invocation.

⚙️ Benchmark Environment & Test Configuration

📌 Single-Thread Design Note: NanoGEMM runs strictly on a single CPU core without thread pools, pthreads, or mutex barriers. For small matrices ($16 \times 16$ to $64 \times 64$), thread synchronization in OpenMP costs more cycles than the matrix multiply itself. NumPy/OpenBLAS incurs dispatch latency, whereas NanoGEMM enters CPU registers directly.

⚖️ Architectural Comparison: NanoGEMM vs OpenBLAS vs LIBXSMM

When evaluating CPU matrix multiplication engines, understanding the design trade-offs between compilation models, binary footprint, dispatch latency, and target matrix shapes is critical:

💡 Key Design Rationale:

1. NanoGEMM vs OpenBLAS / MKL on Small Tensors:
   Traditional BLAS libraries are optimized for massive matrices ($1000 \times 1000+$). On small matrices ($16 \times 16$ to $64 \times 64$), the fixed overhead of thread synchronization, buffer packing, and NumPy ufunc argument sanitization takes 3–5 µs before arithmetic even begins. NanoGEMM eliminates this glue overhead completely, achieving sub-microsecond latency in Python.
2. NanoGEMM vs LIBXSMM:
   LIBXSMM (Intel Labs) is the gold standard for small-matrix GEMM in pure C/C++ when runtime JIT code emission is permissible. NanoGEMM adopts a different engineering philosophy: Zero-JIT, zero dynamic allocations, zero external dependencies, and a ~100 KB footprint. NanoGEMM deploys seamlessly in hardened environments (where W^X / DEP security policies block runtime JIT emission) and installs instantly via standard Python wheels.
3. v0.3.4 Boundary Microkernel Acceleration & Colab Validation:
   In NanoGEMM v0.3.4, dedicated $4 \times 16$ and $2 \times 16$ boundary register tiles (plus $8$-wide column tiles) are introduced. Standard square power-of-two matrices ($8 \times 8$, $16 \times 16$, $32 \times 32$, $64 \times 64$) now compute 100% inside AVX2+FMA vector registers with zero scalar tail fallback, verified live in Google Colab.

🛠 Architectural Design

1. Register Tiling ($6 \times 16$ Microkernel)

• Register allocation: Utilizes 12 ymm registers (ymm0 – ymm11) as 256-bit floating-point accumulators storing a $6 \times 16$ tile of matrix $C$.
• Vector broadcast & FMA: Two ymm registers load vectors from $B$, while individual scalar elements of $A$ are broadcast across ymm using _mm256_set1_ps and accumulated via fused multiply-add (_mm256_fmadd_ps).
• Zero Spilling: Fits completely inside the 16 available x86-64 YMM registers without stack eviction.

2. Multi-Level Cache Blocking

• $L_1$ / $L_2$ Cache Tiling: Matrices are processed in cache blocks ($M_c = 64, N_c = 128, K_c = 128$) to maintain maximum L1/L2 data cache hit ratios and eliminate memory bus thrashing.
• Vectorized Edge Handling: Arbitrary matrix dimensions (non-multiples of 6 or 16) are processed using boundary SIMD edge loops without padding or buffer allocations.

       Matrix A (M x K)              Matrix B (K x N)
     [ . . . . . . . . ]           [ . . . ymm0 . . . ]
     [ . . . . . . . . ]           [ . . . ymm1 . . . ]
     [ a0 a1 a2 a3 . . ]     x     [ . . . . .  . . . ]
     [ . . . . . . . . ]           [ . . . . .  . . . ]
     [ . . . . . . . . ]
             │                             │
             └──────────────┬──────────────┘
                            ▼
                Matrix C (6 x 16 Tile)
             [ ymm0  ymm1  ] -> Row 0
             [ ymm2  ymm3  ] -> Row 1
             [ ymm4  ymm5  ] -> Row 2
             [ ymm6  ymm7  ] -> Row 3
             [ ymm8  ymm9  ] -> Row 4
             [ ymm10 ymm11 ] -> Row 5

3. ARM NEON ($4 \times 8$ Microkernel)

• Target Architecture: Apple Silicon (M1/M2/M3/M4) and Linux AArch64 (AWS Graviton, Ampere Altra, Neoverse).
• Register Tiling: Operates on 8 128-bit float32x4_t accumulator registers (c00–c31) computing a $4 \times 8$ tile of matrix $C$.
• Vector FMA: Inner loop uses vld1q_f32 vector loads and vmlaq_n_f32 fused multiply-accumulates. Standard power-of-two matrix sizes ($16\times 16$, $32\times 32$, $64\times 64$) tile evenly with zero scalar tail fallback.

📦 Installation & Quickstart

Installation via PyPI (Recommended)

pip install nanogemm
# or with uv
uv add nanogemm

Python Compatibility: Fully tested and verified across Python 3.9 through 3.15 (including 3.15.0rc2).

⚡ No-GIL & Free-Threaded Ready (PEP 703):

• Free-Threaded CPython: Explicitly declares Py_MOD_GIL_NOT_USED to run safely without re-enabling the GIL under python3.13t, 3.14t, and 3.15t.
• GIL-Releasing Operations: Automatically releases the GIL (Py_BEGIN_ALLOW_THREADS) during hardware SIMD execution, allowing Python threads (threading.Thread, ThreadPoolExecutor) to run matrix multiplications concurrently across multiple CPU cores without lock contention.

Build from Source

git clone https://github.com/eminsk/nanogemm.git
cd nanogemm
pip install -e .

Python Usage

import nanogemm as ng
import numpy as np

# Verify SIMD hardware acceleration
print("Active ISA:", ng.get_simd_isa())
# Output: Active ISA: AVX2+FMA (256-bit SIMD, 6x16 register tiling)

# Allocate input matrices
A = np.random.randn(32, 64).astype(np.float32)
B = np.random.randn(64, 128).astype(np.float32)

# Direct hardware-accelerated MatMul: C = A @ B (supports 2D, 3D, and 4D tensors)
C = ng.matmul(A, B)

# Pre-allocated zero-copy output buffer for maximum throughput:
out = np.empty((32, 128), dtype=np.float32)
ng.matmul(A, B, out=out)

# Batched Multi-Head Attention GEMM for Transformers (B x H x S x D):
Q = np.random.randn(8, 16, 32, 64).astype(np.float32)
K_t = np.random.randn(8, 16, 64, 32).astype(np.float32)
scores = ng.bmm(Q, K_t)  # Output: (8, 16, 32, 32) with zero Python loop overhead

# Quantized INT8 GEMM (int8 x int8 -> int32, up to 5.0x faster than NumPy):
A_i8 = np.random.randint(-128, 128, size=(32, 64), dtype=np.int8)
B_i8 = np.random.randint(-128, 128, size=(64, 128), dtype=np.int8)
C_i32 = ng.matmul_int8(A_i8, B_i8)

# End-to-end Quantized Linear Layer forward pass:
Y_float = ng.quantized_matmul(A_i8, B_i8, scale_a=0.05, scale_b=0.02)

# Standard BLAS SGEMM interface: C = alpha * (A @ B) + beta * C
res = ng.sgemm(A, B, alpha=2.0, beta=0.5, c=out)

🤖 Native Model Context Protocol (MCP) Server

NanoGEMM includes a native, zero-dependency Model Context Protocol (MCP) server (nanogemm-mcp), allowing AI agents (Claude Desktop, Cursor, Antigravity, OpenManus) to query host SIMD instruction sets, execute ultra-low latency GEMM arithmetic, and run live CPU microbenchmarks.

🛠 Available MCP Tools

🚀 Running the Server

Run directly via the CLI:

# Direct command
nanogemm mcp

# Or via dedicated script
nanogemm-mcp

⚙️ Client Integration Configurations

Cursor (.cursor/mcp.json)

{
  "mcpServers": {
    "nanogemm": {
      "command": "nanogemm-mcp"
    }
  }
}

Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "nanogemm": {
      "command": "python",
      "args": ["-m", "nanogemm.mcp_server"]
    }
  }
}

🚀 Interactive Google Colab Demo

Run NanoGEMM benchmarks interactively in your browser with zero local setup:

The Interactive Benchmark Notebook demonstrates:

• Zero-Setup Installation & Hardware ISA Detection: Compiles native C/AVX2 on Google Colab CPU in seconds.
• 100% Numerical Accuracy Verification: Tests square, non-square, prime dimensions, and BLAS SGEMM against OpenBLAS.
• Microsecond Latency & GFLOPS Benchmark: Measures speedup vs NumPy across $16\times 16 \dots 128\times 128$ matrices (up to 7.26x faster on $24\times 24$).
• Interactive Performance Plotting: Side-by-side latency & speedup bar/line charts using Matplotlib with Edge AI sweet spot highlights.
• Real-Time Edge AI Loop: Demonstrates 100,000 iterations of zero-allocation in-place multiplication (out=C) for $24\times 24$ and $32\times 32$.
• Batched Matrix Multiplication (BMM): Multi-Head Attention benchmarks ($Q @ K^T$) across 8, 16, 32, and 64 heads.
• Quantized INT8 SIMD GEMM: Measures speedup vs NumPy integer multiplication across $16\times 16 \dots 128\times 128$ (up to 5.0x faster).

🧪 Testing & Verification

Run the full pytest suite (covering IEEE-754 correctness, batched Transformer BMM, INT8 SIMD, and FASM assembly):

uv run --extra dev pytest -v
# or with standard pytest
pytest -v

Or run individual test scripts directly:

# Correctness vs reference NumPy OpenBLAS:
python tests/test_correctness.py

# Batched BMM & INT8 SIMD GEMM verification:
python tests/test_bmm_int8.py

# Standalone FASM assembly DLL bridge verification:
python tests/test_fasm.py

# Official performance benchmark vs NumPy BLAS:
python benchmarks/bench_vs_numpy.py

⚡ Standalone Flat Assembler (FASM) 32-bit & 64-bit Engines

NanoGEMM includes native standalone assembly implementations written in Flat Assembler (FASM) for both 64-bit and 32-bit architectures in the asm/ directory:

• x86-64 Engine (asm/nanogemm64.dll, asm/test_nanogemm64.exe):
  • Microkernel with $4 \times 16$ and $4 \times 8$ register-tiled AVX2+FMA instructions (vfmadd231ps, vbroadcastss, vmovups).
  • Complies strictly with the Microsoft x64 ABI (preserving non-volatile registers RBX, RSI, RDI, R12–R15, XMM6–XMM15).
  • Peak throughput exceeding 38–40 GFLOPS on a single CPU core.
• x86 32-bit Engine (asm/nanogemm32.dll, asm/test_nanogemm32.exe):
  • Vectorized $4 \times 4$ SSE2 microkernel (movups, shufps, mulps, addps) using cdecl calling convention.
  • Compatible with all 32-bit x86 environments and 64-bit Windows via WoW64 with zero external dependencies.
  • Delivers 13–15 GFLOPS in pure 32-bit mode.

Building & Running FASM Tests

:: Build all 4 binaries and run native executable suites
cd asm
build.bat

:: Run Python verification and NumPy comparison suite
python tests/test_fasm.py

🌐 High-Performance Systems Ecosystem

NanoGEMM is developed by @eminsk as part of an engineering ecosystem focused on low-level hardware performance, assembly programming, and AI agent infrastructure:

• 🧠 AgentJIT — Just-In-Time Compiler for AI Agent Trajectories with speculative de-optimization guards and Native MCP Server (pip install agentjit).
• ⚡ NanoVector — Bare-metal C99/AVX2 vector search & episodic memory engine (~120KB) with Native MCP Server (pip install nanovector).
• 🖥️ NanoRecall — 100% Private, offline desktop memory & semantic screen search engine with Native MCP Server (pip install nanorecall).
• 🛒 avito-sdk — Headless Avito scraping & data extraction SDK with price drop tracking, Playwright cookies, Telegram/VK bots, and Native MCP Server (pip install avito-sdk).
• 📊 xlsx_vievers — Headless Excel formula engine (129+ functions), desktop spreadsheet viewer, SIMD SSE2 math, and Native MCP Server (pip install xlsx-viewer-pro).
• 📈 yfinance-ta-patterns — Candlestick & chart pattern scanner with AI Confluence Scoring, Backtesting, and Native MCP Server (pip install yfinance-ta-patterns).
• 🎥 screenvideo — Lightweight desktop screen recorder featuring WASAPI loopback audio and a standalone pure x64 Flat Assembler (FASM) native edition.

☕ Support, Community & Donations

If you find this project valuable and would like to support ongoing development:

• ⭐ Star the Repository: If NanoGEMM accelerates your matrix multiplications, give us a star on GitHub — it helps more developers discover high-performance SIMD!
• 💬 Join Discussions: Have ideas, use cases, or hardware benchmarks? Start or join a thread in GitHub Discussions!
• ☕ Donate (USDT TRC-20):
  TDVbEdnpgNgoAhNcn1EwXxWHiR3RKLit5P
  (Network: TRON / TRC-20 | Verify on TronScan)

📄 License

MIT License — Copyright (c) 2026 eminsk.