hypertensor 1.0.0

Per-weight precision + CUDA kernels for PyTorch

HyperTensor

Per-weight mixed-precision for PyTorch. Faster inference, lower VRAM, same accuracy.

HyperTensor is a precision-adaptive quantization framework that assigns the lowest safe numerical precision to each individual weight in a neural network.

---

Why HyperTensor?

Modern neural networks waste compute.

Most models run entirely in FP32 (or globally in FP16/BF16), even though:

• many weights work fine in INT8 or INT4
• some benefit from FP16/BF16
• only a small fraction truly require FP32

HyperTensor solves this by measuring error per weight and selecting the minimum precision that preserves stability.

---

What it does

HyperTensor:

• analyzes each weight independently
• assigns the lowest safe precision
• builds mixed-precision execution buffers
• executes using INT4 / INT8 / FP16 / BF16 / FP32 — within the same model

Result:

• faster inference
• significantly lower VRAM usage
• predictable, stable accuracy

---

Key Features

• Per-weight and per-block precision selection
• INT4 / INT8 / FP16 / BF16 / FP32 execution
• Flexible precision planner: LOWEST, HIGHEST, ALLOW, CLOSEST, AVERAGE, BEST, PER_WEIGHT, PER_BLOCK
• Drop-in HyperLinear layer
• HyperAttention module for LLaMA-style architectures
• Training + inference support
• Inference-only mode (removes FP32 master weights)
• Custom CUDA kernels:
  • INT4 / INT8 GEMM
  • FP16 / BF16 backward
• Serialization of quantized models and precision plans

---

Design Philosophy

HyperTensor is built to be:

• Accuracy-first — every decision is based on measured error
• Granular — operates at the weight level, not just per tensor
• Composable — use the planner, modules, or full stack independently
• Training-aware — STE, backward kernels, and caching are first-class
• Deployment-focused — inference-only mode minimizes VRAM
• Format-minimal — INT4, INT8, FP16, BF16, FP32 only

The system is designed to be extended with new planner modes, kernels, and modules.

---

Benchmarks

All benchmarks were run on an NVIDIA RTX 4080 Laptop GPU (12GB VRAM), reflecting realistic consumer hardware performance.

The main benchmark suite (benchmark.py) evaluates training and inference on a 4096-dim MLP (batch size 16).

---

Training Benchmarks

Training Speed

Training speed remains close to FP32 (~17–18 ms/step). Most planner modes introduce minimal overhead.

Training Memory Usage

Memory usage increases due to quantization buffers and error tracking. PER_WEIGHT and PER_BLOCK are the most expensive; FP32 remains lowest.

Training Drift vs FP32

All modes maintain stable accuracy (~0.31–0.33 MAE), including aggressive configurations.

---

Inference Benchmarks

Inference Speed

Inference improves across most modes. AVERAGE is the fastest, but all benefit from reduced precision.

Inference Memory Usage

Memory drops significantly:

• FP32: >500 MB
• HyperTensor: ~210 MB typical
• PER_BLOCK: ~147 MB (lowest)

Inference Drift vs FP32

Accuracy remains stable (~0.32 MAE) even under aggressive quantization.

---

TinyLLaMA Benchmark

An end-to-end LLM benchmark is included in TinyLLaMA_benchmark.py.

Results:

=== HyperTensor LLM Test ===

[1] Loading FP32 model... Running FP32 benchmark... FP32 inference: 108.732 ms

[2] Unloading FP32 model... [3] Converting to HyperTensor... Scanning weights (CPU)... Applying mixed precision (mode = best)... Moving to GPU... Running HyperTensor benchmark... HyperTensor inference: 49.795 ms

=== Summary === FP32: 108.732 ms HyperTensor: 49.795 ms Speedup: 2.18x

HyperTensor reduces TinyLLaMA inference latency by over 2× while maintaining stable accuracy.

---

Combined Results

Summary:

• Training speed ≈ FP32
• Training memory increases (expected)
• Inference speed improves
• Inference memory drops significantly
• Accuracy remains stable across all modes

HyperTensor delivers consistent accuracy, strong inference performance, and substantial VRAM savings on a 12GB GPU.

---

Documentation

See DOCUMENTATION.md for:

• usage examples
• architecture details
• advanced configuration

---

Installation

git clone https://github.com/CoolPuzzler/HyperTensor.git HyperTensor
cd HyperTensor
pip install -e .

---

Requirements

• Python ≥ 3.12
• PyTorch with CUDA
• NVIDIA GPU (sm_89 target by default)
• Optional: ninja for faster builds

CUDA extensions are compiled on first import via torch.utils.cpp_extension.load.

---

License

This project is licensed under the MIT License. See the LICENSE file for full details.