hypertensor 1.0.1

Per-weight precision + CUDA kernels for PyTorch

HyperTensor

Per-weight mixed-precision for PyTorch — faster inference, lower VRAM, near-FP32 accuracy

HyperTensor enables mixed-precision execution inside a single PyTorch model by assigning numerical precision at the individual weight level (INT4, INT8, FP16, BF16, or FP32) based on measured reconstruction error.

This means a single model can dynamically mix precisions across weights in the same forward pass while remaining fully compatible with standard PyTorch.

In (TinyLLaMA_benchmark.py), converting a model from FP32 to HyperTensor achieved ~2× inference speedup with minimal accuracy loss, alongside significant VRAM reduction on consumer GPUs.

Under the hood, a precision planner builds a per-weight format map, which is compiled into optimized execution buffers and executed using custom CUDA kernels and fused mixed-precision operators.

HyperTensor is modular: the planner, runtime, and model wrappers can be used independently or combined for research and deployment of adaptive precision neural networks.

---

Install (PyPI)

pip install hypertensor

---

Why HyperTensor?

Modern neural networks waste compute.

Most models run entirely in FP32 (or globally in FP16/BF16), even though:

• many weights are safe in INT8 or INT4
• some benefit from FP16/BF16
• only a small fraction require FP32

HyperTensor addresses this by measuring reconstruction error per weight and selecting the lowest precision that preserves stability.

---

What it does

HyperTensor:

• analyzes each weight independently
• assigns the lowest safe precision
• builds mixed-precision execution buffers
• executes INT4 / INT8 / FP16 / BF16 / FP32 within a single model

Result:

• reduced VRAM usage
• faster inference
• stable accuracy relative to FP32 baselines

---

Usage Example

import torch
import torch.nn as nn
import hypertensor
from HyperTensor import WeightOption

class BenchmarkMLP(nn.Module):
    def __init__(self, dim=4096, depth=4, refresh_every=8):
        super().__init__()
        layers = []
        for _ in range(depth):
            layers.append(
                hypertensor.HyperLinear(
                    dim,
                    dim,
                    refresh_every=refresh_every,
                    prefer_bf16=True,
                )
            )
            layers.append(nn.ReLU())
        self.net = nn.Sequential(*layers)

    def forward(self, x):
        return self.net(x)

---

Convert model

plan = hypertensor.scan_model(
    model,
    eps_fp16=0.0,
    eps_bf16=0.0,
    prefer_bf16=True,
)

model_ht = hypertensor.wrap_model_with_hypertensor(
    model,
    plan,
    option=WeightOption.BEST,
    inference_only=True,
)

---

Key Features

• Per-weight and per-block precision selection
• INT4 / INT8 / FP16 / BF16 / FP32 execution
• Precision planner modes: LOWEST, HIGHEST, ALLOW, CLOSEST, AVERAGE, BEST, PER_WEIGHT, PER_BLOCK
• Drop-in HyperLinear replacement for nn.Linear
• HyperAttention for transformer architectures
• Training + inference support
• Inference-only mode (removes FP32 master weights)
• Custom CUDA kernels for INT4/INT8/FP16/BF16
• Serializable quantized models + precision plans

---

Design Philosophy

HyperTensor is built to be:

• Accuracy-first — decisions based on measured reconstruction error
• Granular — operates at weight or block level
• Composable — planner, runtime, and modules are independent
• Training-aware — supports STE + custom backward kernels
• Deployment-focused — inference-only mode minimizes memory
• Format-minimal — only INT4, INT8, FP16, BF16, FP32

---

Benchmarks

All benchmarks were run on an NVIDIA RTX 4080 Laptop GPU (12GB VRAM) using a fixed-seed experimental setup.

The benchmark suite evaluates a 4096-dimensional MLP under both training and inference workloads across all precision planning modes.

---

TinyLLaMA Benchmark

End-to-end transformer evaluation:

• FP32 baseline measured
• Model converted to HyperTensor
• Inference repeated under identical inputs

Result:

• ~2× inference speedup
• Stable output behavior relative to FP32

---

Key Results

• Training speed: ~FP32 equivalent
• Training memory: increased (expected due to buffers)
• Inference speed: improved across most modes
• Inference memory: significantly reduced
• Accuracy: stable within evaluated drift bounds

---

Installation (from source)

git clone https://github.com/CoolPuzzler/HyperTensor.git
cd HyperTensor
pip install -e .

---

Requirements

• Python ≥ 3.12
• PyTorch with CUDA
• NVIDIA GPU (sm_89 target default)
• Optional: ninja (faster CUDA builds)

CUDA extensions compile automatically on first import.

---

Citation / Project Status

HyperTensor is an active research system for fine-grained mixed-precision execution in deep neural networks.

---

License

This project is licensed under the MIT License. See the LICENSE file for full details.