Skip to main content

HyperTier

Tiered-memory inference runtime: run 400 GB AI models on 4 GB GPUs — layer streaming, KV-cache compression, Triton kernels.

License Status Python

⚠️ Early development. The public API does not exist yet. Follow the roadmap below or watch the repo for the first release (v0.1.0).


What is HyperTier?

HyperTier is an open-source, modality-agnostic inference engine and VRAM virtualizer written in Python, PyTorch and OpenAI Triton. Its goal is to run very large models — from 15 GB to 400 GB+ — on consumer GPUs with as little as 4 GB of VRAM (reference hardware: NVIDIA RTX 3050 Laptop, 32 GB RAM, NVMe SSD), cutting the active VRAM footprint by 95–99 % without destroying model accuracy.

It treats GPU memory as the fastest tier of a hierarchy:

VRAM (fast, tiny)  ◄──  pinned RAM (medium)  ◄──  NVMe SSD (large, slow)

and moves weights, experts and caches between tiers just in time, overlapping every transfer with computation.

Target workloads

  1. LLMs and giant MoE models — text, code, reasoning and decision-making agents.
  2. Video and image diffusion / DiT models — Wan 2.x, HunyuanVideo, LTX-Video, Flux.
  3. Multimodal and task-execution models.

The four pillars

Pillar What it does
1. Online KV-cache compression TurboQuant-style vector quantization (random rotation + Lloyd-Max codebooks) to ~2.5–3.5 bits per value, calibration-free, with fused Triton kernels that compute attention directly on compressed data.
2. Asynchronous layer & expert streaming Ping-pong double buffering over dedicated CUDA streams: block i computes while block i+1 is copied from pinned RAM or NVMe (safetensors), plus an LRU expert cache for MoE models.
3. Sequential slicing & VAE tiling Text encoder → DiT → VAE run as separate stages; token-chunked FFN/attention and spatio-temporal VAE tiling remove out-of-memory peaks in video generation.
4. Portability @triton.jit kernels for NVIDIA / AMD / Intel GPUs with no C++ build step, and a vectorized PyTorch (torch.compile) fallback everywhere else.

A bandwidth-aware planner decides what lives in each tier, based on the measured PCIe/NVMe bandwidth of the machine, and predicts throughput before running.

Honest expectations

Streaming makes models fit; the speed is then bounded by the slowest link the active weights must cross:

time_per_pass ≥ max(compute, non_resident_active_bytes / link_bandwidth)
  • Diffusion / video DiT: best case — compute per block hides the transfers.
  • MoE models: practical, thanks to expert caching (only active experts move).
  • Dense 400 GB models: they run, but at minutes per token. Suitable for offline/batch agents, not interactive chat.

Roadmap (v0.1.0 MVP)

Phase Deliverable
F0 Foundations: CI (CPU + Triton interpreter), hardware bandwidth probe, VRAM tracker
F1 Ping-pong layer streaming + NVMe tier + Hugging Face adapter
F2 TurboQuant KV cache + Triton kernels
F3 Video diffusion: sequential pipeline, chunking, VAE tiling (Wan 2.2 TI2V-5B)
F4 MoE expert cache + planner v0
F5 pip install hypertier v0.1.0 with reproducible examples on a 4 GB GPU

Design documents (architecture, bandwidth model and research survey): coming soon.

Installation

pip install hypertier   # placeholder release, no functionality yet

From source:

git clone https://github.com/vfernandezg-dev/HyperTier.git
cd HyperTier
pip install -e ".[dev]"
pytest

Contributing

Contributions, ideas and benchmarks on low-VRAM hardware are welcome. See CONTRIBUTING.md.

License

Licensed under the Apache License, Version 2.0.

HyperTier builds on published research (TurboQuant, PolarQuant, and others listed in the docs). NVIDIA, CUDA, Triton and other names are trademarks of their respective owners and are used here only to describe compatibility.

Metadata

Release files for hypertier 0.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hypertier 0.0.1
File Size Uploaded Provenance Actions
hypertier-0.0.1.tar.gz 9.0 kB Without provenance Details for hypertier-0.0.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for hypertier 0.0.1
File Interpreter ABI Platform Actions
hypertier-0.0.1-py3-none-any.whl Python 3 none any Details for hypertier-0.0.1-py3-none-any.whl

Total release size: 17.4 kB

Release files / hypertier-0.0.1.tar.gz

Download URL hypertier-0.0.1.tar.gz
Size 9.0 kB
Tags Source
SHA-256 checksum
How to use checksums
0323837d43289c48619eb8a2c60f2a83186e4968e93a4ba093b5accb12124699
BLAKE2b-256 checksum
How to use checksums
0490eb8b32ecbdbbc806c6cd1a9873ea7789956371e10cb19abc8563d0b3c70c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.5

Release files / hypertier-0.0.1-py3-none-any.whl

Download URL hypertier-0.0.1-py3-none-any.whl
Size 8.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
32e3e170c0c8b711fadd2f6103d8b3e4b61331b332581e64f1b5723e380aca5d
BLAKE2b-256 checksum
How to use checksums
a4f2b63f043f416e7e58a8586d6138482f8381c3121b82d7f50e9d7aef857d37
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.5

Security for hypertier 0.0.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.0.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page