HyperTier
Tiered-memory inference runtime: run 400 GB AI models on 4 GB GPUs — layer streaming, KV-cache compression, Triton kernels.
⚠️ Early development. The public API does not exist yet. Follow the roadmap below or watch the repo for the first release (v0.1.0).
What is HyperTier?
HyperTier is an open-source, modality-agnostic inference engine and VRAM virtualizer written in Python, PyTorch and OpenAI Triton. Its goal is to run very large models — from 15 GB to 400 GB+ — on consumer GPUs with as little as 4 GB of VRAM (reference hardware: NVIDIA RTX 3050 Laptop, 32 GB RAM, NVMe SSD), cutting the active VRAM footprint by 95–99 % without destroying model accuracy.
It treats GPU memory as the fastest tier of a hierarchy:
VRAM (fast, tiny) ◄── pinned RAM (medium) ◄── NVMe SSD (large, slow)
and moves weights, experts and caches between tiers just in time, overlapping every transfer with computation.
Target workloads
- LLMs and giant MoE models — text, code, reasoning and decision-making agents.
- Video and image diffusion / DiT models — Wan 2.x, HunyuanVideo, LTX-Video, Flux.
- Multimodal and task-execution models.
The four pillars
| Pillar | What it does |
|---|---|
| 1. Online KV-cache compression | TurboQuant-style vector quantization (random rotation + Lloyd-Max codebooks) to ~2.5–3.5 bits per value, calibration-free, with fused Triton kernels that compute attention directly on compressed data. |
| 2. Asynchronous layer & expert streaming | Ping-pong double buffering over dedicated CUDA streams: block i computes while block i+1 is copied from pinned RAM or NVMe (safetensors), plus an LRU expert cache for MoE models. |
| 3. Sequential slicing & VAE tiling | Text encoder → DiT → VAE run as separate stages; token-chunked FFN/attention and spatio-temporal VAE tiling remove out-of-memory peaks in video generation. |
| 4. Portability | @triton.jit kernels for NVIDIA / AMD / Intel GPUs with no C++ build step, and a vectorized PyTorch (torch.compile) fallback everywhere else. |
A bandwidth-aware planner decides what lives in each tier, based on the measured PCIe/NVMe bandwidth of the machine, and predicts throughput before running.
Honest expectations
Streaming makes models fit; the speed is then bounded by the slowest link the active weights must cross:
time_per_pass ≥ max(compute, non_resident_active_bytes / link_bandwidth)
- Diffusion / video DiT: best case — compute per block hides the transfers.
- MoE models: practical, thanks to expert caching (only active experts move).
- Dense 400 GB models: they run, but at minutes per token. Suitable for offline/batch agents, not interactive chat.
Roadmap (v0.1.0 MVP)
| Phase | Deliverable |
|---|---|
| F0 | Foundations: CI (CPU + Triton interpreter), hardware bandwidth probe, VRAM tracker |
| F1 | Ping-pong layer streaming + NVMe tier + Hugging Face adapter |
| F2 | TurboQuant KV cache + Triton kernels |
| F3 | Video diffusion: sequential pipeline, chunking, VAE tiling (Wan 2.2 TI2V-5B) |
| F4 | MoE expert cache + planner v0 |
| F5 | pip install hypertier v0.1.0 with reproducible examples on a 4 GB GPU |
Design documents (architecture, bandwidth model and research survey): coming soon.
Installation
pip install hypertier # placeholder release, no functionality yet
From source:
git clone https://github.com/vfernandezg-dev/HyperTier.git
cd HyperTier
pip install -e ".[dev]"
pytest
Contributing
Contributions, ideas and benchmarks on low-VRAM hardware are welcome. See CONTRIBUTING.md.
License
Licensed under the Apache License, Version 2.0.
HyperTier builds on published research (TurboQuant, PolarQuant, and others listed in the docs). NVIDIA, CUDA, Triton and other names are trademarks of their respective owners and are used here only to describe compatibility.
Metadata
Release files for hypertier 0.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| hypertier-0.0.1.tar.gz | 9.0 kB | Without provenance | Details for hypertier-0.0.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| hypertier-0.0.1-py3-none-any.whl | Python 3 | none | any | Details for hypertier-0.0.1-py3-none-any.whl |
Total release size: 17.4 kB
Release files / hypertier-0.0.1.tar.gz
| Download URL | hypertier-0.0.1.tar.gz |
|---|---|
| Size | 9.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0323837d43289c48619eb8a2c60f2a83186e4968e93a4ba093b5accb12124699
|
|
BLAKE2b-256 checksum How to use checksums |
0490eb8b32ecbdbbc806c6cd1a9873ea7789956371e10cb19abc8563d0b3c70c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.5
|
Release files / hypertier-0.0.1-py3-none-any.whl
| Download URL | hypertier-0.0.1-py3-none-any.whl |
|---|---|
| Size | 8.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
32e3e170c0c8b711fadd2f6103d8b3e4b61331b332581e64f1b5723e380aca5d
|
|
BLAKE2b-256 checksum How to use checksums |
a4f2b63f043f416e7e58a8586d6138482f8381c3121b82d7f50e9d7aef857d37
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.5
|
Security for hypertier 0.0.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.