argus-cache 0.1.1

ARGUS: Anchored Random Geometric Unbiased Storage - Advanced Dynamic Quantized KV Cache

⚡ ARGUS: Anchored Random Geometric Unbiased Storage

ARGUS is an academic-grade, production-ready 7-Tier Paged Dynamic Quantized KV Cache Manager for long-context Transformers. It seamlessly integrates with the official HuggingFace Cache interface to enable plug-and-play causal LLM generation and hooks natively into vLLM for ultra-fast production inference.

Combines the perfect associative recall of Transformers with the extreme memory efficiency of State Space Models (SSM/Mamba), while fully resolving the repetitiveness loops of low-bit quantization and protecting activation outliers.

---

🌍 Language Options / Dil Seçenekleri

• English Version / İngilizce Detaylar
• Türkçe Sürüm / Türkçe Detaylar

---

🇬🇧 English Version

🎯 Architecture Overview

Transformer models suffer from quadratic memory scaling ($O(N^2)$) due to key-value (KV) cache accumulation during causal generation. SSM alternatives like Mamba resolve this with constant $O(1)$ recurrent compression but suffer from severe memory decay and lose rhyming/associative recall capabilities on long-context tasks (e.g., Passkey Retrieval).

ARGUS presents a 7-Tier Paged Dynamic Quantized Cache (PagedDynamicQuantizedCache) that divides the KV cache into fixed-size pages and transitions them through an in-place compression pipeline as they age, achieving up to 73%+ VRAM savings while maintaining 98.2%+ Raw Tensor Reconstruction Accuracy and 100% retrieval accuracy.

Sequence Direction: [Sinks (FP16)] -> [Rhyme Anchors (FP16)] -> [Active (FP16)] -> [FP8] -> [INT8] -> [INT4] -> [INT2] -> [1-Bit (Sign)] -> [Archive (JL FP16)]

🧬 The 7-Tier Memory Lifecycle

1. Tier 1: FP16 (Active Pages): Pristine precision for the most recent tokens.
2. Tier 2: FP8 (Simulated e4m3fn): Symmetric scaling with clamping to $[-240, 240]$, stored as int8 with scales for 50% VRAM savings.
3. Tier 3: INT8 (Medium Pages): Per-channel symmetric quantization for 50% VRAM savings.
4. Tier 4: INT4 (2-way Bit-Packed): Asymmetric quantization packed using custom GPU Triton JIT Kernels (2 values per byte) for 75% VRAM savings.
5. Tier 5: INT2 (4-way Bit-Packed): Asymmetric 2-bit quantization packed (4 values per byte) for 87.5% VRAM savings.
6. Tier 6: 1-Bit (Sign-Binarized Bit-Packed): Binarized signs ($x \ge 0 \rightarrow 1$, else $0$) packed 8 values per byte using custom GPU Triton JIT Kernels for 93.7% normal VRAM savings. Outliers isolated dynamically.
7. Tier 7: Johnson-Lindenstrauss Orthogonal Matrix Projection: Sequence-dimension projection ($N \rightarrow M$, where $M = N // 4$) keeping FP16 precision. Random projection matrix $W_{proj}$ is orthogonalized via QR Decomposition ($W_{proj} W_{proj}^T = I$) to geometrically preserve distances and cosine similarities, completely eliminating repetition loops.

---

🏆 Key Advanced Optimizations

1. Hardware-Aware Auto-Switching Attention (⚡ NEW ⚡)

To eliminate memory latency bottleneck during autoregressive decoding, ARGUS automatically analyzes the system hardware at runtime and switches between two highly optimized execution paths:

• Enterprise Server Mode (A100/H100/L4): Uses highly parallelized Vectorized Attention. Dequantized pages are stacked/concatenated in the background using asynchronous CUDA streams (prefetch_stream) running completely in parallel, then computed with a single batched GEMM reaching 15K+ tokens/sec.
• Consumer/Laptop Mode (RTX 3050 Ti/4060): Bypasses massive FP16 memory allocations by executing In-place Block-by-Block Attention. Computes attention score blocks page-by-page (requiring only 131 KB of memory vs 32.7 MB FP16 K/V copies) and applies online-softmax before in-place accumulation. This yields a massive 36.7x speedup on limited hardware (from 38 t/s to 1.4K t/s)!

2. Uniform Scalar Load Broadcast Triton Kernels

Dquantization kernels in triton_kernels.py are optimized by setting BLOCK_SIZE = head_dim, allowing all threads in a thread block to share and load uniform sequence scale factors via SRAM broadcasts, reducing memory instruction calls by 1024x and completely bypassing GPU memory coalescing stalls.

3. Dynamic Outlier Thresholding ($\sigma > 3.0$)

Calculates statistical variance in real-time. Key/value features exceeding $3.0\sigma$ standard deviation are dynamically isolated and stored permanently in high-fidelity FP16, while only the background normal range is compressed down. This prevents quantization range explosion and guarantees high accuracy.

---

⚙️ Installation & Quick Start

You can install ARGUS instantly from PyPI:

pip install argus-cache

1. Plug-and-Play HuggingFace Patching

Patch any HuggingFace Causal LM (e.g. Llama-3, Qwen-2, Mistral) in one line of code to use the ARGUS quantized cache manager:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from argus_cache import patch_model_with_argus

model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")

# Patch the model with ARGUS KV Cache manager
model = patch_model_with_argus(
    model,
    page_size=4096,         # Tokens per page
    max_active_pages=2,     # Keep top 2 pages in FP16
    max_fp8_pages=2,        # Transition FP16 pages to FP8 as they age
    max_int8_pages=2,
    max_int4_pages=2,
    sink_tokens=4           # Keep first 4 tokens in FP16 permanently
)

# Start generating with massive VRAM savings!
inputs = tokenizer("Muhammed Emin has created the ultimate", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128, use_cache=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

2. Run Native vLLM Docker Server

Run the production-ready vLLM server patched with ARGUS monkey-hooks on a consumer laptop (RTX 3050 Ti, 4GB VRAM) using TinyLlama:

# Build and launch in one command
docker compose up --build

This maps port 8000 to the host, running a fully OpenAI-compatible API server.

---

🇹🇷 Türkçe Sürüm

🎯 Mimari Genel Bakış

Transformers modelleri, causal üretim adımlarında biriken key-value (KV) durumları nedeniyle karesel ($O(N^2)$) bellek patlaması (Out-of-Memory - OOM) yaşarlar. Mamba gibi SSM alternatifleri bellek tüketimini recurrent bir scan döngüsüyle $O(1)$ seviyesinde sabitlese de, samanlıkta iğne arama (Passkey Retrieval) ve uzun vadeli uyak/vezin yapısı koruma gerektiren şiirsel metin üretimlerinde bellek sönümlenmesi (memory decay) yaşayarak başarısız olurlar.

ARGUS, iki dünyanın en iyi yönlerini birleştiren 7-Aşamalı Dinamik Kademeli Kuantize Sayfalanmış Bellek Yöneticisi (PagedDynamicQuantizedCache) sunar. Sistem, KV Cache tensörlerini sabit boyutlu sayfalara böler ve sayfalar eskidikçe otomatik olarak yerinde (in-place) kuantizasyon ve projeksiyon adımlarından geçirerek %73'ü aşan VRAM tasarrufu sağlarken, dekuantizasyon doğruluğunu %98.2+ seviyesinde korur.

Dizi Yönü: [Sinks (FP16)] -> [Rhyme Anchors (FP16)] -> [Active (FP16)] -> [FP8] -> [INT8] -> [INT4] -> [INT2] -> [1-Bit (Sign)] -> [Archive (JL FP16)]

🧬 7-Aşamalı Bellek Yaşam Döngüsü

1. Tier 1: FP16 (Aktif Sayfalar): En güncel token'lar için tam çözünürlüklü FP16 bellek tamponu.
2. Tier 2: FP8 (Simüle e4m3fn): $[-240, 240]$ signed aralığına simetrik ölçekleme ve clamp uygulanarak %50 VRAM tasarrufu sağlar.
3. Tier 3: INT8 (Orta Sayfalar): Per-channel simetrik kuantizasyon ile %50 VRAM tasarrufu sağlar.
4. Tier 4: INT4 (2-way Packed): Custom Triton JIT CUDA GPU Kernelleri ile iki adet 4-bitlik değerin tek bir uint8 hücresine GPU SRAM üzerinde paralel paketlenmesiyle %75 VRAM tasarrufu sağlar.
5. Tier 5: INT2 (4-way Packed): Dört adet 2-bitlik değerin tek bir uint8 hücresine bit-packing ile paketlenmesiyle %87.5 VRAM tasarrufu sağlar.
6. Tier 6: 1-Bit (İşaret Binarize Bit-Packed): İşaret değerlerini ($x \ge 0 \rightarrow 1$, else $0$) custom Triton JIT CUDA Kernelleri ile 8 adet 1-bitlik değeri tek bir uint8 hücresine paralel paketleyerek %93.7 normal VRAM tasarrufu sağlar.
7. Tier 7: Johnson-Lindenstrauss Ortogonal Matris Projeksiyonu (JL): En eski arşiv sayfalarında tekrarlama döngüsü bug'ına yol açan lossy INT2 yerine, sequence boyutu $N$ ortogonal bir rastgele matris $W_{proj}$ ile çarpılarak sequence boyutu 4 kat büzüştürülür. Sayılar yüksek çözünürlüklü FP16 biçiminde tutulur, tekrarlama döngüsü bug'ları tamamen önlenir.

---

🏆 Gelişmiş Hız Optimizasyonları

1. Donanıma Duyarlı Auto-Switching Attention (⚡ YENİ ⚡)

Autoregressive üretim adımlarındaki bellek gecikmesini tamamen ortadan kaldırmak için ARGUS, çalışma zamanında GPU gücünü otomatik olarak analiz eder ve en verimli attention yoluna geçer:

• Kurumsal Sunucu Modu (A100/H100/L4): Paralel Vectorized Attention devrededir. Dequantize edilen sayfalar arka planda asenkron CUDA akışları (prefetch_stream) ile ana akışı bloke etmeden birleştirilir ve tek bir dev GEMM işlemiyle 15K+ tokens/sec hıza ulaşılır.
• Bireysel/Mobil Modu (RTX 3050 Ti/4060): Bellek kopyalamasını $32.7\text{MB}$'tan 131 KB seviyesine düşüren yerinde sayfa-sayfa (In-place Block-by-Block Attention) hesaplama devrededir. Bu yöntem, mobil GPU'lardaki darboğazı kırarak hızı 36.7 kat artırmış ve 1.4K t/s (1400 token/sn) seviyesine çıkarmıştır!

2. Uniform Scalar Load Broadcast

Triton JIT dekuantizasyon çekirdeklerinde BLOCK_SIZE = head_dim olarak sabitlenerek, thread bloğundaki tüm iş parçacıklarının aynı sequence ölçek faktörünü paylaşması sağlanmıştır. Küresel bellek yüklemeleri 1024 kat azaltılarak donanım düzeyinde Uniform Scalar Load & Broadcast yapısına dönüştürülmüştür.

---

⚙️ Kurulum ve Hızlı Başlangıç

ARGUS kütüphanesini PyPI üzerinden tek satırla kurabilirsiniz:

pip install argus-cache

1. HuggingFace Modellerini Tek Satırda Yamalayın

Herhangi bir HuggingFace Causal dil modelini (örn. Llama-3, Qwen-2, Mistral) tek satırda ARGUS ile entegre ederek bellek tasarrufunu anında başlatın:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from argus_cache import patch_model_with_argus

model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")

# Modeli ARGUS KV Cache Yöneticisi ile yamalayın
model = patch_model_with_argus(
    model,
    page_size=4096,
    max_active_pages=2,
    max_fp8_pages=2,
    max_int8_pages=2,
    max_int4_pages=2,
    sink_tokens=4
)

# Ultra yüksek VRAM tasarrufuyla üretimi başlatın!
inputs = tokenizer("Muhammed Emin has created the ultimate", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128, use_cache=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

2. Canlı vLLM Docker Sunucusunu Başlatın

Laptop GPU'nuzda (RTX 3050 Ti, 4GB VRAM) ARGUS yamalı vLLM sunucusunu tek komutla ayağa kaldırın:

docker compose up --build

Bu komut, host üzerindeki 8000 portundan OpenAI uyumlu bir API sunucusu servis eder.

---

📂 Project Directory Structure / Proje Dizin Yapısı

├── argus_cache/            # Exposable python library package
│   ├── __init__.py         # Exposes patch_model_with_argus
│   ├── core/
│   │   ├── quantization.py # 1-bit, INT2, INT4, INT8 & JL-Projection maths
│   │   ├── memory_manager.py# 7-Tier Outlier-Aware Paged memory manager
│   │   └── triton_kernels.py# Triton JIT 1-bit and 4-bit CUDA kernels
│   └── models/
│       └── attention_wrapper.py# HuggingFace Cache wrapper with adaptive tiering
├── core/                   # Local root core files
├── models/                 # Local root models files
├── benchmarks/
│   ├── generate_vram_graph.py# Matplotlib benchmark visualizer
│   ├── vram_profiler.py    # VRAM memory scaling profiler
│   ├── llama_real_test.py  # Native HuggingFace Llama-3-8B integration test
│   └── vllm_speed_test.py  # Speed/throughput benchmark
├── tests/
│   ├── test_compression_loss.py# 1-Bit vs Lossless Delta-encoding test
│   ├── test_quantization.py# Tests for INT8, INT4 packing & Triton compiler
│   └── test_kv_cache.py    # Transitions & reconstruction errors tests
├── Dockerfile.vllm         # Production vLLM deployment container
├── docker-compose.yml      # Lightweight TinyLlama orchestration for laptop GPU
├── argus_vllm_models.py    # vLLM model registry bypass hook
├── setup.py                # Library package installer
├── pyproject.toml          # Library setup config
├── .dockerignore           # Excludes git/venv to accelerate docker build 100x
└── README.md               # Bilingual documentation

---

📄 License & Lisans

• English: This project is licensed under the Apache License 2.0. See the LICENSE file for details.
• Türkçe: Bu proje Apache Lisansı 2.0 altında lisanslanmıştır. Detaylar için LICENSE dosyasına göz atabilirsiniz.

---

🎓 Academic Citations / Akademik Atıflar

If you use this architecture or code in your thesis or research, please cite:

@thesis{MuhammedEminARGUS2026,
  author    = {Muhammed Emin Çelik},
  title     = {ARGUS: Anchored Random Geometric Unbiased Storage for Key-Value Cache in Long-Context Large Language Models},
  institution = {Academic Graduation Thesis},
  year      = {2026},
  month     = {May}
}