GrillyInference

Native fp16 inference engine for Llama-family models — optional grilly extension.

Features

• Native fp16 inference — runs Llama 3.2 3B at ~6.4 GB VRAM, zero quality loss
• Paged KV-Cache — 256-token SRAM pages with LRU eviction, 4x context extension
• H2O Eviction — exponential decay on old KV heads, 32k context on 12GB
• VSA Multi-Scale Summaries — hypervector bind/bundle for 128k effective context
• SmoothQuant INT8 — per-group-64 weight quantization, <1% PPL loss
• 4-bit Block Quantization — run 100B models on 12GB VRAM with layer offloading
• Llama 3.2 Instruct — chat template, streaming generation, top-k/top-p sampling

Quick Start

pip install grillyinference

from grillyinference import LlamaForCausalLM, TextGenerator
from transformers import AutoTokenizer

model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
gen = TextGenerator(model, tokenizer)

# Simple generation
response = gen.generate("What is the meaning of life?", max_tokens=256)
print(response)

# Chat
response = gen.chat([
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain transformers in 3 sentences."},
])
print(response)

# Streaming
for token in gen.generate("Once upon a time", stream=True):
    print(token, end="", flush=True)

Context Extension (12GB VRAM)

from grillyinference import KVCache, LlamaConfig

config = LlamaConfig.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
kv = KVCache(config, raw_window=2048, h2o_lambda=0.0002, enable_vsa=True)

SmoothQuant INT8

from grillyinference.inference.quantize import SmoothQuantCalibrator, SmoothQuantizer

calibrator = SmoothQuantCalibrator(model, tokenizer)
stats = calibrator.calibrate()
quantizer = SmoothQuantizer(group_size=64)
quantized = quantizer.smooth_and_quantize(model._weights, stats)

Requirements

• Python 3.12+
• grilly >= 0.4.0
• numpy, safetensors
• Optional: huggingface_hub, transformers (for from_pretrained)

License

MIT