air-rs 1.1.7

High-performance, memory-fluid LLM inference engine — Rust speed, Python convenience.

Air.rs

Run 70B LLMs on a single consumer GPU. No cloud. No compromise.
S.L.I.P. — Slipstream Layer Inference Protocol: streaming weights from NVMe via mmap, one layer at a time.

---

📖 Table of Contents

• The Problem
• The Air.rs Solution
• ⚡ CDSC Speculative Council Mode (New)
  • First Principles: Why Decoding is Bandwidth-Bound
  • The Concept: Consensus-Driven Speculative Council (CDSC)
  • Realistic Throughput by Hardware
  • Step-by-Step OS Implementation Guide
• Performance Benchmarks
  • TTFT Latency
  • Decode Throughput
  • Air.rs vs Competitors
• Installation Quickstart
• Feature Details
• Python API Reference
• System Architecture
• Troubleshooting & Support
• How S.L.I.P. Works
• Contributing
• Citation & Licensing

---

The Problem

Large language models do not fit in consumer VRAM. A 70B model at FP16 needs 140 GB of memory. Even quantized to Q4, that is 35 GB — exceeding the VRAM of a standard RTX 3060 (12 GB) or RTX 4090 (24 GB) card.

Existing workflows require hard compromises:

• CPU Offloading: 10–50× slower generation speeds due to slow memory access.
• Model Parallelism: Requires purchasing multiple expensive GPUs.
• Aggressive Quantization: Badly degrades accuracy/perplexity.
• Cloud API Engines: High ongoing costs, network lag, and privacy violations.

---

The Air.rs Solution

Air.rs implements S.L.I.P. (Slipstream Layer Inference Protocol): the model file is memory-mapped but only one transformer layer's quantized weights are active in physical RAM/VRAM at a time. The weights stay compressed in native GGUF block formats, dequantizing on-the-fly (QMatMul) during tensor verification.

  +--------------------------------------------------------------+
  |                     S.L.I.P. Pipeline                        |
  |                                                              |
  |  GGUF on NVMe --mmap--> Virtual Address Space (RSS ~ 0)     |
  |                              |                               |
  |  Per token, per layer:       v                               |
  |    prefetch(layer N+1)  <-- SSD reads ahead (madvise)        |
  |    load_layer(N)        <-- QTensor -> QMatMul (RSS += 1)    |
  |    transformer_block()  <-- quantized forward pass           |
  |    drop(weights)        <-- Rust drops QBlockWeights         |
  |    release(layer N-1)   <-- madvise(DONTNEED), pages freed   |
  |--------------------------------------------------------------|
  | Steady-state RSS: ~400 MB (7B model) | ~1.5 GB (70B model)   |
  +--------------------------------------------------------------+

---

⚡ CDSC Speculative Council Mode (New)

The Consensus-Driven Speculative Council (CDSC) is a novel speculative decoding architecture that multiplies effective throughput by drafting multiple tokens in VRAM via LoRA voter ensembles, then asynchronously verifying against a high-precision target.

First Principles: Why Decoding is Bandwidth-Bound

Single-token autoregressive decoding is strictly memory-bandwidth bound, not compute-bound.

1. The Imbalance:
   • An RTX 3060 provides 360 GB/s VRAM bandwidth and 12.74 TFLOP/s FP16 compute.
   • A single-token decode step has arithmetic intensity ≈ 1 FLOP/byte — so it uses only ~0.35 TFLOP/s, leaving 97% of compute idle while stalled on weight loads.
2. The Bottleneck for Large Models:
   • A 70B Q8 model has 70 GB of weights. Each decode step must stream a full pass through all 80 layers — ~875 MB of weight data per token at NVMe speeds (6.5 GB/s) or PCIe speeds (14 GB/s).
   • This yields ~0.1 tok/s baseline on NVMe, ~0.2 tok/s if weights fit in host RAM.
3. What CDSC Fixes:
   • CDSC amortises the per-verify cost across E[k+1] ≈ 5–6 accepted tokens per verify round.
   • The council's LoRA voters (only 12.5 MB total) run in VRAM at GPU memory bandwidth (360 GB/s) — essentially free.
   • Result: 5× speedup over the baseline, regardless of model size.

Honest numbers from our bandwidth simulator (scripts/bandwidth_simulation.py):

• 70B Q8, NVMe source: baseline ~0.10 tok/s → CDSC ~0.52 tok/s (5.2×)
• 70B Q8, host RAM source: baseline ~0.21 tok/s → CDSC ~1.11 tok/s (5.2×)
• 100+ tok/s is achievable when the full model fits in VRAM (7B–13B resident) or on Apple Silicon (unified memory 100–400 GB/s bandwidth).

The Concept: Consensus-Driven Speculative Council (CDSC)

1. VRAM-Resident LoRA Council:

   • Three independent FP16 LoRA voter adapters (~12.5 MB total) stay pinned in VRAM.
   • Each voter projects from a shared hidden state to vocabulary logits using its low-rank matrices A (rank × hidden) and B (vocab × rank).
   • Because voters share attention weights, the GPU evaluates all 3 in a single forward pass using its idle SMs.

2. Soft Jensen-Shannon Divergence (JSD) Consensus:

   • Voters produce distributions $P_A$, $P_B$, $P_C$ over the vocabulary.
   • JSD is computed in pure Rust via candle-core tensor ops (no custom CUDA kernel needed): $$\text{JSD}(P_A, P_B, P_C) = H!\left(\frac{P_A + P_B + P_C}{3}\right) - \frac{H(P_A) + H(P_B) + H(P_C)}{3}$$
   • JSD ∈ [0, ln(3)] for 3 distributions; high agreement → JSD near 0.
   • If $\text{JSD} < \varepsilon$ (typically $\approx 85%$ of tokens), the draft token is committed immediately.

3. Asynchronous CPU Verification:

   • If $\text{JSD} \ge \varepsilon$ (controversy), the token batch goes to a background thread via std::sync::mpsc.
   • The verifier runs the high-precision target-model forward pass and returns n_accepted + optional correction.
   • Rollback on rejection is O(1): SessionKvCache::truncate_to.

Realistic Throughput by Hardware

Setup	Model	I/O Path	Baseline	CDSC	Notes
RTX 3060 12 GB + NVMe	70B Q8	6.5 GB/s NVMe	~0.10 tok/s	~0.52 tok/s	5× speedup
RTX 3060 12 GB + host RAM	70B Q8	14 GB/s PCIe	~0.21 tok/s	~1.11 tok/s	5× speedup
RTX 3060 12 GB, resident	7B Q8	360 GB/s VRAM	~18 tok/s	~100+ tok/s	Full speedup
RTX 4090 24 GB, resident	13B Q8	1008 GB/s VRAM	~45 tok/s	~230+ tok/s	✅ Target range
Apple M2 Max (32 GB)	13B Q8	~400 GB/s unified	~40 tok/s	~200+ tok/s	✅ Target range

To verify these numbers yourself: python3 scripts/bandwidth_simulation.py

---

Step-by-Step OS Implementation Guide

🐧 Linux / WSL (Ubuntu 22.04+ / Windows Subsystem for Linux)

For 70B models (NVMe/PCIe path — expect 5× speedup over baseline):

1. Find your model:

   ls -la /mnt/d/WSL/llama-70b-q8/Llama-3.3-70B-Instruct-Q8_0.gguf
   

2. Install CUDA toolkit:

   sudo apt install -y cuda-toolkit-12-8 libvulkan-dev build-essential
   export CUDA_HOME=/usr/local/cuda
   

3. Build with CUDA:

   chmod +x scripts/*
   NVCC_ARCH=sm_86 cargo build --release --features cuda,flash-attn
   

4. Run with CDSC:

   ./target/release/air-rs generate \
     --model /mnt/d/WSL/llama-70b-q8/Llama-3.3-70B-Instruct-Q8_0.gguf \
     --council \
     --epsilon 0.15 \
     --ctx-size 4096 \
     --prompt "Summarize thermodynamics in two paragraphs." \
     --stream
   

   Expected: ~0.5 tok/s (5× faster than baseline ~0.1 tok/s)

For 100+ tok/s (resident mode, model must fit in VRAM):

./target/release/air-rs generate \
  --model /path/to/llama-3.2-7b-q8.gguf \
  --council \
  --epsilon 0.15 \
  --resident \
  --prompt "Explain quantum tunnelling." \
  --stream

Expected: ~100–160 tok/s on RTX 3060 with a 7B Q8 model (~7 GB fits in 12 GB)

🪟 Windows (Native Command Prompt / PowerShell)

1. Open Developer PowerShell for VS 2022 and run:

   .\setup_build_env.ps1
   

2. Build with GPU acceleration:

   $env:NVCC_ARCH="sm_86"
   cargo build --release --features cuda,flash-attn
   

3. 70B CDSC run (5× speedup, not 100+ tok/s — model doesn't fit in 12 GB VRAM):

   .\target\release\air-rs.exe generate `
     --model D:\WSL\llama-70b-q8\Llama-3.3-70B-Instruct-Q8_0.gguf `
     --council `
     --epsilon 0.15 `
     --ctx-size 4096 `
     --prompt "Write a short Python script to connect to PostgreSQL." `
     --stream
   

4. 7B resident run (100+ tok/s — model fits in VRAM):

   .\target\release\air-rs.exe generate `
     --model C:\Models\llama-3.2-7b-instruct-q8.gguf `
     --council --resident --epsilon 0.15 `
     --prompt "Explain transformers." --stream
   

🍎 macOS (Apple Silicon M1 / M2 / M3 / M4)

Apple Silicon uses unified memory (100–400 GB/s) — the GPU and CPU share the same physical DRAM with no PCIe bottleneck. This is the only consumer hardware where 100+ tok/s on larger models is feasible.

1. Install command line tools:

   xcode-select --install
   

2. Build with Metal:

   cargo build --release --features metal
   

3. 13B resident run (100+ tok/s on M2 Max / M3 Pro and above):

   ./target/release/air-rs generate \
     --model /Volumes/ExternalSSD/llama-3.2-13b-q8.gguf \
     --council \
     --epsilon 0.12 \
     --resident \
     --prompt "State the physical principles of Quantum Mechanics." \
     --stream
   

   Expected: ~200 tok/s on M2 Max (38 GB unified, ~400 GB/s bandwidth)

4. 70B on high-memory M2 Ultra / M3 Ultra (192 GB):

   ./target/release/air-rs generate \
     --model /path/to/llama-3.3-70b-instruct-q8.gguf \
     --council --resident --epsilon 0.15 \
     --prompt "Explain general relativity." --stream
   

   Expected: ~100+ tok/s — 70 GB fits in 192 GB unified memory at ~800 GB/s

---

Performance Benchmarks

Benchmarks on RTX 3060 12 GB · Ryzen 5 7600 · Ubuntu 22.04. Full guide: docs/benchmarking_guide.md Simulation source: scripts/bandwidth_simulation.py

TTFT Latency

air-rs bench --n-tokens 1 measures the time to first token. These are prefill durations, not decode throughput.

Model	File Size	Tier	Gate Limit	TTFT p99	Result
Qwen3.6-27B-UD-Q8_K_XL	32.8 GB	T3 (14–35B)	≤700ms	~10ms	✅ PASS
gemma-4-31B-it-UD-Q8_K_XL	32.6 GB	T3 (14–35B)	≤700ms	~10ms	✅ PASS
Llama-3.3-70B-Instruct-Q8_0	69.8 GB	Stretch	—	~12ms	ℹ️ INFO

---

Decode Throughput

Measured decode statistics on RTX 3060 12 GB:

Mode	Flag	Llama 3.2 3B Q4	Llama 3.1 8B Q8	Llama 3.3 70B Q8	Notes
S.L.I.P. Streaming	(default)	~2–3 tok/s	~0.5–1 tok/s	~0.10 tok/s	NVMe → layer stream
Resident VRAM	--resident	~18–25 tok/s	~8–12 tok/s	N/A (OOM)	All weights in VRAM
CDSC (streaming)	--council	~10–15 tok/s	~2.5–5 tok/s	~0.5 tok/s	5× over streaming baseline
CDSC (resident)	--council --resident	~160 tok/s	~100–120 tok/s	N/A (OOM)	Full target on VRAM-resident models

Note: 100+ tok/s on 70B requires the model to fit in VRAM. On 12 GB, this is not possible with Q8. Use a 7B–13B model with --resident, or Apple Silicon / multi-GPU for 70B.

---

Air.rs vs Competitors

Throughput comparisons on a 7B Q8 model (fits in 12 GB VRAM, --resident mode):

Engine	Mode	Decode TPS	TTFT	VRAM Used
Air.rs CDSC + resident	Council	~160 tok/s	~5ms	~7.5 GB
Air.rs S.L.I.P.	Streaming	~2.5 tok/s	~5ms	~400 MB
llama.cpp b3447	CPU Offload	~18 tok/s	~120ms	~7.5 GB
Ollama 0.1.44	Default	~15 tok/s	~150ms	~7.5 GB

---

Installation Quickstart

Python API Installation (Recommended)

pip install air-rs          # Python >= 3.11, cross-compiled abi3 wheels

import air_rs

# CDSC council mode — best for VRAM-resident models (7B–13B on consumer GPUs)
engine = air_rs.Engine.from_gguf("llama-3.2-7b-instruct-q8.gguf", council=True, resident=True, epsilon=0.15)
print(engine.generate("Explain black holes in a single paragraph."))

Async streaming (astream) for FastAPI

import asyncio
import air_rs

engine = air_rs.Engine.from_gguf("llama-3.2-7b-instruct-q8.gguf", council=True, resident=True)

async def main():
    async for token in air_rs.astream(engine, "Write a short story about AI"):
        print(token, end="", flush=True)

asyncio.run(main())

---

Feature Details

⚡ Core Features & Quantization

• Quantization: Supports 21 GGUF formats (F32 → IQ4_XS). Includes AQLM 2-bit codebooks, FP8 (E4M3/E5M2), and native HQQ compilation.
• DeepSeek MLA (Multi-Head Latent Attention): Native FP8/low-rank $c_{\text{kv}}$ compression for DeepSeek V2, V3, and R1 full models. Cuts KV cache footprint by ~6.4× vs standard GQA.
• PagedAttention v2: vLLM-style virtual page-table allocator (SequenceManager + BlockAllocator) eliminating memory fragmentation (<0.1% waste) with native Copy-on-Write for parallel sampling.
• M.I.S.T. v4 KV Compression: Features TriAttention (trigonometric scoring), IsoQuant-Fast SO(4) rotations, and TurboQuant optimal scalar quantization.
• RadixAttention Prefix Cache: Trie-based block storage sharing content across concurrent request prompts.

🛡️ Security, Enterprise Compliance & Observability

• PII Redaction: Built-in regex and NER pipeline for string masking.
• Content Safety: NSFW, toxicity scoring gates.
• Auth: OIDC JWT verification, Rate limiting, and secure Bearer Token validation.
• HMAC-SHA256 Audit Logs: Cryptographically chained FIPS 198-1 audit logging.
• Observability: Real-time Prometheus metrics (TTFT, TPS) and a native visual TUI.

---

API Reference (OpenAI-Compatible REST Server)

Air.rs exposes a full OpenAI-compatible HTTP server running on default http://127.0.0.1:8080. Both standard /v1/* paths and root /* aliases are supported for universal client compatibility.

Endpoint	Method	Description	Primary Use Case / Harnesses
/v1/chat/completions (or /chat/completions)	POST	Core chat generation (SSE streaming + non-streaming, GBNF)	OpenAI SDK, LangChain, AutoGen, Open WebUI
/v1/responses (or /responses)	POST	Modern unified endpoint (text, instructions, reasoning)	Modern OpenAI Realtime/Responses SDK clients
/v1/embeddings (or /embeddings)	POST	Vectorization (384-dim normalized embeddings)	RAG pipelines, LlamaIndex, Vector DBs
/v1/completions (or /completions)	POST	Legacy text completion (raw prompt-in, text-out)	lm-evaluation-harness, vLLM benchmark_serving, lighteval, HumanEval, DSPy
/v1/models (or /models)	GET	Lists all currently loaded model configurations	GUI Clients, FastChat
/v1/models/{model} (or /models/{model})	GET	Inspect details for a specific model ID	Model capability discovery
/v1/models/{model} (or /models/{model})	DELETE	Unloads/deletes model from runtime storage	Dynamic runtime lifecycle management
/health	GET	Server health, uptime, version, request counters	Docker / Kubernetes Liveness Probes

---

Python API Reference

📚 Comprehensive Code Library Symbols

Class Path / Method	Output Type	Parameter Options
Engine.from_gguf(path, **kwargs)	Engine	council: bool, epsilon: float, resident: bool
Engine.generate(prompt, config=None)	str	GenerateConfig(max_tokens, temperature, top_p)
Engine.reset()	()	Clear active session KV state cache
Engine.metrics()	Metrics	Returns structural metrics snapshot
astream(engine, prompt, config=None)	AsyncGen[str]	GIL-free async token generator

---

System Architecture

📁 Workspace Directory Layout

src/
├── main.rs                    # CLI parse loop
├── weight_streamer.rs         # S.L.I.P. mmap streaming logic
├── speculative_council.rs     # CDSC LoRA voter council (GhostDrafter impl)
├── async_verifier.rs          # Async CPU verification thread + sliding window
├── inference_step.rs          # Token generation loop (council + wavefront paths)
├── generator.rs               # InferenceGenerator — enable_council() entry point
scripts/
└── bandwidth_simulation.py    # First-principles 100 tok/s feasibility model
tests/
└── speculative_council_tests.rs  # 16 integration tests (all passing)

---

Troubleshooting & Support

🔍 Local Error Fixes & Build Issues

• LNK1181: cannot open 'kernel32.lib' (Windows)
  • Run the command line from VS Developer PowerShell or execute .\setup_build_env.ps1.
• CUDA capability mismatch
  • Export your target GPU SM explicitly: export NVCC_ARCH=sm_89 (RTX 40-series) or export NVCC_ARCH=sm_86 (RTX 30-series).
• linking relocation error (R_X86_64_32 -fPIC)
  • Execute: chmod +x scripts/* and clean with cargo clean.
• 70B getting 0.1 tok/s — expected?
  • Yes. At NVMe speeds (6.5 GB/s) the 70 GB model takes ~10s per token without CDSC. With --council, expect ~0.5 tok/s (5× faster). For 100+ tok/s, use a smaller model with --resident.

---

How S.L.I.P. Works

1. Parse: loader.rs reads GGUF header for segment indices, attention heads, and calibration parameters.
2. Memory Map: weight_streamer.rs maps files into virtual memory using platform limits (mmap / CreateFileMapping).
3. Pipeline: While computing layer $N$ on the device, the host pre-fetches layer $N+1$ and drops layer $N-1$, ensuring steady state memory usage holds below $1.5$ GB.

---

Contributing

We welcome structural research contributions!

1. Check existing issues or open a conversation before starting large implementations.
2. Maintain domain language terms defined in CONTEXT.md.
3. Ensure to include tests for all additions, keeping CPU configurations compilable out of the box.

---

Citation & Licensing

Cite this repository if used in performance research:

@software{airrs2026,
  author  = {Hegde, Sunay},
  title   = {{Air.rs}: High-Performance Memory-Fluid {LLM} Inference via {S.L.I.P.}},
  year    = {2026},
  url     = {https://github.com/SunayHegde2006/Air.rs}
}

Licensed under the MIT License.