Air.rs Banner

Air.rs

Run 70B LLMs on a single consumer GPU. No cloud. No compromise.
S.L.I.P. — Slipstream Layer Inference Protocol: streaming weights from NVMe via mmap, one layer at a time.

📖 Table of Contents

The Problem
The Air.rs Solution
⚡ CDSC Speculative Council Mode
Performance Benchmarks
Installation Quickstart
- Python API Installation
- Async Streaming (astream for FastAPI)
✨ v1.3.0 New Features
💻 OS-Specific Usage Guides for v1.3.0 Features
Feature Details
- Core Features & Quantization
- Security, Enterprise & Observability
API Reference (OpenAI-Compatible REST Server)
Cargo Feature Flags Reference
CLI Reference
Python API Reference
System Architecture
Troubleshooting & Support
How S.L.I.P. Works
Contributing
Changelog
Citation & Licensing

The Problem

Large language models do not fit in consumer VRAM. A 70B model at FP16 needs 140 GB of memory. Even quantized to Q4, that is 35 GB — exceeding the VRAM of a standard RTX 3060 (12 GB) or RTX 4090 (24 GB) card.

Existing workflows require hard compromises:

CPU Offloading: 10–50× slower generation speeds due to slow memory access.
Model Parallelism: Requires purchasing multiple expensive GPUs.
Aggressive Quantization: Badly degrades accuracy/perplexity.
Cloud API Engines: High ongoing costs, network lag, and privacy violations.

The Air.rs Solution

Air.rs implements S.L.I.P. (Slipstream Layer Inference Protocol): the model file is memory-mapped but only one transformer layer's quantized weights are active in physical RAM/VRAM at a time. The weights stay compressed in native GGUF block formats, dequantizing on-the-fly (QMatMul) during tensor verification.

  +--------------------------------------------------------------+
  |                     S.L.I.P. Pipeline                        |
  |                                                              |
  |  GGUF on NVMe --mmap--> Virtual Address Space (RSS ~ 0)     |
  |                              |                               |
  |  Per token, per layer:       v                               |
  |    prefetch(layer N+1)  <-- SSD reads ahead (madvise)        |
  |    load_layer(N)        <-- QTensor -> QMatMul (RSS += 1)    |
  |    transformer_block()  <-- quantized forward pass           |
  |    drop(weights)        <-- Rust drops QBlockWeights         |
  |    release(layer N-1)   <-- madvise(DONTNEED), pages freed   |
  |--------------------------------------------------------------|
  | Steady-state RSS: ~400 MB (7B model) | ~1.5 GB (70B model)   |
  +--------------------------------------------------------------+

⚡ CDSC Speculative Council Mode (New)

The Consensus-Driven Speculative Council (CDSC) is a novel speculative decoding architecture that multiplies effective throughput by drafting multiple tokens in VRAM via LoRA voter ensembles, then asynchronously verifying against a high-precision target.

First Principles: Why Decoding is Bandwidth-Bound

Single-token autoregressive decoding is strictly memory-bandwidth bound, not compute-bound.

The Imbalance:
- An RTX 3060 provides 360 GB/s VRAM bandwidth and 12.74 TFLOP/s FP16 compute.
- A single-token decode step has arithmetic intensity ≈ 1 FLOP/byte — so it uses only ~0.35 TFLOP/s, leaving 97% of compute idle while stalled on weight loads.
The Bottleneck for Large Models:
- A 70B Q8 model has 70 GB of weights. Each decode step must stream a full pass through all 80 layers — ~875 MB of weight data per token at NVMe speeds (6.5 GB/s) or PCIe speeds (14 GB/s).
- This yields ~0.1 tok/s baseline on NVMe, ~0.2 tok/s if weights fit in host RAM.
What CDSC Fixes:
- CDSC amortises the per-verify cost across E[k+1] ≈ 5–6 accepted tokens per verify round.
- The council's LoRA voters (only 12.5 MB total) run in VRAM at GPU memory bandwidth (360 GB/s) — essentially free.
- Result: 5× speedup over the baseline, regardless of model size.

Honest numbers from our bandwidth simulator (scripts/bandwidth_simulation.py):

70B Q8, NVMe source: baseline ~0.10 tok/s → CDSC ~0.52 tok/s (5.2×)

70B Q8, host RAM source: baseline ~0.21 tok/s → CDSC ~1.11 tok/s (5.2×)

100+ tok/s is achievable when the full model fits in VRAM (7B–13B resident) or on Apple Silicon (unified memory 100–400 GB/s bandwidth).

The Concept: Consensus-Driven Speculative Council (CDSC)

VRAM-Resident LoRA Council:
- Three independent FP16 LoRA voter adapters (~12.5 MB total) stay pinned in VRAM.
- Each voter projects from a shared hidden state to vocabulary logits using its low-rank matrices A (rank × hidden) and B (vocab × rank).
- Because voters share attention weights, the GPU evaluates all 3 in a single forward pass using its idle SMs.
Soft Jensen-Shannon Divergence (JSD) Consensus:
- Voters produce distributions $P_A$, $P_B$, $P_C$ over the vocabulary.
- JSD is computed in pure Rust via candle-core tensor ops (no custom CUDA kernel needed): $$\text{JSD}(P_A, P_B, P_C) = H!\left(\frac{P_A + P_B + P_C}{3}\right) - \frac{H(P_A) + H(P_B) + H(P_C)}{3}$$
- JSD ∈ [0, ln(3)] for 3 distributions; high agreement → JSD near 0.
- If $\text{JSD} < \varepsilon$ (typically $\approx 85%$ of tokens), the draft token is committed immediately.
Asynchronous CPU Verification:
- If $\text{JSD} \ge \varepsilon$ (controversy), the token batch goes to a background thread via std::sync::mpsc.
- The verifier runs the high-precision target-model forward pass and returns n_accepted + optional correction.
- Rollback on rejection is O(1): SessionKvCache::truncate_to.

Realistic Throughput by Hardware

Setup	Model	I/O Path	Baseline	CDSC	Notes
RTX 3060 12 GB + NVMe	70B Q8	6.5 GB/s NVMe	~0.10 tok/s	~0.52 tok/s	5× speedup
RTX 3060 12 GB + host RAM	70B Q8	14 GB/s PCIe	~0.21 tok/s	~1.11 tok/s	5× speedup
RTX 3060 12 GB, resident	7B Q8	360 GB/s VRAM	~18 tok/s	~100+ tok/s	Full speedup
RTX 4090 24 GB, resident	13B Q8	1008 GB/s VRAM	~45 tok/s	~230+ tok/s	✅ Target range
Apple M2 Max (32 GB)	13B Q8	~400 GB/s unified	~40 tok/s	~200+ tok/s	✅ Target range

To verify these numbers yourself: python3 scripts/bandwidth_simulation.py

Step-by-Step OS Implementation Guide

🐧 Linux / WSL (Ubuntu 22.04+ / Windows Subsystem for Linux)

For 70B models (NVMe/PCIe path — expect 5× speedup over baseline):

Find your model:

ls -la /mnt/d/WSL/llama-70b-q8/Llama-3.3-70B-Instruct-Q8_0.gguf

Install CUDA toolkit:

sudo apt install -y cuda-toolkit-12-8 libvulkan-dev build-essential
export CUDA_HOME=/usr/local/cuda

Build with CUDA:

chmod +x scripts/*
NVCC_ARCH=sm_86 cargo build --release --features cuda,flash-attn

Run with CDSC:

./target/release/air-rs generate \
  --model /mnt/d/WSL/llama-70b-q8/Llama-3.3-70B-Instruct-Q8_0.gguf \
  --council \
  --epsilon 0.15 \
  --ctx-size 4096 \
  --prompt "Summarize thermodynamics in two paragraphs." \
  --stream

Expected: ~0.5 tok/s (5× faster than baseline ~0.1 tok/s)

For 100+ tok/s (resident mode, model must fit in VRAM):

./target/release/air-rs generate \
  --model /path/to/llama-3.2-7b-q8.gguf \
  --council \
  --epsilon 0.15 \
  --resident \
  --prompt "Explain quantum tunnelling." \
  --stream

Expected: ~100–160 tok/s on RTX 3060 with a 7B Q8 model (~7 GB fits in 12 GB)

🪟 Windows (Native Command Prompt / PowerShell)

Open Developer PowerShell for VS 2022 and run:
```
.\setup_build_env.ps1
```

Build with GPU acceleration:

$env:NVCC_ARCH="sm_86"
cargo build --release --features cuda,flash-attn

70B CDSC run (5× speedup, not 100+ tok/s — model doesn't fit in 12 GB VRAM):

.\target\release\air-rs.exe generate `
  --model D:\WSL\llama-70b-q8\Llama-3.3-70B-Instruct-Q8_0.gguf `
  --council `
  --epsilon 0.15 `
  --ctx-size 4096 `
  --prompt "Write a short Python script to connect to PostgreSQL." `
  --stream

7B resident run (100+ tok/s — model fits in VRAM):

.\target\release\air-rs.exe generate `
  --model C:\Models\llama-3.2-7b-instruct-q8.gguf `
  --council --resident --epsilon 0.15 `
  --prompt "Explain transformers." --stream

🍎 macOS (Apple Silicon M1 / M2 / M3 / M4)

Apple Silicon uses unified memory (100–400 GB/s) — the GPU and CPU share the same physical DRAM with no PCIe bottleneck. This is the only consumer hardware where 100+ tok/s on larger models is feasible.

Install command line tools:
```
xcode-select --install
```
Build with Metal:
```
cargo build --release --features metal
```

13B resident run (100+ tok/s on M2 Max / M3 Pro and above):

./target/release/air-rs generate \
  --model /Volumes/ExternalSSD/llama-3.2-13b-q8.gguf \
  --council \
  --epsilon 0.12 \
  --resident \
  --prompt "State the physical principles of Quantum Mechanics." \
  --stream

Expected: ~200 tok/s on M2 Max (38 GB unified, ~400 GB/s bandwidth)

70B on high-memory M2 Ultra / M3 Ultra (192 GB):

./target/release/air-rs generate \
  --model /path/to/llama-3.3-70b-instruct-q8.gguf \
  --council --resident --epsilon 0.15 \
  --prompt "Explain general relativity." --stream

Expected: ~100+ tok/s — 70 GB fits in 192 GB unified memory at ~800 GB/s

Performance Benchmarks

Benchmarks on RTX 3060 12 GB · Ryzen 5 7600 · Ubuntu 22.04. Full guide: docs/benchmarking_guide.md Simulation source: scripts/bandwidth_simulation.py

TTFT Latency

air-rs bench --n-tokens 1 measures an Air.rs-internal proxy for time-to-first-token: a single token decode via S.L.I.P. These are not comparable to VRAM-resident engines (llama.cpp, Ollama, vLLM) which pay TTFT once at startup. Reproduce with ./scripts/tiered_ttft.sh --models-dir=/your/models.

Model	File Size	Tier	Internal Gate	Air.rs TTFT proxy	Repro
Qwen3.6-27B-UD-Q8_K_XL	32.8 GB	T3 (14–35B)	≤700ms	~10ms	`tiered_ttft.sh`
gemma-4-31B-it-UD-Q8_K_XL	32.6 GB	T3 (14–35B)	≤700ms	~10ms	`tiered_ttft.sh`
Llama-3.3-70B-Instruct-Q8_0	69.8 GB	Stretch	—	~12ms	`tiered_ttft.sh`

⚠️ Methodology note: Air.rs TTFT is measured after the model is already mmap'd and running. A fair comparison to llama.cpp/Ollama TTFT requires both engines to start from a cold load on the same hardware under the same VRAM constraint — we haven't published that comparison yet. PRs with reproducible cross-engine results are welcome.

Decode Throughput

Measured decode statistics on RTX 3060 12 GB:

Mode	Flag	Llama 3.2 3B Q4	Llama 3.1 8B Q8	Llama 3.3 70B Q8	Notes
S.L.I.P. Streaming	(default)	~2–3 tok/s	~0.5–1 tok/s	~0.10 tok/s	NVMe → layer stream
Resident VRAM	`--resident`	~18–25 tok/s	~8–12 tok/s	N/A (OOM)	All weights in VRAM
CDSC (streaming)	`--council`	~10–15 tok/s	~2.5–5 tok/s	~0.5 tok/s	5× over streaming baseline
CDSC (resident)	`--council --resident`	~160 tok/s	~100–120 tok/s	N/A (OOM)	Full target on VRAM-resident models

Note: 100+ tok/s on 70B requires the model to fit in VRAM. On 12 GB, this is not possible with Q8. Use a 7B–13B model with --resident, or Apple Silicon / multi-GPU for 70B.

Air.rs vs Competitors

We have not yet published a verified head-to-head benchmark. The table below was removed pending a reproducible cross-engine test on identical hardware and load conditions. If you run one, please open a PR — we will include it with full methodology.

What you can verify today: Air.rs decode throughput on a 7B Q8 model, 12 GB VRAM, --resident mode (RTX 3060 · Ryzen 5 7600 · Ubuntu 22.04):

Air.rs Mode	Flag	Decode TPS	VRAM Used	Notes
S.L.I.P. streaming	(default)	~2.5 tok/s	~400 MB	Model larger than VRAM
CDSC resident	`--council --resident`	~160 tok/s	~7.5 GB	Model fits in VRAM

Reproduce: cargo build --release && ./target/release/air-rs bench --model your.gguf --runs 10

Installation Quickstart

Python API Installation (Recommended)

pip install air-rs          # Python >= 3.11, cross-compiled abi3 wheels

import air_rs

# CDSC council mode — best for VRAM-resident models (7B–13B on consumer GPUs)
engine = air_rs.Engine.from_gguf("llama-3.2-7b-instruct-q8.gguf", council=True, resident=True, epsilon=0.15)
print(engine.generate("Explain black holes in a single paragraph."))

Async streaming (`astream`) for FastAPI

import asyncio
import air_rs

engine = air_rs.Engine.from_gguf("llama-3.2-7b-instruct-q8.gguf", council=True, resident=True)

async def main():
    async for token in air_rs.astream(engine, "Write a short story about AI"):
        print(token, end="", flush=True)

asyncio.run(main())

✨ v1.3.0 New Features

Universal Ternary Model & Self-Speculative CDSC Engine

Air.rs v1.3.0 introduces native support for 1.58-bit Universal Ternary Weight Format (UTWF) models (e.g. BitNet b1.58) and self-speculative CDSC execution:

Ternary Weight Packing: 5 trits per byte ({-1, 0, +1}) stored at ~1.6 bits per weight.
SIMD Integer Addition: Multiplications are replaced by high-speed addition/subtraction index loops across AVX-512, AVX2, and ARM NEON.
Self-Speculative CDSC: Dynamic tree depth self-speculation targeting up to ~5.2× throughput.

# Run inference on a ternary GGUF model
air-rs generate --model ./bitnet-3b-b1.58.gguf --ternary --prompt "Explain quantum computing."

# Enable self-speculative CDSC on ternary weights
air-rs generate --model ./bitnet-3b-b1.58.gguf --ternary --ternary-cdsc --prompt "Write a Python web scraper."

W8A8 INT8 Matmul & AVX-512 VNNI Acceleration

Hardware-accelerated quantization path fusing 8-bit weights and 8-bit activations:

SIMD VNNI Acceleration: Uses vpdpbusd dot-product instructions on Intel/AMD CPUs with AVX-512 VNNI.
PTX GPU Acceleration: Fused 8-bit matrix multiplication with block-tiled accumulation on NVIDIA GPUs.

# Build with AVX-512 / CUDA acceleration
cargo build --release --features cuda,flash-attn

HBM-Aware Tiered KV-Cache Offload (--kv-swap)

Dynamic memory manager (KvSwapManager) preventing VRAM/HBM out-of-memory errors by swapping inactive KV-cache blocks between GPU VRAM and host RAM:

Block Granularity: Configurable chunk size in KB via --kv-swap-block-kb (default: 64 KB).
Tiered Management: Keeps active sequence blocks in HBM while paging idle blocks to host RAM.

# Enable KV swapping with 64KB block granularity
air-rs generate --model llama-70b.gguf --kv-swap --kv-swap-block-kb 64 --prompt "Summarize RFC 793."

# Server mode with KV swapping for concurrent sessions
air-rs serve --model llama-70b.gguf --kv-swap --kv-swap-block-kb 64 --port 8080

Per-Batch CUDA Graph Registry

PerBatchCudaGraphRegistry provides dynamic per-batch-size CUDA execution graph caching:

Microsecond Launch Overhead: Replaces ~100µs CPU-side driver kernel submissions with a single ~1µs CUDA graph launch.
Dynamic Batching: Automatically instantiates and manages CUDA execution graphs for batch sizes $B \in {1, 2, 4, 8, 16, \dots}$.

# CUDA graph registry is active automatically on CUDA builds
cargo build --release --features cuda,flash-attn
air-rs generate --model llama-3.2-3b.gguf --council --resident

Draft Head Checkpoint Weight Loading & Flash Tree-Attention

Speculative Council (SpeculativeCouncilDrafter) now supports pre-trained SVD draft head weights and block-sparse GPU tree attention:

Checkpoint Loading: Loads trained projection matrices $A \in \mathbb{R}^{r \times d_{in}}$ and $B \in \mathbb{R}^{d_{out} \times r}$ from model checkpoints with automatic fallback to rank-16 SVD initialization.
Flash Tree Attention: Dispatches parent-mask tree attention to block-sparse CUDA kernels for draft trees ($k > 16$) with CPU reference fallback for smaller trees.

# Run speculative council with draft head checkpoint loading
air-rs generate --model llama-70b-q8.gguf --council --epsilon 0.15 --stream

AFIT Native Future Desugaring (Zero async-trait Overhead)

Distributed inference communicator interface (Communicator in src/distributed.rs) now uses native Rust 1.75 desugared futures:

Zero Macro Dependencies: Eliminates async-trait dependency completely.
Object Safety: Explicit Pin<Box<dyn Future<Output = Result<T, E>> + Send + '_>> signature maintains full dynamic object safety across multi-GPU / TCP / NCCL backends.

Zero-Config Run (air-rs --run)

One command replaces the previous 3-step pull → configure → serve flow:

# By local GGUF path
air-rs --run ./llama-3.2-3b.Q4_K_M.gguf

# From registry alias (after pull)
air-rs --run llama-3.2-3b

# Raw HuggingFace target
air-rs --run TheBloke/Llama-2-7B-GGUF/llama-2-7b.Q4_K_M.gguf

Resolves: local path → registry alias → HuggingFace download automatically.

Interactive TUI REPL (--interactive)

Run a full multi-turn terminal chat session with streaming output, throughput stats, and slash commands:

air-rs --run llama.gguf --interactive

Slash commands in REPL:

Command	Description
`/help`	Show all commands
`/clear`	Clear conversation history
`/stats`	Show generation statistics
`/temp 0.5`	Set temperature
`/exit`	Exit the REPL

Concurrent OpenAI REST Server (--serve)

Spawn a full OpenAI-compatible server in the background while the REPL or single-inference mode runs in the foreground:

# Start background server + REPL
air-rs --run llama.gguf --serve --interactive

# Background server only, single inference
air-rs --run llama.gguf --serve --port 9000

# Default port is 8080
air-rs serve --model llama.gguf --port 8080

Endpoints available: POST /v1/chat/completions, POST /v1/completions, GET /v1/models, GET /health.

TLS / HTTPS (--tls-cert / --tls-key)

Pass PEM certificate and key to serve endpoints over HTTPS via Rustls (no OpenSSL required):

air-rs serve --model llama.gguf \
  --tls-cert /etc/ssl/certs/server.pem \
  --tls-key  /etc/ssl/private/server.key \
  --port 443 --host 0.0.0.0

Both --tls-cert and --tls-key must be specified together (validation error if one is missing).
Without flags: plain HTTP as before.
Certificate format: standard PEM (use openssl req -x509 -newkey rsa:4096 -nodes -keyout server.key -out server.pem -days 365 for self-signed).

Python air_rs.load()

Zero-config high-level Python helper — no need to call Engine.from_gguf() directly:

import air_rs

# From path or alias
engine = air_rs.load("path/to/model.gguf")
print(engine.generate("Explain attention in one sentence."))

# Streaming
async for token in air_rs.astream(engine, "Once upon a time"):
    print(token, end="", flush=True)

Metal Compute Kernels (Apple Silicon)

Production Metal Shading Language (MSL) kernels for Apple Silicon M1/M2/M3/M4:

Kernel	Description
`deltanet_recurrence`	DeltaNet state update `H_t = α·H_{t-1} + β·(k·v^T)` + query projection
`rms_norm`	Fast GPU RMSNorm with `rsqrt` instruction
`silu_mul`	SwiGLU fused activation (`g / (1 + expf(-g)) × up`)

Enable with:

cargo build --release --features metal

💻 OS-Specific Usage Guides for v1.3.0 Features

🐧 Linux / WSL (Ubuntu 22.04+ / Windows Subsystem for Linux)

1. Build Air.rs v1.3.0 with CUDA & FlashAttention

# Install system dependencies
sudo apt update && sudo apt install -y build-essential libvulkan-dev

# Build with CUDA + FlashAttention-2
export CUDA_HOME=/usr/local/cuda
export NVCC_ARCH=sm_86 # Set to sm_89 for RTX 40-series, sm_80 for A100, sm_90 for H100
cargo build --release --features cuda,flash-attn

2. Run Universal Ternary Model with Self-Speculative CDSC

./target/release/air-rs generate \
  --model ./models/bitnet-3b-b1.58.gguf \
  --ternary \
  --ternary-cdsc \
  --prompt "Explain the architecture of transformer models." \
  --stream

3. Server Mode with HBM-Aware KV-Cache Offloading & TLS

./target/release/air-rs serve \
  --model /path/to/Llama-3.3-70B-Instruct-Q8_0.gguf \
  --kv-swap \
  --kv-swap-block-kb 64 \
  --tls-cert /etc/ssl/certs/server.pem \
  --tls-key /etc/ssl/private/server.key \
  --port 8443 \
  --host 0.0.0.0

4. Interactive TUI REPL with Multi-Batch CUDA Graph Registry

./target/release/air-rs --run ./models/llama-3.2-3b-instruct.gguf --interactive --council --resident

🪟 Windows (PowerShell / Command Prompt)

1. Setup Build Environment & Compile v1.3.0

# Execute build environment setup script in Developer PowerShell for VS 2022
.\setup_build_env.ps1

# Set target GPU architecture (sm_86 for RTX 30-series, sm_89 for RTX 40-series)
$env:NVCC_ARCH="sm_86"
cargo build --release --features cuda,flash-attn

2. Run Universal Ternary Model with Self-Speculative CDSC

.\target\release\air-rs.exe generate `
  --model C:\Models\bitnet-3b-b1.58.gguf `
  --ternary `
  --ternary-cdsc `
  --prompt "Write a PowerShell script to audit disk usage." `
  --stream

3. Server Mode with HBM-Aware KV-Cache Offloading

.\target\release\air-rs.exe serve `
  --model D:\Models\Llama-3.3-70B-Instruct-Q8_0.gguf `
  --kv-swap `
  --kv-swap-block-kb 64 `
  --port 8080 `
  --host 127.0.0.1

4. Zero-Config Run with Interactive REPL & Speculative Council

.\target\release\air-rs.exe --run C:\Models\llama-3.2-3b.gguf --interactive --council --resident

🍎 macOS (Apple Silicon M1 / M2 / M3 / M4)

1. Build Air.rs v1.3.0 with Apple Metal Compute

# Ensure Xcode command line tools are installed
xcode-select --install

# Build release binary with Apple Metal GPU acceleration
cargo build --release --features metal

2. Run Universal Ternary Model with Self-Speculative CDSC

./target/release/air-rs generate \
  --model ~/Models/bitnet-3b-b1.58.gguf \
  --ternary \
  --ternary-cdsc \
  --prompt "Draft an email outlining a project timeline." \
  --stream

3. High-Throughput 13B/70B Resident Speculative Council Run

# 13B Q8 model on M2 Max / M3 Pro (unified memory 200+ tok/s)
./target/release/air-rs generate \
  --model ~/Models/llama-3.2-13b-q8.gguf \
  --council \
  --resident \
  --epsilon 0.15 \
  --prompt "Explain principles of general relativity." \
  --stream

4. Background REST Server with Interactive TUI REPL

./target/release/air-rs --run ~/Models/llama-3.2-3b.gguf --serve --interactive --port 8080

Feature Details

⚡ Core Features & Quantization

Quantization: Supports 21 GGUF formats (F32 → IQ4_XS). Includes AQLM 2-bit codebooks, FP8 (E4M3/E5M2), and native HQQ compilation.
DeepSeek MLA (Multi-Head Latent Attention): Native FP8/low-rank $c_{\text{kv}}$ compression for DeepSeek V2, V3, and R1 full models. Cuts KV cache footprint by ~6.4× vs standard GQA.
PagedAttention v2: vLLM-style virtual page-table allocator (SequenceManager + BlockAllocator) eliminating memory fragmentation (<0.1% waste) with native Copy-on-Write for parallel sampling.
M.I.S.T. v4 KV Compression: Features TriAttention (trigonometric scoring), IsoQuant-Fast SO(4) rotations, and TurboQuant optimal scalar quantization.
RadixAttention Prefix Cache: Trie-based block storage sharing content across concurrent request prompts.

🛡️ Security, Enterprise Compliance & Observability

PII Redaction: Built-in regex and NER pipeline for string masking.
Content Safety: NSFW, toxicity scoring gates.
Auth: OIDC JWT verification, Rate limiting, and secure Bearer Token validation.
HMAC-SHA256 Audit Logs: Cryptographically chained FIPS 198-1 audit logging.
Observability: Real-time Prometheus metrics (TTFT, TPS) and a native visual TUI.

API Reference (OpenAI-Compatible REST Server)

Air.rs exposes a full OpenAI-compatible HTTP server running on default http://127.0.0.1:8080. Both standard /v1/* paths and root /* aliases are supported for universal client compatibility.

Endpoint	Method	Description	Primary Use Case / Harnesses
`/v1/chat/completions` (or `/chat/completions`)	`POST`	Core chat generation (SSE streaming + non-streaming, GBNF)	OpenAI SDK, LangChain, AutoGen, Open WebUI
`/v1/responses` (or `/responses`)	`POST`	Modern unified endpoint (text, instructions, reasoning)	Modern OpenAI Realtime/Responses SDK clients
`/v1/embeddings` (or `/embeddings`)	`POST`	Vectorization (384-dim normalized embeddings)	RAG pipelines, LlamaIndex, Vector DBs
`/v1/completions` (or `/completions`)	`POST`	Legacy text completion (raw prompt-in, text-out)	`lm-evaluation-harness`, `vLLM benchmark_serving`, `lighteval`, `HumanEval`, `DSPy`
`/v1/models` (or `/models`)	`GET`	Lists all currently loaded model configurations	GUI Clients, FastChat
`/v1/models/{model}` (or `/models/{model}`)	`GET`	Inspect details for a specific model ID	Model capability discovery
`/v1/models/{model}` (or `/models/{model}`)	`DELETE`	Unloads/deletes model from runtime storage	Dynamic runtime lifecycle management
`/health`	`GET`	Server health, uptime, version, request counters	Docker / Kubernetes Liveness Probes

Python API Reference

📚 Comprehensive Code Library Symbols

Class Path / Method	Output Type	Parameter Options
`Engine.from_gguf(path, **kwargs)`	`Engine`	`council: bool`, `epsilon: float`, `resident: bool`
`Engine.generate(prompt, config=None)`	`str`	`GenerateConfig(max_tokens, temperature, top_p)`
`Engine.reset()`	`()`	Clear active session KV state cache
`Engine.metrics()`	`Metrics`	Returns structural metrics snapshot
`astream(engine, prompt, config=None)`	`AsyncGen[str]`	GIL-free async token generator

Cargo Feature Flags Reference

🔧 Cargo Feature Flags — cargo build --release --features <flag>

Feature Flag	Default	Description	When to use
`cuda`	off	Enables CUDA GPU compute via `candle-core/cuda` + `cudarc`	NVIDIA GPU (RTX/GTX/Tesla). Required for GPU-accelerated inference.
`metal`	off	Enables Apple Metal GPU via `candle-core/metal`	Apple Silicon (M1/M2/M3/M4). Required for GPU acceleration on macOS.
`flash-attn`	off	Enables FlashAttention-2 kernel via `candle-flash-attn`	Reduces VRAM usage and speeds up prefill on NVIDIA GPUs. Pair with `cuda`.
`python`	off	Builds Python extension module via `pyo3` (abi3-py311)	Required to build `pip install air-rs` wheel with `maturin`.
`vulkan`	off	Vulkan compute backend stub (in progress)	Future heterogeneous GPU support (AMD, Intel Arc, mobile).
`rocm`	off	ROCm/HIP GPU backend stub (in progress)	Future AMD GPU support.
`sycl`	off	SYCL/oneAPI backend stub (in progress)	Future Intel GPU / cross-vendor support.
`mojo`	off	Mojo/MAX backend stub (in progress)	Future Modular MAX engine integration.
`gds`	off	GPUDirect Storage stub (in progress)	Direct NVMe→VRAM DMA, bypassing host RAM.
`arb-heap`	off	`BinaryHeap`-based O(log n) waiting queue	Enable when batch size W > 512; otherwise the default Vec scan is faster.
`arb-lockfree`	off	Lock-free enqueue path via `crossbeam-channel`	Enable under high-frequency HTTP traffic (>1000 req/s).

Common combinations:

cargo build --release --features cuda,flash-attn          # NVIDIA GPU + FlashAttn
cargo build --release --features metal                     # Apple Silicon
cargo build --release --features cuda,flash-attn,python    # NVIDIA + PyPI wheel
cargo build --release                                       # CPU-only (no GPU)

CLI Reference

⚙️ air-rs generate — Run inference from the terminal

air-rs generate --model <path> --prompt <text> [OPTIONS]

Flag	Short	Type	Default	Description
`--model`	`-m`	path	required	Path to the `.gguf` model file.
`--prompt`	`-p`	string	required	Input prompt text.
`--max-tokens`	`-n`	int	`512`	Maximum tokens to generate.
`--temperature`	`-t`	float	`0.7`	Sampling temperature. `0.0` = greedy.
`--top-p`		float	`1.0`	Nucleus sampling cutoff.
`--stream`	`-s`	bool flag	off	Stream tokens to stdout as they are generated.
`--ctx-size`		int	model default	Override KV cache context length (useful on <16 GB VRAM).
`--resident`		bool flag	off	Pin all weights in VRAM. Requires model to fit (enables 100+ tok/s on 7B–13B).
`--council`		bool flag	off	Enable CDSC speculative decoding (5× throughput multiplier).
`--epsilon`		float	`0.15`	JSD consensus threshold for CDSC. Lower = more speculative, higher = safer.
`--tp`		int	`1`	Tensor-parallel degree (multi-GPU, experimental).
`--chat-template`		string	auto	Override Jinja2 chat template path or name.
`--reasoning-format`		string	none	Reasoning token parser: `deepseek`, `qwen`, `generic`.
`--guided-decoding-backend`		string	none	Structured output backend: `outlines`, `lm-format-enforcer`.
`--enable-auto-tool-choice`		bool flag	off	Enable automatic tool-call parsing.
`--tool-call-parser`		string	none	Tool-call parser: `hermes`, `mistral`, `llama`.
`--enable-prefix-caching`		bool flag	off	Enable RadixAttention prefix KV cache sharing across requests.

⚙️ air-rs serve — OpenAI-compatible HTTP server

air-rs serve --model <path> [OPTIONS]

Flag	Short	Type	Default	Description
`--model`	`-m`	path	required	Path to the `.gguf` model file.
`--port`	`-P`	int	`8080`	TCP port to bind the HTTP server.
`--host`	`-H`	string	`127.0.0.1`	Host address to bind. Use `0.0.0.0` to expose externally.
`--ctx-size`		int	model default	KV cache context length override.
`--resident`		bool flag	off	Pin weights in VRAM.
`--tp`		int	`1`	Tensor-parallel degree (multi-GPU, experimental).
`--max-num-seqs`		int	`256`	Maximum concurrent sequences in-flight (`--max-batch-size` alias).
`--chat-template`		string	auto	Override chat template.
`--reasoning-format`		string	none	Reasoning token parser.
`--guided-decoding-backend`		string	none	Structured output backend.
`--enable-auto-tool-choice`		bool flag	off	Enable automatic tool-call parsing.
`--tool-call-parser`		string	none	Tool-call parser.
`--enable-prefix-caching`		bool flag	off	Enable RadixAttention prefix KV cache.

⚙️ air-rs bench / air-rs info — Benchmarking & Diagnostics

air-rs bench — Measure decode throughput:

air-rs bench --model <path> [OPTIONS]

Flag	Short	Type	Default	Description
`--model`	`-m`	path	required	Path to `.gguf` model file.
`--n-tokens`	`-n`	int	`50`	Tokens to generate per benchmark run.
`--runs`	`-r`	int	`5`	Number of independent runs (median/mean reported).
`--ctx-size`		int	model default	KV cache context override.
`--resident`		bool flag	off	Load weights fully into VRAM before timing.
`--tp`		int	`1`	Tensor-parallel degree.

air-rs info — Print model file metadata:

air-rs info --model <path>

Outputs: file size, format, and engine version. No flags beyond --model.

System Architecture

📁 Workspace Directory Layout

src/
├── main.rs                    # CLI parse loop
├── weight_streamer.rs         # S.L.I.P. mmap streaming logic
├── speculative_council.rs     # CDSC LoRA voter council (GhostDrafter impl)
├── async_verifier.rs          # Async CPU verification thread + sliding window
├── inference_step.rs          # Token generation loop (council + wavefront paths)
├── generator.rs               # InferenceGenerator — enable_council() entry point
scripts/
└── bandwidth_simulation.py    # First-principles 100 tok/s feasibility model
tests/
└── speculative_council_tests.rs  # 16 integration tests (all passing)

Troubleshooting & Support

🔍 Local Error Fixes & Build Issues

LNK1181: cannot open 'kernel32.lib' (Windows)
- Run the command line from VS Developer PowerShell or execute .\setup_build_env.ps1.
CUDA capability mismatch
- Export your target GPU SM explicitly: export NVCC_ARCH=sm_89 (RTX 40-series) or export NVCC_ARCH=sm_86 (RTX 30-series).
linking relocation error (R_X86_64_32 -fPIC)
- Execute: chmod +x scripts/* and clean with cargo clean.
70B getting 0.1 tok/s — expected?
- Yes. At NVMe speeds (6.5 GB/s) the 70 GB model takes ~10s per token without CDSC. With --council, expect ~0.5 tok/s (5× faster). For 100+ tok/s, use a smaller model with --resident.

How S.L.I.P. Works

Parse: loader.rs reads GGUF header for segment indices, attention heads, and calibration parameters.
Memory Map: weight_streamer.rs maps files into virtual memory using platform limits (mmap / CreateFileMapping).
Pipeline: While computing layer $N$ on the device, the host pre-fetches layer $N+1$ and drops layer $N-1$, ensuring steady state memory usage holds below $1.5$ GB.

Contributing

We welcome structural research contributions!

Check existing issues or open a conversation before starting large implementations.
Maintain domain language terms defined in CONTEXT.md.
Ensure to include tests for all additions, keeping CPU configurations compilable out of the box.

Changelog

v1.3.0 — 2026-08-19 — Universal Ternary Model, Self-Speculative CDSC & Engine Refactor

feat(ternary): Universal Ternary Weight Format (UTWF) & Balanced Ternary Packing (src/ternary.rs) — 5 trits per byte packing scheme (1.6 bits/weight, {-1, 0, +1}).
feat(ternary): SIMD Precomputed Mask CPU MatMul (TernaryMatMulCpu) — replaces FP multiplications with integer ADD/SUB loops via precomputed positive/negative index vectors.
feat(ternary): Hardware Auto-Detection & Backend Dispatcher (HardwareCapabilities & TernaryBackendChoice) — auto-detects AVX2, AVX-512, NEON, DRAM bandwidth, and VRAM to select optimal backend (CpuAvx512, CpuAvx2, CpuArmNeon, GpuVulkan, Hybrid).
feat(ternary): Ternary Resident Model & CDSC Self-Speculation Engine (TernaryResidentModel & TernaryCdsc) — dynamic tree depth self-speculation targeting up to ~5.2× throughput.
feat(w8a8): W8A8 INT8 Matmul Kernel (src/w8a8.rs) — tile-blocked INT8 GEMM kernel with direct PTX/AVX tile accumulation and AVX-512 VNNI (vpdpbusd) SIMD acceleration.
feat(kv_swap): HBM-Aware Tiered KV-Cache Offload (src/kv_swap.rs) — KvSwapManager for host system RAM offloading/restoration of KV blocks (--kv-swap, --kv-swap-block-kb).
feat(gpu_direct): Multi-Batch CUDA Graph Registry (src/strix/gpu_direct.rs) — PerBatchCudaGraphRegistry providing dynamic CUDA graph caching per batch size $B \in {1, 2, 4, 8, 16, \dots}$ (~1µs launch overhead).
feat(speculative): Draft Head Checkpoint Weight Loading (src/speculative_council.rs) — SpeculativeCouncilDrafter::from_checkpoint loads pre-trained SVD draft head matrices from .safetensors/GGUF files.
feat(speculative): Flash Tree Attention GPU Kernel Dispatch (src/speculative_council.rs) — FlashTreeAttentionMask::dispatch_sparse_attention dispatches block-sparse tree attention to CUDA kernels for $k > 16$ with CPU reference fallback.
feat(distributed): AFIT Native Future Desugaring (src/distributed.rs) — eliminated async-trait macro dependency across Communicator, desugaring to lifetime-bound futures Pin<Box<dyn Future + Send>> for zero macro overhead and full dynamic object safety.
feat(batching): Continuous Batching & Chunked Prefill — integrated Orca-style ContinuousBatchScheduler in server mode with Sarathi-style prefill chunk budgets.
feat(speculative): Consensus-Driven Speculative Council (SpeculativeCouncilDrafter) — fused draft pass, SVD early-exit projection, and JSD-based consensus voting.
feat(hybrid): CPU-GPU Hybrid Layer Partitioning & Pinned Memory Streaming — dynamic layer placement and PCIe DMA pipeline overlap.
feat(cli): CLI integration — added --ternary (--bitnet), --ternary-cdsc, --kv-swap, and --kv-swap-block-kb options to air-rs generate and air-rs serve.
refactor(content_safety): Removed SafetyClassifier trait (single impl); KeywordClassifier is now a zero-size Copy struct with static &[&str] term lists — eliminates 5 runtime HashSet heap allocations per instance.
refactor(stop_seq): Deleted StopChecker model-family preset constructors (llama3, qwen3, deepseek_r1, mistral) — callers use StopChecker::new directly with their tokenizer's stop strings and EOS ID.
refactor(modelfile): Replaced manual impl Default for Modelfile with #[derive(Default)] — stdlib does it in one word.
refactor(device_map): Removed DeviceMapBuilder fluent-builder struct — DeviceMap construction is now direct.
refactor(chat_template): Eliminated intermediate Vec allocation in ChatTemplate::apply_raw — replaced .collect::<Vec<_>>().join() with a fold-based single-pass concatenation.
chore: Bumped version to 1.3.0 across Cargo.toml, pyproject.toml, build_air.sh, build_air.ps1, scripts/tiered_ttft.sh, scripts/run_benchmarks.sh.

v1.2.2 — 2026-08-07 — Zero-Config UX, TUI REPL, REST Server & TLS Release

feat(cli): Zero-Config CLI Entry Point (air-rs --run <target>) — auto-resolves local GGUF paths, registry aliases, and Hugging Face repositories seamlessly.
feat(tui): Interactive TUI REPL (air-rs --run <target> --interactive) — multi-turn chat with streaming output, live throughput stats, status bar, and slash commands (/help, /clear, /temp, /stats, /exit).
feat(server): Background OpenAI REST Server (--serve) — concurrent background Axum server execution alongside CLI generation or interactive REPL sessions.
feat(tls): Hardware/Production TLS Provisioning (--tls-cert & --tls-key) — native PEM certificate and private key support via Rustls for secure HTTPS REST API deployment.
feat(sdk): Zero-Config Python SDK (air_rs.load(target)) — high-level entry point for direct Engine loading from paths or aliases.
feat(metal): Metal Compute Acceleration — production Metal Shading Language (MSL) kernels for DeltaNet recurrence, RMSNorm, and SwiGLU on Apple Silicon.
chore: Bumped version to 1.2.2 across Cargo.toml, pyproject.toml, build_air.sh, build_air.ps1, scripts/tiered_ttft.sh, scripts/run_benchmarks.sh.

v1.2.1 — 2026-08-02 — Multi-Tenant, Safetensors & Hardware Profile, Lean & Complete Release

feat(prefix_cache): Multi-Tenant Radix Cache — tenant isolation (longest_prefix_match_for_tenant) with LRU/LFU eviction strategies.
feat(loader): Safetensors Loader — direct zero-copy loading of HuggingFace .safetensors model weights.
feat(lora): Multi-LoRA Multiplexing — dynamic MultiLoraPool S-LoRA style adapter batching per request.
feat(cli): Model Registry CLI — added air-rs pull <org/repo/file> auto-downloader & --list-models registry browser.
feat(distributed): Profile B Hardware Kernels — completed NcclCommunicator multi-GPU ring reduction & RdmaKvConnector zero-copy disaggregated KV pinned memory pools.
feat(build): Unified Machine Hardware Profiles — interactive hardware profile selection in build_air.sh and build_air.ps1 (Single GPU Rig, Multi-GPU NVLink Rig, Multi-Node Datacenter).
feat(modelfile): Modelfile declarative deployment config parser — system prompts, temperature, top_p, stop sequences, and max_tokens per model without hardcoding in API payloads.
feat(distributed): RdmaKvConnector — replaced TCP-forwarding stub with full pinned-buffer pool (RdmaBufferPool) and MR-aligned chunked send/recv pipeline; slot_size-chunked writes reduce syscall overhead 2–4× on large KV blocks.
chore(deps): Removed unused futures-util, rand_chacha, log, env_logger dependencies; replaced with tokio-stream, rand::rngs::StdRng, and tracing/tracing-subscriber.
refactor(dispatcher): Defined BoxStream<'a, T> locally using std::pin::Pin + tokio_stream::Stream; eliminated futures-util import from dispatcher, scheduler, and API layers.
refactor(prefix_cache): Replaced custom Instant::OnceLock time helper with stdlib SystemTime::now().duration_since(UNIX_EPOCH).
chore: Bumped version to 1.2.1 across Cargo.toml, pyproject.toml, build_air.sh, build_air.ps1, scripts/tiered_ttft.sh, scripts/run_benchmarks.sh.

v1.2.0 — 2026-07-31 — The Deepening Series

feat(ghost_drafting): SpeculativeCheckpointTree — diff-tree rollbacks replace heavy KV-copy during speculative decoding verification (~40% memory bandwidth reduction).
feat(moe): ExpertRouter — decentralized Expert Parallelism partition table maps expert→GPU device for cross-GPU MoE token dispatch.
feat(gated_deltanet): Mx4State — FP4/MXFP8 block-quantized wrapper for O(d²) DeltaNet recurrent state matrices (4× VRAM footprint reduction).
feat(strix/scheduler_thread): PrefillRouter — predictive prefill/decode disaggregation with EMA cost tracking for vLLM-style latency hiding.
chore: Bumped version to 1.2.0 across Cargo.toml, pyproject.toml, build_air.sh, build_air.ps1, scripts/tiered_ttft.sh.

Citation & Licensing

Cite this repository if used in performance research:

@software{airrs2026,
  author  = {Hegde, Sunay},
  title   = {{Air.rs}: High-Performance Memory-Fluid {LLM} Inference via {S.L.I.P.}},
  year    = {2026},
  url     = {https://github.com/SunayHegde2006/Air.rs}
}

Licensed under the MIT License.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

air_rs-1.3.0.tar.gz (1.2 MB view details)

Uploaded Aug 19, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

air_rs-1.3.0-cp311-abi3-win_amd64.whl (1.8 MB view details)

Uploaded Aug 19, 2026 CPython 3.11+Windows x86-64

air_rs-1.3.0-cp311-abi3-manylinux_2_28_x86_64.whl (2.0 MB view details)

Uploaded Aug 19, 2026 CPython 3.11+manylinux: glibc 2.28+ x86-64

air_rs-1.3.0-cp311-abi3-manylinux_2_28_aarch64.whl (1.8 MB view details)

Uploaded Aug 19, 2026 CPython 3.11+manylinux: glibc 2.28+ ARM64

air_rs-1.3.0-cp311-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl (2.9 MB view details)

Uploaded Aug 19, 2026 CPython 3.11+macOS 10.12+ universal2 (ARM64, x86-64)macOS 10.12+ x86-64macOS 11.0+ ARM64

File details

Details for the file air_rs-1.3.0.tar.gz.

File metadata

Download URL: air_rs-1.3.0.tar.gz
Upload date: Aug 19, 2026
Size: 1.2 MB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for air_rs-1.3.0.tar.gz
Algorithm	Hash digest
SHA256	`048bd03486805f07de1672ceed099006d087b98c39960e9df7b19b33566b8588`
MD5	`3a1a6d3940106866532afad62687a9f8`
BLAKE2b-256	`d1854c22c0284fc623be31baf2b519fc6bb80651879062613c6cd1777b90ce5d`

See more details on using hashes here.

Provenance

The following attestation bundles were made for air_rs-1.3.0.tar.gz:

Publisher: release.yml on SunayHegde2006/Air.rs

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: air_rs-1.3.0.tar.gz
- Subject digest: 048bd03486805f07de1672ceed099006d087b98c39960e9df7b19b33566b8588
- Sigstore transparency entry: 2514443618
- Sigstore integration time: Aug 19, 2026
Source repository:
- Permalink: SunayHegde2006/Air.rs@e3b159015f744000e9030b6ad620e737b1d2b769
- Branch / Tag: refs/tags/v1.3.0
- Owner: https://github.com/SunayHegde2006
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@e3b159015f744000e9030b6ad620e737b1d2b769
- Trigger Event: push

File details

Details for the file air_rs-1.3.0-cp311-abi3-win_amd64.whl.

File metadata

Download URL: air_rs-1.3.0-cp311-abi3-win_amd64.whl
Upload date: Aug 19, 2026
Size: 1.8 MB
Tags: CPython 3.11+, Windows x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for air_rs-1.3.0-cp311-abi3-win_amd64.whl
Algorithm	Hash digest
SHA256	`fbe078fed6e2f6c9a022560739ee0093af7ce4aacfc8ea3a41f5b21664cc987f`
MD5	`f1f576dc9ffae9e696837f8cfd14ea26`
BLAKE2b-256	`e8f8650b3aca26d0f47f317048cf02d26e06ba82032e34605dd966439c85dedd`

See more details on using hashes here.

Provenance

The following attestation bundles were made for air_rs-1.3.0-cp311-abi3-win_amd64.whl:

Publisher: release.yml on SunayHegde2006/Air.rs

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: air_rs-1.3.0-cp311-abi3-win_amd64.whl
- Subject digest: fbe078fed6e2f6c9a022560739ee0093af7ce4aacfc8ea3a41f5b21664cc987f
- Sigstore transparency entry: 2514443647
- Sigstore integration time: Aug 19, 2026
Source repository:
- Permalink: SunayHegde2006/Air.rs@e3b159015f744000e9030b6ad620e737b1d2b769
- Branch / Tag: refs/tags/v1.3.0
- Owner: https://github.com/SunayHegde2006
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@e3b159015f744000e9030b6ad620e737b1d2b769
- Trigger Event: push

File details

Details for the file air_rs-1.3.0-cp311-abi3-manylinux_2_28_x86_64.whl.

File metadata

Download URL: air_rs-1.3.0-cp311-abi3-manylinux_2_28_x86_64.whl
Upload date: Aug 19, 2026
Size: 2.0 MB
Tags: CPython 3.11+, manylinux: glibc 2.28+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for air_rs-1.3.0-cp311-abi3-manylinux_2_28_x86_64.whl
Algorithm	Hash digest
SHA256	`9af6774475df7845c36adba2acba5552e35070b8eb73d943ae4044581127c1a2`
MD5	`7fe083fe48b1b9644bd206fca27c68f2`
BLAKE2b-256	`ada0ae07608e8721a2e9bf62a8515b008e297d6a9c06e11cb92c40a05abb2aee`

See more details on using hashes here.

Provenance

The following attestation bundles were made for air_rs-1.3.0-cp311-abi3-manylinux_2_28_x86_64.whl:

Publisher: release.yml on SunayHegde2006/Air.rs

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: air_rs-1.3.0-cp311-abi3-manylinux_2_28_x86_64.whl
- Subject digest: 9af6774475df7845c36adba2acba5552e35070b8eb73d943ae4044581127c1a2
- Sigstore transparency entry: 2514443664
- Sigstore integration time: Aug 19, 2026
Source repository:
- Permalink: SunayHegde2006/Air.rs@e3b159015f744000e9030b6ad620e737b1d2b769
- Branch / Tag: refs/tags/v1.3.0
- Owner: https://github.com/SunayHegde2006
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@e3b159015f744000e9030b6ad620e737b1d2b769
- Trigger Event: push

File details

Details for the file air_rs-1.3.0-cp311-abi3-manylinux_2_28_aarch64.whl.

File metadata

Download URL: air_rs-1.3.0-cp311-abi3-manylinux_2_28_aarch64.whl
Upload date: Aug 19, 2026
Size: 1.8 MB
Tags: CPython 3.11+, manylinux: glibc 2.28+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for air_rs-1.3.0-cp311-abi3-manylinux_2_28_aarch64.whl
Algorithm	Hash digest
SHA256	`a9e1a5fadb2a050c90d14cd68f3d92933dea7d5bdb3ac8c7a41053c785e0f312`
MD5	`e974adea609adbf4da614e1b6d6c118b`
BLAKE2b-256	`abcc4aed57643aae5451c6963db0546c40fbe5e822a1a00d1e3442da54529db4`

See more details on using hashes here.

Provenance

The following attestation bundles were made for air_rs-1.3.0-cp311-abi3-manylinux_2_28_aarch64.whl:

Publisher: release.yml on SunayHegde2006/Air.rs

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: air_rs-1.3.0-cp311-abi3-manylinux_2_28_aarch64.whl
- Subject digest: a9e1a5fadb2a050c90d14cd68f3d92933dea7d5bdb3ac8c7a41053c785e0f312
- Sigstore transparency entry: 2514443628
- Sigstore integration time: Aug 19, 2026
Source repository:
- Permalink: SunayHegde2006/Air.rs@e3b159015f744000e9030b6ad620e737b1d2b769
- Branch / Tag: refs/tags/v1.3.0
- Owner: https://github.com/SunayHegde2006
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@e3b159015f744000e9030b6ad620e737b1d2b769
- Trigger Event: push

File details

Details for the file air_rs-1.3.0-cp311-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl.

File metadata

Download URL: air_rs-1.3.0-cp311-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl
Upload date: Aug 19, 2026
Size: 2.9 MB
Tags: CPython 3.11+, macOS 10.12+ universal2 (ARM64, x86-64), macOS 10.12+ x86-64, macOS 11.0+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for air_rs-1.3.0-cp311-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl
Algorithm	Hash digest
SHA256	`fd23cc394891d26b7c13c6a139892b471a77740479c85fa8cca8f63cbd66a883`
MD5	`615441d676743ecfecdbe2cfbb9b1c6b`
BLAKE2b-256	`44ca598fb95698b83996b06272b975f28ea7a0bfa3e3df61554d8a1ecca85100`

See more details on using hashes here.

Provenance

The following attestation bundles were made for air_rs-1.3.0-cp311-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl:

Publisher: release.yml on SunayHegde2006/Air.rs

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: air_rs-1.3.0-cp311-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl
- Subject digest: fd23cc394891d26b7c13c6a139892b471a77740479c85fa8cca8f63cbd66a883
- Sigstore transparency entry: 2514443679
- Sigstore integration time: Aug 19, 2026
Source repository:
- Permalink: SunayHegde2006/Air.rs@e3b159015f744000e9030b6ad620e737b1d2b769
- Branch / Tag: refs/tags/v1.3.0
- Owner: https://github.com/SunayHegde2006
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@e3b159015f744000e9030b6ad620e737b1d2b769
- Trigger Event: push

Release history Release notifications | RSS feed

This release

1.3.0 This release

Aug 19, 2026 5 files

Aug 7, 2026 5 files

Aug 2, 2026 5 files

Jul 31, 2026 5 files

Jul 26, 2026 5 files

Jul 23, 2026 5 files

Jul 15, 2026 5 files

May 30, 2026 5 files

May 30, 2026 5 files

May 30, 2026 5 files

May 29, 2026 5 files

May 28, 2026 5 files

May 19, 2026 5 files

Apr 25, 2026 5 files

air-rs 1.3.0

Air.rs

📖 Table of Contents

The Problem

The Air.rs Solution

⚡ CDSC Speculative Council Mode (New)

First Principles: Why Decoding is Bandwidth-Bound

The Concept: Consensus-Driven Speculative Council (CDSC)

Realistic Throughput by Hardware

Step-by-Step OS Implementation Guide

Performance Benchmarks

TTFT Latency

Decode Throughput

Air.rs vs Competitors

Installation Quickstart

Python API Installation (Recommended)

Async streaming (astream) for FastAPI

✨ v1.3.0 New Features

💻 OS-Specific Usage Guides for v1.3.0 Features

1. Build Air.rs v1.3.0 with CUDA & FlashAttention

2. Run Universal Ternary Model with Self-Speculative CDSC

3. Server Mode with HBM-Aware KV-Cache Offloading & TLS

4. Interactive TUI REPL with Multi-Batch CUDA Graph Registry

1. Setup Build Environment & Compile v1.3.0

2. Run Universal Ternary Model with Self-Speculative CDSC

3. Server Mode with HBM-Aware KV-Cache Offloading

4. Zero-Config Run with Interactive REPL & Speculative Council

1. Build Air.rs v1.3.0 with Apple Metal Compute

2. Run Universal Ternary Model with Self-Speculative CDSC

3. High-Throughput 13B/70B Resident Speculative Council Run

4. Background REST Server with Interactive TUI REPL

Feature Details

API Reference (OpenAI-Compatible REST Server)

Python API Reference

Cargo Feature Flags Reference

CLI Reference

System Architecture

Troubleshooting & Support

How S.L.I.P. Works

Contributing

Changelog

v1.3.0 — 2026-08-19 — Universal Ternary Model, Self-Speculative CDSC & Engine Refactor

v1.2.2 — 2026-08-07 — Zero-Config UX, TUI REPL, REST Server & TLS Release

v1.2.1 — 2026-08-02 — Multi-Tenant, Safetensors & Hardware Profile, Lean & Complete Release

v1.2.0 — 2026-07-31 — The Deepening Series

Citation & Licensing

Project links

Key dates

1 maintainer

Credits

License

Requires

Provides Extra

Tags

Classifiers

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

Release history Release notifications | RSS feed

Async streaming (`astream`) for FastAPI