vaultlm 1.1.0

Local AI inference for privacy-first edge deployment — Bonsai-8B with TurboAgent KV compression

VaultLM

Local AI inference for privacy-first edge deployment.

Runs Bonsai-8B (Qwen3-8B-Q1_0) on any NVIDIA GPU (RTX 30xx/40xx/50xx) with optional TurboAgent KV compression (v1.1) for 5× memory efficiency at long contexts.

All inference is local — no data leaves your machine.

---

Requirements

• Python 3.10+
• NVIDIA GPU: RTX 30xx (SM 86), RTX 40xx (SM 89), or RTX 50xx (SM 120)
• CUDA Toolkit (installed automatically by vaultlm setup)
• ~13 GB disk space (model + binaries)
• Windows 10/11 or Linux

---

Install

pip install vaultlm[all]

The PyPI wheel contains only the Python code and CLI. The model, PrismML binaries, and the custom llama-cpp-python build are downloaded and compiled on first run:

vaultlm setup

setup will:

1. Detect your GPU (SM version)
2. Install CUDA Toolkit if missing (winget on Windows, apt on Linux)
3. Clone llama-cpp-python and vendor the PrismML fork into it
4. Build llama-cpp-python with CUDA + correct SM architecture
5. Download the PrismML CUDA binary and DLLs
6. Download Bonsai-8B-Q1_0.gguf from Hugging Face

All files land in ~/.vaultlm/.

---

Quickstart

# One-time setup
vaultlm setup

# Start server (TurboAgent KV compression enabled automatically if ABI check passes)
vaultlm serve

# Run smoke tests
vaultlm test

# Run specific suite
vaultlm test --suite coherence
vaultlm test --suite niah
vaultlm test --suite kv        # TurboAgent KV compression validation
vaultlm test --suite stress    # 60-request marathon benchmark

---

Optional: GPU monitoring extras

pip install vaultlm[gpu]
vaultlm test --suite gpu

---

v1.0 vs v1.1

Feature	v1.0	v1.1
Backend	subprocess llama-server	TurboAgent (+ subprocess fallback)
KV compression	None	turbo3 (~5.28×)
Context @ 12 GB VRAM	~32k	~65k+
GPU support	RTX 30xx	RTX 30xx / 40xx / 50xx

v1.1 falls back silently to v1.0 behaviour if the PrismML ABI build is not present (i.e. if vaultlm setup has not been run, or was run with --skip-build).

---

CPU / Vulkan

vaultlm serve --cpu    # Vulkan backend, no CUDA required
vaultlm test --suite throughput -- --path cpu

---

Results directory

Stress test JSON results are saved to ~/.vaultlm/results/.

---

License

Apache 2.0