bit-ttt-engine 0.8.0

Fast local LLM inference with TTT (Test-Time Training) and LoRA — the model that learns while it runs

🧠 Bit-TTT-Engine

Fast local LLM inference that learns while it runs.

• 🏎️ 47+ tok/s on RTX 4060 Ti (7B Q4_K_M)
• 🧠 TTT (Test-Time Training) — adapts during inference (world's first!)
• 🎨 LoRA — fine-tune with one flag
• 📦 5 models — Llama-2/3, Gemma-2, Qwen2.5, Mistral
• 🔌 OpenAI-compatible API — drop-in replacement

🚀 Quick Start

pip install bit-ttt-engine

import cortex_rust

# Load any GGUF model (auto-downloads from HuggingFace!)
model = cortex_rust.load("user/model-GGUF")

# Chat
response = model.chat([
    {"role": "user", "content": "Hello!"}
])
print(response)

# Stream
for token in model.chat_stream([
    {"role": "user", "content": "Tell me a story"}
]):
    print(token, end="", flush=True)

🖥️ CLI

# Interactive chat
bit-ttt chat model.gguf

# Generate text
bit-ttt generate model.gguf -p "Once upon a time"

# OpenAI-compatible API server
bit-ttt serve model.gguf --port 8000

# With LoRA + Q8 KV cache
bit-ttt chat model.gguf --lora adapter.bin --q8-cache

🧠 TTT — Test-Time Training

The model learns while it generates. No other local LLM does this.

model = cortex_rust.load("model.gguf")
model.enable_ttt(True)

# Each conversation makes the model smarter
response = model.chat([{"role": "user", "content": "My name is Alice"}])
# Next time, it remembers context better!

⚡ Performance

Model	Speed	VRAM
Llama-2 7B (Q4_K_M)	47.8 tok/s	~5 GB
Llama-3 8B (Q4_K_M)	36.8 tok/s	~6 GB
Mistral 7B (Q4_K_M)	40.8 tok/s	~5 GB
Qwen2.5 1.5B (Q4_K_M)	70.4 tok/s	~2 GB

With --q8-cache: 82% VRAM reduction for KV cache.

🔌 OpenAI-Compatible API

bit-ttt serve model.gguf --port 8000

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
response = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Hi!"}],
    stream=True,
)

📖 Links

• GitHub
• Documentation

💖 License

MIT License