tq-serve 0.4.22

TurboQuant model server manager — auto-configured llama-server with KV cache compression

tq — TurboQuant Model Server Manager

Auto-configured llama-server with KV cache compression.

Install

# macOS (Apple Silicon) — one-liner
curl -fsSL https://raw.githubusercontent.com/xt8086/tq/main/install.sh | bash

# Windows (x64, NVIDIA GPU required) — PowerShell
pip install tq-serve; python -m tq install

# Any platform — pip
pip install tq-serve && python3 -m tq install

Note: tq-serve is a pure Python package — it does NOT include the llama-server binary. The binary comes from TheTom/llama-cpp-turboquant, a fork of llama.cpp with TurboQuant KV cache compression built in. tq install downloads the correct binary for your platform (macOS Metal, Windows CUDA, Linux CUDA/ROCm/CPU).

PATH issue: If tq is not found after pip install, use python3 -m tq instead (e.g. python3 -m tq install, python3 -m tq chat). This always works because python3/python is on PATH by default on all platforms.

Windows: An NVIDIA GPU is required — there is no CPU-only Windows binary available. Windows without NVIDIA GPU is not currently supported.

Quick Start

python3 -m tq doctor                        # Verify setup
python3 -m tq list                           # List local GGUF models
python3 -m tq search "qwen2.5 coder 7b"      # Search HuggingFace (numbered, pick # to download)
python3 -m tq serve 1                        # Launch with auto-configured TurboQuant
python3 -m tq chat                           # Interactive coding agent

How It Works

tq serve automatically:

1. Detects your hardware (GPU, RAM)
2. Parses model metadata (quant type, layers, context length)
3. Calculates optimal TurboQuant cache settings
4. Launches llama-server with the right flags

Example: A Q4_K_M model on Apple M1 with 8GB RAM gets:

• ctk=q8_0 (protect K cache)
• ctv=turbo4 (compress V cache 3.8x)
• Context capped to safe memory limit
• Idle auto-stop after 5 min

Commands

Command	Description
tq list	List local GGUF models
tq search <query>	Search HuggingFace for GGUF models (numbered, with download prompt)
tq download <model>	Download a model from HuggingFace
tq remove <model>	Remove a downloaded model
tq serve <model>	Launch with auto TurboQuant config
tq serve 1	Serve by list number
tq serve 1 --dry-run	Show command without running
tq status	Check if server is running
tq stop	Stop the server
tq logs	View server logs
tq validate <model>	Pre-flight check
tq install	Download TurboQuant+ binary (from TheTom/llama-cpp-turboquant)
tq doctor	Verify setup
tq config show	Show/edit configuration
tq chat	Interactive coding agent (local AI)

API

The server exposes an OpenAI-compatible API:

POST http://127.0.0.1:8080/v1/chat/completions

No auth needed. Works with any OpenAI client:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="your-model.gguf",
    messages=[{"role": "user", "content": "Hello"}]
)

Configuration

Config stored at ~/.tq/config.toml:

tq config show              # Show all settings
tq config set port 9090     # Change port
tq config set idle_timeout 600  # 10 min idle timeout (0 to disable)

TurboQuant Cache Types

Type	Bits	Compression	Use Case
f16	16	1x	No compression (baseline)
q8_0	8	2x	Safe for K cache
turbo4	4.25	3.8x	Best quality/compression for V
turbo3	3.25	4.9x	Aggressive, for large models

Requirements

• Python 3.10+
• macOS (Apple Silicon), Windows (x64, NVIDIA GPU), or Linux (x86_64 with NVIDIA/AMD GPU)
• ~2GB free RAM minimum (depends on model)