trillim 0.5.0

The fastest inference framework to run BitNet models on CPUs

Trillim

What is Trillim?

Quick Start

Installation

• Python 3.12+ required
• Install with uv (recommended) or pip

Pick your platform for full instructions:

• macOS
• Linux
• Windows

Note: The rest of this README shows bare trillim commands. If you're using uv, prefix each command with uv run (e.g. uv run trillim chat ...).

Quantize your own model

If you have a HuggingFace BitNet model with safetensors weights:

# Quantize model weights → qmodel.tensors + rope.cache
trillim quantize <path-to-model> --model

# Optionally extract a PEFT LoRA adapter → qmodel.lora
trillim quantize <path-to-model> --adapter <path-to-adapter>

Chat

Start an interactive conversation in your terminal:

trillim chat Trillim/BitNet-TRNQ

Multi-turn conversations are supported with automatic prompt caching for fast follow-ups. Use /new to start a fresh conversation, or q to quit.

See the Chat guide for details on LoRA adapters, sampling parameters, and performance tips.

Search-Augmented Chat

Trillim supports pluggable inference harnesses. For web-search-enabled models, use:

trillim chat Trillim/BitNet-Search-TRNQ --harness search

By default, search uses DuckDuckGo (ddgs). To use Brave:

export SEARCH_API_KEY=<your_api_key>
trillim chat Trillim/BitNet-Search-TRNQ --harness search --search-provider brave

The search harness emits status markers while it runs search and synthesis steps. See Chat for full behavior and troubleshooting.

API Server

Trillim includes an OpenAI-compatible API server:

# Start the server
trillim serve Trillim/BitNet-TRNQ

# With voice pipeline (speech-to-text + text-to-speech)
# Requires optional `voice` dependencies:
# docs/server.md -> "Voice Optional Dependencies"
trillim serve Trillim/BitNet-TRNQ --voice

Endpoints:

• POST /v1/chat/completions — chat completions (streaming supported)
• POST /v1/completions — text completions
• GET /v1/models — list loaded models
• POST /v1/models/load — hot-swap models, LoRA adapters, and harness/search settings at runtime
• POST /v1/audio/transcriptions — speech-to-text (with --voice)
• POST /v1/audio/speech — text-to-speech (with --voice)
• GET /v1/voices — list available TTS voices
• POST /v1/voices — register a custom voice from audio (see Voice Cloning Setup)

For server-side search harness, start normally and then set "harness": "search" (plus optional "search_provider") through POST /v1/models/load.

Works with the OpenAI Python client out of the box:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
    model="BitNet-TRNQ",
    messages=[{"role": "user", "content": "Hello!"}],
)

See the Server guide for full endpoint documentation, request/response schemas, the Python SDK, and voice pipeline usage.

LoRA Adapters

Trillim supports PEFT LoRA adapters as bf16 corrections on top of the ternary base model. The adapter lives in its own directory (separate from the base model) and must be quantized first:

# Quantize a PEFT adapter into Trillim's format
trillim quantize <path-to-base-model> --adapter <path-to-adapter>

# Chat with the base model + adapter
trillim chat Trillim/BitNet-TRNQ --lora <adapter-dir>

# Or pull a pre-quantized adapter and use it by ID
trillim pull Trillim/BitNet-GenZ-LoRA-TRNQ
trillim chat Trillim/BitNet-TRNQ --lora Trillim/BitNet-GenZ-LoRA-TRNQ

Adapters can also be hot-swapped at runtime via the API server's POST /v1/models/load endpoint. See the Server guide for details.

Runtime Quantization

Separately from the offline trillim quantize step (which converts model weights to ternary), Trillim can quantize specific layers at inference time to reduce memory usage. This is controlled with two flags available on both chat and serve:

• --lora-quant <type> — quantize LoRA adapter layers. Options: none, int8, q4_0, q5_0, q6_k, q8_0. Only applies when using --lora.
• --unembed-quant <type> — quantize the unembedding (output projection) layer. Options: int8, q4_0, q5_0, q6_k, q8_0.

# Quantize LoRA layers to int8 for lower memory
trillim chat Trillim/BitNet-TRNQ --lora <adapter-dir> --lora-quant int8

# Quantize the unembed layer to q4_0
trillim chat Trillim/BitNet-TRNQ --unembed-quant q4_0

# Both at once
trillim serve Trillim/BitNet-TRNQ --lora-quant q8_0 --unembed-quant q4_0

Lower quantization levels (e.g. q4_0) use less memory at a small quality cost. These options can also be set per-request when hot-swapping models via POST /v1/models/load. See the CLI reference for the full flag list.

Voice Cloning Setup

The voice pipeline (--voice) includes 8 predefined voices that work out of the box: alba, marius, javert, jean, fantine, cosette, eponine, azelma.

To register custom voices (voice cloning via POST /v1/voices), you need to accept the PocketTTS model terms and authenticate with HuggingFace:

1. Go to kyutai/pocket-tts on HuggingFace and accept the model's terms.
2. Create a token on HuggingFace (under Access Tokens) with Read permissions.
3. Log in locally so the token is available to download the voice cloning weights:

hf auth login

This only needs to be done once. After that, custom voice registration works automatically. If you skip this step, you'll get an error when trying to register a custom voice — predefined voices will still work fine.

Supported Architectures

• BitnetForCausalLM — BitNet with ternary weights and ReLU² activation
• LlamaForCausalLM — Llama-style with SiLU activation

Platform Support

Platform	Status
x86_64 (AVX2)	Supported
ARM64 (NEON)	Supported

Thread count is auto-detected as num_cores - 2. Override by passing a --threads N CLI argument.

Documentation

• What is Trillim? — overview, motivation, and who it's for
• Install — macOS | Linux | Windows
• CLI Reference — all commands and flags
• Chat — interactive chat interface
• Server — API endpoints, Python SDK, and OpenAI client usage

License

The Trillim Python SDK source code is MIT-licensed. The C++ inference engine binaries (inference, trillim-quantize) bundled in the pip package are proprietary — you may use them as part of Trillim but may not reverse-engineer or redistribute them separately. See LICENSE for full terms.