trillim 0.8.0

The fastest inference framework to run AI on CPUs

Trillim

Trillim is a local AI stack for CPUs. It gives you a CLI, a Python SDK, and a FastAPI server for running Trillim-formatted LLM bundles, plus optional speech-to-text and text-to-speech support.

Trillim supports both BitNet-style ternary bundles and Bonsai 1-bit (binary) bundles through the same managed model store and runtime surfaces.

DarkNet and the quantization tooling bundled with the package do the heavy inference work. The Python package is the orchestration layer around those binaries.

Install

• Python 3.12 or newer is required.
• Linux wheels target glibc >= 2.27.
• uv is the recommended installer.
• Voice features require the optional voice extra.

Platform guides:

• macOS
• Linux
• Windows

If you install Trillim with uv, prefix CLI commands with uv run.

Quick Start

Install the package:

uv add trillim

Pull a model and chat with it:

uv run trillim pull Trillim/BitNet-TRNQ
uv run trillim chat Trillim/BitNet-TRNQ

Start the local API server:

uv run trillim serve Trillim/BitNet-TRNQ

Use the Python SDK synchronously through Runtime:

from trillim import LLM, Runtime

with Runtime(LLM("Trillim/BitNet-TRNQ")) as runtime:
    reply = runtime.llm.chat(
        [{"role": "user", "content": "Give me one sentence about local CPU inference."}]
    )
    print(reply)

Common Workflows

Pull and Inspect Bundles

trillim models lists bundles published by the Trillim Hugging Face organization. trillim list lists what you already have locally.

uv run trillim models
uv run trillim list

Quantize a Local Model or Adapter

trillim quantize takes raw local filesystem paths and publishes the output under ~/.trillim/models/Local/.

# Quantize a model bundle
uv run trillim quantize /path/to/model

# Quantize a LoRA adapter against its base model
uv run trillim quantize /path/to/base-model /path/to/adapter

Qwen3-based Bonsai checkpoints quantize into binary bundles, but Trillim still manages them under the same Local/...-TRNQ store naming and load flow.

Use an Adapter

chat accepts an optional second positional argument for the adapter store ID:

uv run trillim chat Trillim/BitNet-TRNQ Trillim/BitNet-GenZ-LoRA-TRNQ

Enable Voice Support

Install the extra first:

uv add "trillim[voice]"

Then start the voice-enabled server:

uv run trillim serve Trillim/BitNet-TRNQ --voice

Documentation

Learn

• What Is Trillim?
• Install: macOS, Linux, Windows
• CLI Reference

Extend and Serve

• Python SDK
• API Server

Advanced

• Advanced SDK and Server Notes
• Benchmarks

License

For the short license summary, see What Is Trillim?. Full terms are in LICENSE.