btb (Beyond the Box)

btb is a x86_64 / ARM / CUDA / MLX / NEON optimized harness that prioritizes using all available hardware on the machine, semi-greedily.

Run a 180B+ model without GPU, without needing quantization / GGUF, and without waiting a week! The better your machine's hardware, the faster your inference will be. It will run on a laptop and a gaming PC alike.

It implements a tiered access system, prioritizing VRAM, RAM, and disk in that order. The harness is designed to allow for regular machine usage, such as playing a game, launching a memory-heavy application, et al. While it is greedy, it keeps a configurable buffer of free memory for the OS. When that buffer shrinks, RAM is freed accordingly from inference for other applications.

Notice: This software pushes upwards of 3-4GB/s read from your drive, constantly. I WOULDN'T recommend running this on a HDD, though HDD support was taken into consideration. It however, does attempt to absolutely minimize the number of reads, see the Bus Pass scheduler.

Disclaimer: This project contains AI written code, human reviewed, and experimental.

Features

• 🏡 At-home access to LLM inference for everyone :)
• ☮️ Windows, macOS, Linux
• 🏎️ CUDA / MLX / NEON perf. that meets or beats llama.cpp
• 🌈 Fast(er) CPU inference / No GPU required (sorry NVIDIA)
  • AVX2, AVX-512, NEON support
  • The more cores, the more performance
  • Single-core machines may struggle more due to inherent contention
• 🔍 BF16 first, quant second
  • btb is bf16 first, opting to never sacrifice quality for speed
  • MoE included - Yes, you can even run large MoE models in bf16. It won't be the fastest, but it's better than impossible.
  • No extra disk space required - Run the models you already have
  • GGUF / Quant. support - However, if you already have a .gguf or quantized model, you can use it all the same.
  • 12-bit packing
    • For low memory systems, or large models where you don't want to sacrifice quality; btb offers its own form of lossless packing "pack12".
    • This does require about ~0.75x of the space of the base model in space on the disk, but it opt-in (see the pack subcommand).
• ⛓️ Deterministic inference
  • Deterministic sampling and inference across various devices
  • Blazing performance through some sorcery
  • n-gram and MTP tree optimizations + tuning
  • Support for top-k, top-p, and greedy - all highly optimized
• 📚 Full context windows + automatic YaRN
  • And no, you're not just limited to 256 or something.
  • Scale context windows to the model's max, or higher, with automatic YaRN configuration.
  • Operate at context lengths that definitely don't fit in VRAM :)
  • Conversational prefill / KV caching
• 🛠️ Tools Included:
  • btb serve [model] - OpenAI server
  • btb ollama [model] --gui - Open the Ollama GUI
  • btb pi - Configure a pi provider and start a server (tools ARE supported)
  • btb chat - Basic interactive chat without tools
• 😇 Annoys Sam Altman
• 🐍 A plain Python API

Commercial Use

For commercial use, please contact sales@relta.net. See the LICENSE file for more information about the FSL-v1.1-ALv2 license.

Installing

This package provides both wheel files for common platforms/architectures, as well as a sdist package (though not recommended).

pip

pip install beyondthebox
pip install beyondthebox[mlx]      # Apple silicon: adds MLX, and the GPU is used by default

From Source

To rebuild the library and the wheel from source, with a Rust toolchain:

git clone https://github.com/EvanDarwin/btb && cd btb
python build.py     # this machine, with the card's kernels when nvcc is here; --target <triple> cross-builds
pip install dist/*.whl

See Building btb for the prerequisites (Rust, CUDA/nvcc, MLX), the dev checks and tests, and cross-building.

Usage

OpenAI server

btb serve Qwen/Qwen3-4B --port 8000

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="btb")
for chunk in client.chat.completions.create(model="qwen3-4b", stream=True,
        messages=[{"role": "user", "content": "Why is the sky blue?"}]):
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Every complete model in the Hugging Face cache (and --models-dir) is listed by name and loaded on demand, least recently used unloaded to make room; --models-filter REGEX narrows the list.

Pi Agent

btb pi

This will modify your Pi configuration to add a new btb provider, which you can then launch with pi --provider btb. Tools are supported.

Ollama

# For interactive terminal chat
btb ollama [model]

# For the GUI
btb ollama [model] --gui

Starts the server and runs ollama run qwen3-4b; anything after the options goes to ollama run. --gui opens the Ollama desktop app on it instead (quit the app before restarting btb, it holds the port).

One prompt, or a chat in the terminal

btb run Qwen/Qwen3-4B -q -p "What is the capital of France?"
# The capital of France is **Paris**.

btb run Qwen/Qwen3-4B -q --json -p "What does CPU stand for?"
# {"answer": "CPU stands for **Central Processing Unit**. …", "took": 4.25, "stop": "eos", "stats": {"tokens_per_sec": 13.9, "tokens_in": 18, "tokens_out": 59, "tokens_per_pass": 1.0}}

btb chat Qwen/Qwen3-4B
# Basic interactive chat window

The model path is a local directory or a Hugging Face repo id (downloaded into the cache if missing).

From code

import btb

with btb.load("Qwen/Qwen3-4B") as model:
    print(model.ask("What is the capital of France?"))
    for piece in model.stream("Name three uses of a paperclip.", max_new=64):
        print(piece, end="", flush=True)

Options

These options apply to all commands that accept a model name / path.

btb is designed to run automatically, picking the best configuration for your current hardware setup at runtime. These options are simply if you want to play/experiment or fine-tune placement of weight layers between CPU/GPU.

Every command that loads a model takes these. The placement is planned from the machine's free memory at load; the flags override it.

Each command's own arguments (btb <command> --help lists them):

• run PATH: -p, --prompt (else stdin, else a sample prompt), --file JSONL (one {"prompt": ...} per line, a fresh context each), --new N, --greedy (one token at a time even if the model can speculate), --raw (keep special tokens), -q, --quiet, --json (one record per prompt), --out PATH.
• chat PATH: --file JSONL (scripted turns, one {"user": ...} per line), --new N; /reset clears, /quit exits.
• bench PATH --prompts JSONL: --rows I,J,... (default all), --new N,N,... (default 64,256,1024), --label LABEL, --out JSONL; always greedy.
• serve [PATH]: --host (default 127.0.0.1), --port (default 8000), --new N (a ceiling per request), --api-key KEY (required on every request as Authorization: Bearer KEY; default BTB_API_KEY from the environment, else none), --models-dir DIR (repeatable), --models-filter REGEX; no path starts empty.
• ollama [PATH]: serve's arguments, --port default 11435, --gui (the desktop app instead of the terminal chat); anything unrecognized goes to ollama run.
• pi [PATH]: serve's arguments (--api-key included, written into the provider entry), --no-configure (print the provider entry instead of writing ~/.pi/agent/models.json), --config PATH (write it elsewhere).
• pack PATH [OUT]: the lossless 12-bit model beside the model (<model>-pack12, a model like any other), or at OUT.
• devices: -q (one name per line), --json.

The 12-bit store

Optional and lossless: the same bf16 numbers in 12 bits each, 0.75x the read from disk. Written only by pack, into its own directory; the checkpoint is untouched; runs find the store on their own.

btb pack /models/Qwen3.8-27B              # writes /models/Qwen3.8-27B-pack12
btb pack /models/Qwen3.8-27B /fast/disk/Qwen3.8-27B-pack12

pack checks the free space first and refuses if the store does not fit. It pays when the model reads from disk every token (the 27B on 64 GB: 54 GB a pass from the checkpoint, 40 GB from the store).

Models

Supported families (so far, I'm out of disk space):

• Qwen3.5-style hybrids (qwen3_5)
• Qwen3 dense (qwen3)
• Phi-3-style dense (phi3) and the Qwen4 MoE (qwen4_exp; transformers 5.16 or later)
• OpenAI gpt-oss (gpt_oss): attention sinks, 128-token windows, MXFP4 experts streamed from the drive as stored (the 120B's experts are 61 GB, never expanded)

How?

• I refused to accept quantization
• A 10-week research project into LLM inference
• Iterative optimization
• Avoid reading layers you don't need
• Take lossless shortcuts
• Proper utilization of scheduling between devices
• A tree, determinism, and a touch of magic

Benchmarks (Windows)

All benchmarks use the same bench/questions.json prompts, and bench.py and compare.py scripts - and they are provided for reproducibility.

The following configuration is used:

• No bf16 for CPU runs
• Runs to $64$, $256$, and $1024$ tokens
• tok/s is the rate after the first token, averaged over the prompts
• first_token is the prefill at $256$
• Peak RAM represents the process's peak working set
• Peak VRAM is peak reserved VRAM by the scheduler
• Greedy decoding for now (will do temp. later)

bench/matrix.py runs every device configuration against every complete model in the cache (--devices, --filter, --models), one btb bench process per cell, and writes the table (--markdown) and a JSON of every cell under bench/results/.

Long contexts: Qwen3-4B with the weights on the card and the attention cache in RAM (--kv-host 1), a document followed by two questions about it; --context 131072 past the model's 40,960-token window applies YaRN 4.

Benchmarks (Apple silicon)

Benchmarked on:

• Apple M3 Pro (12 cores, 18-core GPU)
• 36 GB unified memory
• macOS 26.6

The same prompts and answer lengths; peak MLX is the GPU's share of unified memory (part of peak RAM). The Qwen rows are bench/results/m3pro-20260910.json (btb f371e14) and m3pro-20260910-compare.json; the Phi-4-mini and gpt-oss-120b btb rows are the earlier run's.

Long contexts: Qwen3.5-4B (hybrid), a prompt of that many tokens, then decoding.

Tests

python build.py check           # ruff (lint + format), mypy, cargo fmt + clippy: what a change must pass
python build.py test --fast     # tests/test_unit.py: discovery, the 12-bit format, the scheduler, the bench's planning, the native boundary checks (seconds)
python tests/test_receipts.py
python tests/test_mlx.py        # on a Mac: the Metal kernels, and their batch invariance bit for bit

python build.py check needs ruff, mypy, pytest and a Rust toolchain with clippy; python build.py test runs every suite.

The receipts run the engine on seed-0 fixture models (a Qwen3.5 hybrid, a Phi-3, a Qwen3, a Qwen4 MoE; no third-party weights) against banked reference tensors and must print 0.000e+00 on x86-64 (banked by the AVX-512 kernel; scalar and NEON held to 1e-5). On MLX the same receipts are held to 1e-4 with identical tokens.

License

Functional Source License, Version 1.1, ALv2 Future License (FSL-1.1-ALv2). See LICENSE.md.

Copyright 2026 Evan Darwin.