Banach

A trillion parameters on the computer you already own.

Run Mixture-of-Experts models bigger than your RAM, faster than llama.cpp,
with byte-for-byte the same answers. Know the speed before you download a single weight.

Watch the film  ·  Get started  ·  The giants on your machine  ·  Every number

The giants on your machine

The biggest open models in the world, on machines you can buy at a normal shop. Speeds from banach plan, which reads only the model headers. It under-promises: on the model we measured end to end it said 2.1 tok/s, and Banach delivered 5.4 tok/s.

Exact weights in every cell, no lossy shortcuts. Ask about your own machine in ten seconds:

banach probe && banach plan hf:unsloth/Kimi-K2-Instruct-GGUF/UD-Q2_K_XL/Kimi-K2-Instruct-UD-Q2_K_XL-00001-of-00008.gguf

Ran one of these? Post your numbers with the output of banach probe and banach run. Measured results replace the planner's numbers in this table, with your name on them.

See it

$ pip install banach

# one trillion parameters, a 64 GB desktop with one gaming GPU: ask first, it reads 16 MB
$ banach plan hf:unsloth/Kimi-K2-Instruct-GGUF/UD-Q2_K_XL/Kimi-K2-Instruct-UD-Q2_K_XL-00001-of-00008.gguf --hw phase1/machines/ref-64gb-rtx5080.json
| Kimi-K2-Instruct  (deepseek2)                                          |
| 381.9 GB on disk, 384 experts x 60 layers, top-8                       |
| Exact                3.56 tok/s  (range 3.00 to 4.12)                  |

# a 22 GB model on a 16 GB machine, from a cold start
$ banach-engine -m Qwen3.6-35B-A3B-UD-Q4_K_M.gguf -f story.txt -n 256 --budget-mb 10000
banach: prefill 18 tok 3.44 s; decode 256 tok 5.40 tok/s, hit 95.7% (page cache 95.7%)

# and the answers are the exact ones
$ banach-engine -m Qwen3.6-35B-A3B-UD-Q4_K_M.gguf -f story.txt -n 64 --budget-mb 10000 --exact-check
exact-check: PASS byte-identical (64 tokens, logits bit-equal at every step)

How a trillion parameters fit

A Mixture-of-Experts model is huge on disk but small per token: Kimi K2 has a trillion parameters and uses about 32 billion of them for each token. The trick is to have the right 3% in memory at the right moment.

When the file does not fit, plain llama.cpp leaves it to the operating system, which pages bytes in 4 KB at a time with no idea which experts come next. Banach knows the model:

• Plan first. banach plan reads a few MB of headers and works out the speed, the hot set and the split across GPU, RAM and disk before you download anything.
• Stream on purpose. Whole experts, fetched in parallel the moment the router picks them, inside a RAM budget you choose. Experts earn their place by how often they are used.
• Prove it. Every configuration is checked against stock llama.cpp, token for token and logit for logit, in CI on every commit. Optional faster low-precision experts come with a live fidelity meter, so a shortcut is never silent.

Proof: a model bigger than RAM, measured end to end

A 22 GB model on a 16 GB machine. Qwen3.6-35B-A3B (Q4_K_M) on a 4-core cloud VM with no swap and a 1.27 GB/s disk. Cold start, same prompt, same greedy decode.

Squeeze the RAM budget further and race llama.cpp paging experts in at that same budget: 2.1x faster generation. The stock llama.cpp speed rows run with weight repacking off, the only way it runs here. Machine, commit and command for every number: docs/VALIDATION.md.

No approximations. No re-quantizing. No quality loss. The exact weights stay on disk. Banach just moves them better.

Get started in one line

pip install banach

The banach command and its engine, prebuilt for Linux x86_64, Linux arm64 and macOS on Apple Silicon. Windows and GPU builds: from source.

1. See the speed before you download:

   banach plan hf:unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
   

2. Serve it. --lazy starts answering while the file is still downloading:

   banach serve hf:unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --lazy
   

3. Use it from anything that speaks OpenAI or Anthropic, coding agents included:

   from openai import OpenAI
   client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="local")
   print(client.chat.completions.create(
       model="local", messages=[{"role": "user", "content": "Hello!"}]).choices[0].message.content)
   

   import anthropic
   client = anthropic.Anthropic(base_url="http://127.0.0.1:8080", api_key="local")
   print(client.messages.create(
       model="local", max_tokens=200, messages=[{"role": "user", "content": "Hello!"}]).content[0].text)
   

4. Watch it live at http://127.0.0.1:8080/banach/status: engine, fidelity meter, download.

Runs everywhere

Byte-exact against stock llama.cpp in CI on Linux x64, Linux arm64, macOS and Windows; CUDA and Vulkan builds compiled in CI.

Everything in the box

Build from source

git clone https://github.com/gautam1858/banach && cd banach
cargo build --release
git clone https://github.com/ggml-org/llama.cpp ../llama.cpp
(cd ../llama.cpp && git checkout 9309bf1 && git apply ../banach/engine/patches/0001-banach-precision-ladder.patch)
cmake -S engine -B build-engine -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLAMA_DIR=$PWD/../llama.cpp
ninja -C build-engine banach-engine
export BANACH_ENGINE=$PWD/build-engine/banach-engine

• GPU: add -DGGML_CUDA=ON or -DGGML_VULKAN=ON (Metal is on by default on macOS) and serve with --gpu-layers 99.
• Windows: the same steps from a Visual Studio developer prompt; the engine is build-engine\banach-engine.exe.
• A source build is tuned for your exact CPU and is faster still.

Good to know: one request at a time (others queue); the hard RAM budget is Linux, while macOS and Windows give identical output with a softer budget; small models (1 to 3B) call tools fine but are weak coding agents.

Join in

• Star the repo so more people find it.
• Post your numbers. Every machine and model measured makes the planner sharper for everyone. Open a benchmark issue.
• Fund the hardware. The trillion-parameter runs need big NVMe arrays and GPUs:

Why "Banach"?

The Banach-Tarski paradox: cut one ball into a handful of pieces, reassemble them, and you have two balls the same size as the first. More model than your machine should hold.

Credits

Built on llama.cpp and ggml (MIT), which run every kernel. Shaped by the llama.cpp expert-streaming pull requests (25294, 31469) and MoE cache (29887), by colibri's lossless NVMe streaming and published measurements, and by research on expert locality (Si et al., Liang et al.), mixed-precision experts (SliceMoE, DyMoE, HOBBIT) and betting confidence sequences (Waudby-Smith and Ramdas).

License

Apache-2.0, see LICENSE.