Banach

Run AI models bigger than your RAM. Faster than llama.cpp. Byte-for-byte the same answers.

Watch the film  |  Install  |  60-second start  |  Benchmarks

A 22 GB model on a 16 GB machine

Qwen3.6-35B-A3B (Q4_K_M, 22.1 GB) on a 4-core cloud VM with 16 GB of RAM, no swap and a 1.27 GB/s disk. Cold start, same prompt, same greedy decode.

Squeeze the RAM budget further and race llama.cpp paging experts in at that same budget: 2.1x faster generation (details).

No approximations, no re-quantizing, no quality loss. The exact weights stay on disk; Banach just moves them better.

How

A Mixture-of-Experts model only uses a few of its experts per token. When the file does not fit in memory, plain llama.cpp leaves it to the operating system, which pages bytes in 4 KB at a time with no idea which experts come next. Banach knows the model:

• It plans before you download. banach plan hf:... reads a few MB of headers and tells you the speed to expect on your machine and how the model splits across GPU, RAM and disk.
• It streams experts on purpose. Whole experts, fetched in parallel the moment the router picks them, held in a RAM budget you choose, with the most-used experts kept hot.
• It proves it. Every configuration is checked against stock llama.cpp token for token. Optional faster low-precision experts come with a live fidelity meter, so a shortcut is never silent.

The name comes from the Banach-Tarski paradox: one ball, cut into pieces and reassembled, becomes two. More model than your machine should hold.

Install

pip install banach

That is the whole install: the banach command and its engine, prebuilt for Linux x86_64, Linux arm64 and macOS on Apple Silicon. Windows and GPU builds: from source.

60-second start

1. See the speed before you download. This reads a few MB of headers, not 22 GB:

   banach plan hf:unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
   

2. Serve it with an OpenAI and Anthropic compatible API. --lazy starts answering while the file is still downloading:

   banach serve hf:unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --lazy
   

3. Use it from anything that speaks OpenAI or Anthropic, coding agents included (tool use works on both APIs):

   from openai import OpenAI
   client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="local")
   print(client.chat.completions.create(
       model="local", messages=[{"role": "user", "content": "Hello!"}]).choices[0].message.content)
   

   import anthropic
   client = anthropic.Anthropic(base_url="http://127.0.0.1:8080", api_key="local")
   print(client.messages.create(
       model="local", max_tokens=200, messages=[{"role": "user", "content": "Hello!"}]).content[0].text)
   

4. Watch it at http://127.0.0.1:8080/banach/status: engine, fidelity meter, download.

Runs everywhere

Byte-exact against stock llama.cpp in CI on every commit, on Linux x64, Linux arm64 and Windows; CUDA and Vulkan builds are compiled in CI too.

Features

Commands

Build from source

git clone https://github.com/gautam1858/banach && cd banach
cargo build --release
git clone https://github.com/ggml-org/llama.cpp ../llama.cpp
(cd ../llama.cpp && git checkout 9309bf1 && git apply ../banach/engine/patches/0001-banach-precision-ladder.patch)
cmake -S engine -B build-engine -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLAMA_DIR=$PWD/../llama.cpp
ninja -C build-engine banach-engine
export BANACH_ENGINE=$PWD/build-engine/banach-engine

• GPU: add -DGGML_CUDA=ON or -DGGML_VULKAN=ON to the cmake line (Metal is on by default on macOS) and serve with --gpu-layers 99.
• Windows: the same steps from a Visual Studio developer prompt; the engine is build-engine\banach-engine.exe.
• A source build is tuned for your exact CPU and is faster still than the prebuilt binaries.

Good to know

• One request at a time; concurrent requests queue.
• The hard RAM budget is Linux; macOS and Windows give identical output with a softer budget.
• Small models (1 to 3B) call tools fine but are weak coding agents; use a bigger model for agent work.

Every number in this README, with machine, model, commit and command: docs/VALIDATION.md.

Support

Banach is free and open source. If it saves you a hardware upgrade, help fund test machines and GPUs for the next round of benchmarks:

Credits

Built on llama.cpp and ggml (MIT), which run every kernel. Shaped by the llama.cpp expert-streaming pull requests (25294, 31469) and MoE cache (29887), by colibri's lossless NVMe streaming and published measurements, and by research on expert locality (Si et al., Liang et al.), mixed-precision experts (SliceMoE, DyMoE, HOBBIT) and betting confidence sequences (Waudby-Smith and Ramdas).

License

Apache-2.0, see LICENSE.