bigrig 0.2.0

Quickstart · Install · Models · CLI · How it works · Measurements · Contributing

About

A 30B MoE model uses only a handful of its experts for any given word, but every runtime keeps all of them in memory. bigrig keeps a bounded set in RAM and reads the rest from SSD, or shrinks the model to fit — your choice, asked before anything changes — while a quality meter watches the output and tells you if the trade went too far.

• Three strategies, chosen for you. Run it untouched, shrink it to fit, or stream it from disk. Picked in increasing order of what each costs you: nothing, then accuracy, then speed.
• It asks before changing your model. Shrinking is the only path that alters weights, so it is the only one that needs your agreement — and every run afterwards still prints which mode it is serving.
• Streaming is bit-exact. Not "close". The same model resident and streamed at a 36.75% cache-miss rate scored identical perplexity to ten decimal places.
• A quality meter, running live. No reference answer, no second forward pass. It separates a healthy model from a damaged one at 0.0% vs 36.7% of tokens flagged.
• Nothing extra on disk. Experts are read straight out of the downloaded safetensors, so preparing a model writes nothing at all.
• Speaks OpenAI and Anthropic. Including a one-command hook into Claude Code, and a web interface for people who would rather not use a terminal.

Getting started

Install

git clone https://github.com/arjvnv/bigrig.git && cd bigrig
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[engine]"

rig and bigrig are the same command. Full details in install.md.

Run a model

rig prepare mlx-community/OLMoE-1B-7B-0125-Instruct-4bit
rig serve   OLMoE-1B-7B-0125-Instruct-4bit

prepare downloads the model and writes nothing else. serve tells you exactly what it is about to serve, then where to find it:

  bigrig serving OLMoE-1B-7B-0125-Instruct-4bit on http://127.0.0.1:8080
  running EXACT at 4-bit, fully in RAM (untouched)
  quality monitor: on

  open  http://127.0.0.1:8080  in a browser to chat and watch quality
  api   127.0.0.1:8080/v1/chat/completions   (OpenAI)
        127.0.0.1:8080/v1/messages           (Anthropic -- Claude Code)
  agent bigrig launch OLMoE-1B-7B-0125-Instruct-4bit

That second line is printed on every run. Whatever mode you are in, the precision being served is never a surprise.

Then pick whichever suits you

A browser — open http://127.0.0.1:8080. A chat box, streaming replies, and along the top: the model, the mode it is running in, memory held, live tokens/second, and a quality indicator that turns amber mid-generation if the model starts looping or losing coherence. One self-contained page, no CDN, nothing to install.

Claude Code — one command, which starts the server and points the agent at it:

rig launch OLMoE-1B-7B-0125-Instruct-4bit

Nothing on disk is changed: the environment variables are set on the agent's process only.

Your own code — both APIs are served on the same port, so any client library works by changing its base URL:

POST /v1/chat/completions      OpenAI
POST /v1/messages              Anthropic

A terminal — rig run <model>.

What happens when a model doesn't fit

bigrig stops and asks, because the two ways forward cost different things:

    [1] Shrink it to fit     full speed, but THE WEIGHTS CHANGE
    [2] Keep it exact        bit-for-bit identical, streamed from disk, slower

Your answer is remembered, and every run still prints which one it is serving. In a script with neither --compress nor --exact, it refuses rather than guessing — quietly serving a degraded model is worse than stopping.

It will also tell you not to use it. If a model already fits, bigrig loads it normally and says so; putting the engine in front of a model that doesn't need it only makes it slower.

Measurements

On an M4 MacBook Air (24 GB). Nothing here is projected.

Shrinking vs streaming, at the same memory (OLMoE-1B-7B, wikitext-2):

Below 3 bits is a cliff, not a bargain — so 3 bits is the floor, and going lower takes an explicit --min-bits 2.

Full write-up, including the negative results: docs/MEASUREMENTS.md.

The quality meter on its own

bigrig_layer is independent of the engine. It watches any model's output distribution and flags degradation with no reference answer and no second forward pass — and it works against Ollama, llama.cpp and the OpenAI API too:

from bigrig_layer import AdaptiveMeter

m = AdaptiveMeter()
for step in generation:
    m.observe(probs)
    m.observe_token(tok)
    if m.is_degraded():
        print("quality problem:", m.reason())

Repository layout

bigrig_engine/     the engine — strategy, streaming, precision, serving, web UI, CLI
bigrig_layer/      the quality meter (standalone, engine-agnostic)
tests/               ./run-tests.sh runs every assertion
docs/                install, quickstart, models, cli, design, measurements
src/  notes/         research scripts and lab notebook — not shipped

See CONTRIBUTING.md for how the tests are written and why.

Requirements

Apple Silicon (M1 or later), macOS, Python 3.10+.

Licence

Apache License 2.0. Use it, modify it, ship it commercially — keep the notice and don't sue us over patents.