boyle

Archived, October 2026. boyle is no longer maintained. Its expert-offload runtime now lives in oMLX; to run a mixture-of-experts model larger than memory on Apple silicon, use oMLX's MoE expert offload. This repository remains as the measurement record: what each memory budget bought on real hardware, how well the speed forecasts held, and the research report behind both.

boyle ran mixture-of-experts models inside a declared memory budget on Apple silicon, keeping a fraction of each layer's experts resident and streaming the rest from the checkpoint, with decode outputs bit-identical to the fully resident model. It also forecast decode speed for a budget from checkpoint headers alone, before any weights were downloaded. The name is from Boyle's law (PV = k): memory is traded for speed at a measured rate.

Where each part went

The last release

The final release, 0.2.1, still installs and runs; no further fixes will follow.

uv tool install boyle==0.2.1
boyle predict mlx-community/Qwen3.5-397B-A17B-4bit --budget 90GB   # headers only, nothing downloaded
boyle bench   mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit --budget 12GB

Supported models and how each was qualified are listed in COMPATIBILITY.md. Qwen3.8-Flash-Next needs mlx-lm with qwen4_exp (PR #1788).

What a budget bought, measured

Two machines: an M5 Max (128 GB) and a 2021 M1 Pro (32 GB).

Decode is bit-identical to the resident model at every budget (asserted token by token in the test suite); over-capacity prefill is rounding-equivalent (the same math, batched differently). Accuracy is therefore a property of the model, not the budget: the 397B at 90 GB scored 0.96 on gsm8k (n=100). Loads take seconds because expert layers are wrapped before weights materialize; the first request then pays the cold expert fill (~35 s on the 397B).

Qwen3.8-Flash-Next across budgets

Flash-Next spends 32 GB of its 111.5 GB checkpoint on a hashed n-gram embedding table that is read 16 rows per token. boyle streamed that table by row, so the table, not expert I/O, sets the decode floor, and shrinking the expert budget cost nothing until the experts became very tight (bench, M5 Max 128 GB):

With a routing curve captured by boyle trace and one bench anchor, every budget above fell inside predict's band, 24 GB included.

How well the forecasts held

predict output for the 397B row above:

$ boyle predict mlx-community/Qwen3.5-397B-A17B-4bit --budget 90GB --max-context 16384
boyle predict — mlx-community/Qwen3.5-397B-A17B-4bit
  budget 90.00 GB: FITS (fraction 0.36, slots 77.29 GB, resident 6.43 GB)
  decode ~9.5 tok/s (band 7.6–11.9) — expert hit rate ~83% [qwen3_5_moe curve, measured]
  first request after load: up to ~37 s (cold expert fill; load itself is seconds)
  context: 16384 guaranteed at this budget (headroom to ~18566)
  disk: 223.86 GB checkpoint
  accuracy [measured]: gsm8k (answer mode) = 0.96 (n=100)

Out of sample, the forecast landed at 12.5 vs 11.7 tok/s measured (235B, 70 GB) and 14.7 vs 14.5 (30B on an M1 Pro never measured before; 19.3 vs 17.5 at 20 GB, in band). The known miss is the 397B: 7.2 tok/s through the HTTP server against a band of 7.6–11.9, because the model does not include serving overhead. Trace replay predicted live hit rates within 1–3 points on four configurations, and routing curves were identical at 4-bit and 8-bit. predict never forecast accuracy; its accuracy line is a lookup into measured rows.

Limits that were measured

• Single stream. Diverse-prompt batching is drive-bound at ~9.5 tok/s aggregate regardless of batch size.
• A sync per MoE layer per token (~50 ms/token at 397B scale), because the router's output decides which weights must be present. Polling, event tricks and speculative decoding were measured and did not beat it.
• Small-expert models (records under ~2 MB, e.g. OLMoE) are bound by per-read latency; forecasts there are upper bounds.
• 4-bit was the best measured trade-off; the drop to 3-bit is severe on some tasks.

Why speed was forecastable

Expert routing is flat: across three model families there is no hot set, and LFU loses to LRU at every budget. Speed is then a function of two numbers, the resident fraction (through one hit-rate curve per family) and bytes per miss, which is why a forecast from checkpoint headers plus a short disk probe landed within a ±25% band. The full record, including every lever tried and every dead end, is in docs/report.md.

Lineage

The runtime descends from the expert-offload patch developed for oMLX (PR #2595, Apache-2.0; see NOTICE), by way of a measurement program whose adopted levers this package shipped. Related upstream work: mlx PR #4249 (GPU-visible mmap weights, closed), mlx issue #2878.

License

Apache-2.0. Portions derive from oMLX; see NOTICE.