justfit-llm 2.3.0

Get started · Results · Documentation · 简体中文

JustFit is a local inference system for serving a 27B language-and-vision model on memory-constrained Macs. It coordinates compressed context, a resident speculative draft, concurrent requests and CPU/GPU/ANE work under one unified-memory budget.

Stable version: 2.3.0. This source tree prepares the stable release. Package-index and release-asset publication are pending; use the reviewed wheel and native runtime supplied with this release candidate. The immutable RC24 preview remains available separately.

Why JustFit

• Keep long histories alongside the draft. Warm-prefix tests reach a 320K aggregate page capacity across two live requests while retaining the speculative draft. This is a tested B2 bound, not a 320K single-request or cold-prefill guarantee.
• Make concurrent work useful. Bound transient workspaces, share compressed pages and preserve committed state across scheduling and cold restarts.
• Put idle accelerators to work. Use CPU-assisted prefill on M4 Pro, GPU A8 paths on M5 Pro/M6, and qualified ANE work when execution phase and memory allow it. Vision offload aims to leave GPU capacity for text; it does not guarantee lower image latency.
• Serve through a familiar API. OpenAI-compatible chat, images, streaming, structured output and native-enforced thinking budgets.

Measured results

Capacity and speed are separate experiments. Solid bars show tested B1 bounds; hatching extends them to the B2 total. Both JustFit generations reach a 320K page capacity using their respective warm-prefix protocols; v2 retains its draft. Speed uses eight fixed SPEED-Bench 16K tasks with 512 forced output tokens. Weight formats, decoders and profiles differ. M4 Pro/M5 Pro have physical 24 GiB; M6 has physical 32 GiB, with 8 GiB locked only for the v1/v2 B2 capacity tests. M6 speed runs use physical 32 GiB.

At 16K, v2's median decode rate is 2.0–2.5× the v1 rate across these configurations. Relative to upstream Splash 1.3.0, median prefill improves 9–46%, while singleton decode declines 2–9%. These whole-system results are not isolated mechanism gains or a chip-generation ranking.

Protocols, exact rates and limitations →

Get started

Supported setup

• Apple silicon and macOS 27.0+. Reference configurations: M4 Pro, M5 Pro and M6, with at least 24 GiB. Other chip variants are not individually qualified.
• Python 3.12 recommended; the package declares support for Python 3.12–3.14.
• Compatible Xcode/CoreAI SDK and Metal compiler toolchain for this source preview. justfit serve finds installed tools; it does not install Xcode or the SDK.
• Internet access and storage for the pinned model pack: approximately 17.4 GB, plus dependencies, build outputs and caches. The model repository is publicly accessible without a Hugging Face token at the checked revision.

Installing the Python package on Linux does not enable native inference there. The standalone GPU engine's macOS 26.4 source target is separate from this complete macOS 27/CoreAI setup.

Install the reviewed release

Create a fresh environment. Install the reviewed wheel downloaded alongside this source tree:

python3.12 -m venv .venv
source .venv/bin/activate

python -m pip install --index-url https://pypi.org/simple/ -r requirements-runtime.txt
python -m pip install ./justfit_llm-2.3.0-py3-none-any.whl
justfit verify
justfit doctor
justfit serve

First run: create a versioned workspace, build the native runtime, download and verify the pinned model, choose the device preset, then serve on http://127.0.0.1:8000. Later runs reuse the build and model. Ctrl+C stops the foreground server. GPU vision works without a compiled ANE vision profile.

If the Metal compiler is missing, install the component with xcodebuild -downloadComponent MetalToolchain after installing compatible Xcode. Step-by-step setup and overrides cover custom ports, local models and prebuilt-runtime inputs. Use the matching 2.3.0 runtime and published SHA256 to skip local compilation; see the linked instructions.

Send a request

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "incoai/Qwen3.8-27B-Splash",
    "messages": [{"role": "user", "content": "What is 17 multiplied by 19?"}],
    "max_tokens": 64,
    "reasoning_effort": "none"
  }'

The model ID retains the upstream runtime pack's name; the engine and CLI are JustFit. The default model is pinned to revision 9d27070b71f7142c6b6025f03ac011d70a73cb48.

How it works

ANE behavior

Vision eligibility requires qualified 4K/8K assets, a matching chip and OS build, and memory admission. 4K/8K count image patches, not text tokens or screen pixels. Idle requests, unsupported inputs and unavailable resources use GPU. M6 reserves ANE for language prefill before admitting vision work. ANE setup and download interface.

Documentation

The default endpoint is local. Network deployment needs appropriate authentication and TLS; see security scope. Native runtime sources are in source/, ANE tooling in vision-tools/, and optional managed-service components in service/.

Paper and citation

JustFit v2: Long-Context Local LLM Serving with Just-in-Time State Management — Yuhua Chen, 2026. Read the preprint (PDF) · Paper and citation files. This dated version remains available as the manuscript evolves; an arXiv link will be added when public.

If JustFit supports your research, please cite:

@unpublished{chen2026justfitv2,
  author = {Chen, Yuhua},
  title = {{JustFit v2}: Long-Context Local LLM Serving with Just-in-Time State Management},
  year = {2026},
  month = oct,
  note = {Preprint, version 2026-10-23 revision 2},
  url = {https://github.com/YuhuaBillChen/justfit-llm/blob/paper-v2-2026-10-23-r2/paper/2026-10-23/JustFit-v2.pdf}
}

Machine-readable citation · JustFit v1 paper

Acknowledgments and license

JustFit's native runtime is derived from Splash 1.3.0. Upstream authors, licenses and implementation credits are preserved in NOTICE and third-party notices. Model weights and derived assets retain their own licenses.

Code is available under Apache 2.0. JustFit maintainers handle this project's support; please direct JustFit issues here rather than to upstream maintainers.