justfit-llm 2.3.1

Get started · Results · Documentation · 简体中文

JustFit is a local inference system for serving a 27B language-and-vision model on memory-constrained Macs. It coordinates compressed context, a resident speculative draft, concurrent requests and CPU/GPU/ANE work under one unified-memory budget.

2.3.1 is available on GitHub Releases with source, Python packages and an Apple-notarized native runtime; the six ANE bundles of 2.3.0 apply unchanged. Install from PyPI using the instructions below.

Why JustFit

• Keep long histories alongside the draft. Warm-prefix tests reach a 320K aggregate page capacity across two live requests while retaining the speculative draft. This is a tested B2 bound, not a 320K single-request or cold-prefill guarantee.
• Make concurrent work useful. Bound transient workspaces, share compressed pages and preserve committed state across scheduling and cold restarts.
• Put idle accelerators to work. Use CPU-assisted prefill on M4 Pro, GPU A8 paths on M5 Pro/M6, and qualified ANE work when execution phase and memory allow it. Vision offload aims to leave GPU capacity for text; it does not guarantee lower image latency.
• Serve through a familiar API. OpenAI-compatible chat, images, streaming, structured output and native-enforced thinking budgets.

Measured results

Capacity and speed are separate experiments. Solid bars show tested B1 bounds; hatching extends them to the B2 total. Both JustFit generations reach a 320K page capacity using their respective warm-prefix protocols; v2 retains its draft. Speed uses eight fixed SPEED-Bench 16K tasks with 512 forced output tokens. Weight formats, decoders and profiles differ. M4 Pro/M5 Pro have physical 24 GiB; M6 has physical 32 GiB, with 8 GiB locked only for the v1/v2 B2 capacity tests. M6 speed runs use physical 32 GiB.

At 16K, v2's median decode rate is 2.0–2.5× the v1 rate across these configurations. Relative to upstream Splash 1.3.0, median prefill improves 9–46%, while singleton decode declines 2–9%. These whole-system results are not isolated mechanism gains or a chip-generation ranking.

Protocols, exact rates and limitations →

Get started

Supported setup

• Apple silicon and macOS 27.0+. Reference configurations: M4 Pro, M5 Pro and M6, with at least 24 GiB. Other chip variants are not individually qualified.
• Python 3.12 recommended; the package declares support for Python 3.12–3.14.
• The prebuilt installation below requires no local compilation. Building from source separately requires a compatible Xcode/CoreAI SDK and Metal compiler toolchain.
• Internet access and storage for the pinned model pack: approximately 17.4 GB, plus dependencies, build outputs and caches. The model repository is publicly accessible without a Hugging Face token at the checked revision.

Installing the Python package on Linux does not enable native inference there. The standalone GPU engine's macOS 26.4 source target is separate from this complete macOS 27/CoreAI setup.

Install the reviewed release

Create a fresh environment and install the pinned release from PyPI:

python3.12 -m venv .venv
source .venv/bin/activate

python -m pip install justfit-llm==2.3.1
justfit verify
justfit doctor
justfit serve \
  --runtime-manifest https://github.com/YuhuaBillChen/justfit-llm/releases/download/v2.3.1/runtime-manifest.json \
  --runtime-sha256 <runtime-manifest.json SHA256 from the release SHA256SUMS> \
  --ane-manifest https://github.com/YuhuaBillChen/justfit-llm/releases/download/v2.3.0/ane-index.json \
  --ane-manifest-sha256 fd1b71cfa9d4e785741dc2e0ba4d43f2a4dd556a62c0a61f37838b7f9e3ec99d

First run with these options downloads and verifies the prebuilt runtime, matching ANE assets and pinned model, then serves on http://127.0.0.1:8000. ANE bundles require an exact chip and macOS build 26A434 match; on other builds, omit the two ANE options to use GPU vision. Later runs of justfit serve reuse installed components. Ctrl+C stops the foreground server.

A first run without runtime options or an installed runtime uses the source-build path and requires developer tools. Quickstart covers source builds, ports and models.

Send a request

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "incoai/Qwen3.8-27B-Splash",
    "messages": [{"role": "user", "content": "What is 17 multiplied by 19?"}],
    "max_tokens": 64,
    "reasoning_effort": "none"
  }'

The model ID retains the upstream runtime pack's name; the engine and CLI are JustFit. The default model is pinned to revision 9d27070b71f7142c6b6025f03ac011d70a73cb48.

How it works

ANE behavior

Vision eligibility requires qualified 4K/8K assets, a matching chip and OS build, and memory admission. 4K/8K count image patches, not text tokens or screen pixels. Idle requests, unsupported inputs and unavailable resources use GPU. M6 reserves ANE for language prefill before admitting vision work. ANE setup and download interface.

Documentation

The default endpoint is local. Network deployment needs appropriate authentication and TLS; see security scope. Native runtime sources are in source/, ANE tooling in vision-tools/, and optional managed-service components in service/.

Paper and citation

JustFit v2: Long-Context Local LLM Serving with Just-in-Time State Management — Yuhua Chen, 2026. Read the preprint (PDF) · Paper and citation files. This link always points to the current manuscript; an arXiv link will be added when public.

Full-set AIME 2026: on M5 Pro, the same four-bit target with TQ4 KV scores 86/90 (95.6%) with A8 off and 85/90 (94.4%) with A8 on, covering all 30 questions at three seeds. Each arm has three capped generations included in the denominator. This tests A8 within the quantized configuration.

If JustFit supports your research, please cite:

@unpublished{chen2026justfitv2,
  author = {Chen, Yuhua},
  title = {{JustFit v2}: Long-Context Local LLM Serving with Just-in-Time State Management},
  year = {2026},
  month = oct,
  note = {Preprint},
  url = {https://github.com/YuhuaBillChen/justfit-llm/blob/main/paper/JustFit-v2.pdf}
}

Machine-readable citation · JustFit v1 paper

Acknowledgments and license

JustFit's native runtime is derived from Splash 1.3.0. Splash already provides language ANE FFN, paged KV tiering and composite-state persistence; JustFit extends their integration, retention and resource coordination. Upstream authors, licenses and implementation credits are preserved in NOTICE and third-party notices. Model weights and derived assets retain their own licenses.

Code is available under Apache 2.0. JustFit maintainers handle this project's support; please direct JustFit issues here rather than to upstream maintainers.