fermion-research 0.1.12

Five-value sub-2-bit LLMs: chat with a ~2 GB 8B container at native-runtime speed, load it as a Transformers model, or serve it on an OpenAI-compatible endpoint

fermion

Five-value sub-2-bit language models by Fermion Research. One CLI chats with a ~2 GB 8B container (TRTC v4) or a 2.5 KB QR-code model — and serves either one on an OpenAI-compatible HTTP endpoint (fermion serve — with native OpenAI tool calling, token streaming, and a persistent session runtime that reuses the KV cache across turns), so it drops into Open WebUI, Continue, LangChain, LlamaIndex or any agent harness that speaks /v1/chat/completions.

Tool calling is an 8B capability. serve accepts and injects tool schemas for every SKU, but only Neutrino-8B reliably emits tool_calls. The 0.6B models answer in prose instead — measured, and consistent with our banked finding that function calling has a parameter-count floor. Point agent frameworks at the 8B.

NAME FINAL 2026-07-24: fermion is the final pip/CLI name (rename chain from the pre-launch working slug documented in RELEASE_RUNBOOK.md §5). HF org fermionresearch, GitHub org fermionresearch. Model family = Neutrino (2026-07-24 launch-shape addendum): the CLI's default model is the one published SKU fermionresearch/Neutrino-8B, whose repo also bundles the prebuilt native fermion-run binaries under bin/.

Quickstart (2 commands)

Recent releases in one breath — 0.1.6: batched prefill (4-5x) and download progress. 0.1.7: OpenAI tool calling on serve, persistent session runtime (agent turns up to 72x faster), tolerant tool-call parsing. 0.1.8: multi-conversation KV cache (agent frameworks no longer evict your session) and tool-name aliasing for MCP-style clients. 0.1.9: the 8B now downloads as a 2.56 GB coded transport and unpacks locally (sha-verified against the manifest), and serve streams prose even on tool-calling requests. 0.1.10: fp16 KV cache by default (halves KV memory; --kv-dtype) and experimental YaRN long context (--yarn-factor) on the native runtime. 0.1.11: free-disk preflight before every model download, full-disk errors that name the directory that filled, FERMION_CACHE_DIR to relocate the model cache, and the transport is deleted after its sha-verified decode (8B cache footprint ~6.4 GB → ~3.9 GB). 0.1.12: fermion inspect reports the int8 KV column (it shipped in 0.1.10 but the table only showed fp16/fp32), plus honest scoping on --yarn-factor and on which SKU emits tool calls.

--yarn-factor: leave it off unless you need >40,960 tokens. YaRN extends addressable positions; it does not improve recall. It also applies a global attention-temperature term (0.1·ln(factor)+1) that is not gated by position, so it changes output at every length, short prompts included. Our needle probe measures retrieval falling off well inside the native window with YaRN off entirely, so the ceiling is the model, not the flag.

pip install fermion-research
fermion chat                # downloads fermionresearch/Neutrino-8B, opens REPL

That first run fetches 3.89 GB — the container, the tokenizer, the config and the native runner — and nothing else. The model repo also carries a GGUF build and a coded transport that this CLI never opens; they are filtered out. (FERMION_DOWNLOAD_ALL=1 fetches the whole repo if you want them.)

Disk space: what the quickstart needs, and where

"No space left on device" with hundreds of GB free almost always means the free space is on a different volume than the one being written. Everything the quickstart writes goes to directories on your root/home volume unless you say otherwise:

• Linux: pip install fermion-research pulls torch's CUDA wheel stack — ~8 GB for the install, and pip stages ~3 GB of it in /tmp (often a RAM-backed tmpfs). The 8B model itself is ~4 GB in ~/.cache/huggingface. CPU-only box? Install torch from the CPU index first — ~1 GB instead of ~7.5 GB, and the /tmp trap disappears: pip install torch --index-url https://download.pytorch.org/whl/cpu
• Apple Silicon: the pip install is small, but the first 8B fetch needs ~6.5 GB free in the cache volume during the one-time transport decode (2.6 GB download + 3.9 GB decoded container, both on disk for a moment). Once the decoded container passes its sha256 check the transport is deleted, so the steady-state footprint is ~3.9 GB (FERMION_KEEP_TRANSPORT=1 keeps the transport instead).

Point the caches at the big disk and both traps disappear:

export HF_HOME=/big/disk/hf              # the model cache (the standard HF knob)
export FERMION_CACHE_DIR=/big/disk/fermion   # or: this CLI's model downloads only
export TMPDIR=/big/disk/tmp              # pip's staging area, if `pip install` is what failed

Since 0.1.11 the CLI checks free space in the cache directory before downloading and prints the directory, the bytes needed and the bytes free when the model will not fit; a disk that fills mid-download or mid-decode gets the same named-path message instead of a traceback.

With a local file (no download):

fermion chat --model /path/to/neutrino-8b_v4.bin         # 8B container
fermion chat --model /path/to/qr_chatmax_artifact.bin    # the QR-code model
fermion generate "hello" --model ... --max-new 32        # one-shot
fermion info --model ...                                 # header + integrity check
fermion serve --model ...                                # OpenAI-compatible API

Sampling defaults: chat and serve are sampled, generate is deterministic

fermion chat and fermion serve default to the graded shipping config — temperature 0.01, top-p 1.0, repetition penalty 1.05 over a 256-token window — because that is the configuration the conversational surfaces were graded at. fermion generate defaults to deterministic greedy (temperature 0, no penalty) because it is the scriptable, pipeable path that our receipts, fermion verify and the token-identity gates depend on reproducing. Every knob is a flag on all three, so either default is one argument away.

fermion info — did my download actually work?

info prints the container header and proves the file is the whole file: it checks the length against the record structure the container itself describes, and against the sha256 the model repo's MANIFEST.json pins. It exits non-zero on a truncated or altered container, so it is safe to use in a script:

$ fermion info --model ./Neutrino-8B
./Neutrino-8B/neutrino-8b_v4.bin: TRTC v4 arch=3 layers=36 hidden=4096 vocab=151936 (3.61 GiB)
[fermion] integrity OK: 3,875,404,812 bytes · length matches its own record
          structure · length matches MANIFEST.json · sha256 matches MANIFEST.json

$ fermion info --model ./half-downloaded.bin ; echo $?
./half-downloaded.bin: INTEGRITY CHECK FAILED
  container is truncated: input embedding weights needs bytes up to 622,329,932
  but the file is only 67,108,864 bytes
1

--no-checksum skips the hash (the length checks always run).

fermion serve — OpenAI-compatible endpoint

fermion serve                                  # 127.0.0.1:8000, default model
fermion serve --model /path/to/neutrino-8b_v4.bin --port 8000

POST /v1/chat/completions    messages, temperature, top_p, max_tokens, stop, stream
POST /v1/completions         plain-text completion for older clients
GET  /v1/models              the loaded model id
GET  /health                 ok + the loaded container's sha256

Streaming is real SSE (data: {...}\n\n chunks with choices[].delta, terminated by data: [DONE]); non-streaming returns choices[].message plus a usage block. Errors come back OpenAI-shaped ({"error": {...}}).

Point any OpenAI client at it. Python:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")
print(client.chat.completions.create(
    model="neutrino-8b_v4.bin",
    messages=[{"role": "user", "content": "What is 2+2?"}],
).choices[0].message.content)

Open WebUI — Settings → Connections → OpenAI API:

Base URL: http://127.0.0.1:8000/v1
API key:  not-needed          # any non-empty string

(Open WebUI in Docker: use http://host.docker.internal:8000/v1.)

Continue (~/.continue/config.json):

{"models": [{"title": "Neutrino 8B", "provider": "openai",
             "model": "neutrino-8b_v4.bin", "apiKey": "not-needed",
             "apiBase": "http://127.0.0.1:8000/v1"}]}

curl:

curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"What is 2+2?"}],"max_tokens":32}'

Notes:

• Localhost only by default. The server is unauthenticated; pass --api-key SECRET to require Authorization: Bearer SECRET, and expect a printed warning if you bind a non-loopback --host. --cors adds Access-Control-Allow-Origin: * for browser-side clients (off by default).
• Sampler defaults are the graded shipping config (same as fermion chat, and unlike fermion generate, which stays greedy — see above): temperature 0.01, top-p 1.0, repetition penalty 1.05 over a 256-token window. A client that asks for temperature: 0 gets a deterministic argmax with the repetition penalty still applied on either backend — the C runtime no-ops its sampler flags at temp 0, so the native path emulates argmax on top of a live sampler rather than dropping the penalty. Send the vLLM-style "repetition_penalty": 1.0 extension to turn the penalty off.
• One request at a time (single resident model, serialised behind a lock); n > 1, embeddings and function-calling are not implemented and say so.
• Chat templating is the same code path as fermion chat, and served output is gated token-identical to fermion generate at matched settings.
• --draft PATH attaches a second container as a speculative-decoding draft model (assisted generation; the draft must share the tokenizer).

What backs it

Two decode backends, and the CLI tells you which one you got

Since 0.1.4 the CLI runs the prebuilt native runtime by default.

backend	what it is	when it is used
native	bin/fermion-run-<platform>, the C runtime downloaded with the model — the path every published tokens/s number was measured on	automatically, whenever a binary exists for your platform and --device cpu
torch	the hf_ternary reference loader + transformers.generate	everywhere else, and whenever you ask for it

fermion info --model ...       # prints the active backend and why
fermion generate --backend torch ...   # force the reference path
fermion generate --backend native ...  # fail loudly instead of running slow
FERMION_THREADS=8 fermion chat ...     # override the thread count

fermion serve reports the same thing in GET /health as "backend".

Native runtimes exist for macOS arm64 and Linux x86-64 only. On Windows, Linux arm64, or with --device cuda/mps, you get the torch path: correct, and one to two orders of magnitude slower. The published speed figures are native figures and do not describe the torch path.

Two known differences on the native path, both deliberate and documented in fermion/native.py:

• The C runtime disables its whole sampler at --temp 0, so a greedy request that also carries a repetition penalty is mapped to --temp 0.01 --min-p 0.999 --seed 0 — argmax by construction, penalty intact, still byte-reproducible.
• Greedy output is token-identical to the torch reference at float32, not at the CLI's bfloat16 default, and identity is a near-tie property rather than a guarantee: two independent implementations pick different tokens when the top-2 logits are within measurement noise. Measured agreement and the divergence analysis are in handoffs/results_pip_native/RESULTS.md.

What backs it

• TRTC v4 containers load through the hf_ternary integration (vendored verbatim, sha-recorded): native Qwen3ForCausalLM etc. with packed five-value planes resident (98.9% memory honesty), correctness-gated at 0 greedy mismatches over 768 tokens vs the expander reference.
• The torch path is the honest reference: correct everywhere torch runs, fast nowhere. It is what fermion verify and --draft (speculative decoding is a torch-graph feature) use, and what the native path is gated against. Measured numbers live in the model card and eval-receipts, each with venue+version+date.
• CLI activation dtype defaults to bfloat16 (--dtype to override): fp16 NaN-overflows at 8B scale (measured; receipts in packaging/acceptance/), while --dtype float16 reproduces the fp16-identity receipts on the small twin containers gated that way.
• Tiny models (QR 2.5 KB GRU / GIF 170 KB transformer) run through the float64 reference decoder the browser demo is bit-exactness-gated against.

import fermion
from transformers import AutoModelForCausalLM
fermion.write_transformers_config("neutrino-8b_v4.bin", "cfg-dir")
model = AutoModelForCausalLM.from_pretrained("cfg-dir")   # native Qwen3

Loading in plain transformers

hf download fermionresearch/Neutrino-8B --local-dir Neutrino-8B \
    --exclude "gguf/*" --exclude "*.tv4z"

import fermion          # <-- REQUIRED, and it must come first
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Neutrino-8B")
tokenizer = AutoTokenizer.from_pretrained("Neutrino-8B")

import fermion is what registers the trtc_v4 model type with the Transformers Auto classes. Pass a local directory, not a Hub id.

If you see this error, you forgot the import

ValueError: The checkpoint you are trying to load has model type `trtc_v4`
but Transformers does not recognize this architecture. ... You can update
Transformers with the command `pip install --upgrade transformers`.

Ignore that advice. Upgrading Transformers will never help and neither will installing it from source — trtc_v4 is our model type, registered at import time by this package, so no Transformers release will ever know it. trust_remote_code=True does not help either: these repos carry no auto_map. Add import fermion above the Transformers import and it loads.

Dev

pip wheel --no-deps -w dist .     # build the wheel
python -m venv /tmp/v && /tmp/v/bin/pip install dist/*.whl
/tmp/v/bin/fermion --version

License: Apache-2.0 (flagship is a Qwen3-8B derivative, Apache-2.0 upstream).