fermion

Low-bit models by Fermion Research, and one CLI that runs them. Two families: Neutrino, five-value sub-2-bit language models, and Phonon, a speech recognition model for Apple silicon. Both are pulled and run through the same commands, and both serve on an OpenAI-compatible HTTP endpoint (fermion serve — with native OpenAI tool calling, token streaming, and a persistent session runtime that reuses the KV cache across turns), so it drops into Open WebUI, Continue, LangChain, LlamaIndex or any agent harness that speaks /v1/chat/completions.

Tool calling is an 8B capability. serve accepts and injects tool schemas for every SKU, but only Neutrino-8B reliably emits tool_calls. The 0.6B models answer in prose instead — measured, and consistent with our banked finding that function calling has a parameter-count floor. Point agent frameworks at the 8B.

NAME FINAL 2026-07-24: fermion is the final pip/CLI name (rename chain from the pre-launch working slug documented in RELEASE_RUNBOOK.md §5). HF org fermionresearch, GitHub org fermionresearch. Model family = Neutrino (2026-07-24 launch-shape addendum): the CLI's default model is the one published SKU fermionresearch/Neutrino-8B, whose repo also bundles the prebuilt native fermion-run binaries under bin/.

Speech: Phonon

fermion transcribe turns audio into text, fermion listen dictates live in the terminal, and fermion serve exposes an OpenAI-compatible /v1/audio/transcriptions endpoint plus a WebSocket stream for live audio. The models are the Phonon-1 family, Apache 2.0.

Phonon-1 is the default: a 415 MB download measuring 2.640 % word error on LibriSpeech test-clean and 5.699 % on test-other, on the full test sets. Across five real-world benchmarks (AMI, Earnings-22, GigaSpeech, SPGISpeech, TED-LIUM) no downloadable model we could find is both smaller and more accurate. It decodes at a median 23.9x realtime on a base M5 MacBook Air, and the decoder was trained at 2.4 bits per weight from the start. Full benchmark table and protocol are on the model card.

Transcribe a file

pip install fermion-research
pip install mlx mlx-audio mlx-lm soundfile scipy zstandard
fermion models
fermion transcribe meeting.wav

The second line is the speech runtime — a small MLX stack, not a dependency of this package, required only on Apple silicon and never installed automatically, so Linux and Intel installs are unaffected. fermion models lists every model the lab publishes and marks the ones already on the machine; it reads no network.

The first speech command fetches the model, checks the archive against a SHA-256 pinned inside this package before unpacking, and checks every file against the manifest before writing it. A failed checksum aborts rather than installing. Later runs reuse the unpacked model and fetch nothing.

Listen live at the microphone

fermion listen captures the microphone and prints the transcript while you speak: a live partial hypothesis updates one terminal line, and each phrase is printed permanently once you pause. Ctrl-C stops and prints the full transcript to stdout — everything else goes to stderr, the same pipe discipline as transcribe, so fermion listen > note.txt captures exactly the words.

fermion listen
fermion listen --wav meeting.wav   # the same live path, fed from a file

--wav streams a file through the identical streaming stack, paced to real time; it exists so the live path is testable without a microphone. Decoding is deterministic and uses transcribe's exact configuration; --model takes the same values as transcribe. Microphone capture uses sounddevice, which the install line above already provides (it ships with mlx-audio).

Three builds, one family

Select one with --model:

fermion transcribe meeting.wav --model FermionResearch/Phonon-1-Big

The full evaluation — per-benchmark tables, noise ladder, protocol — is published at https://fermionresearch.com/research/phonon-1 and on each card.

Serve speech on an OpenAI-compatible endpoint

fermion serve mounts the audio transcription route when the model it is given is a speech model, so existing OpenAI clients work unchanged against a local server.

fermion serve --model FermionResearch/Phonon-1

curl http://127.0.0.1:8000/v1/audio/transcriptions \
  -F file=@meeting.wav \
  -F model=FermionResearch/Phonon-1

The same endpoint works from the openai Python client by pointing base_url at the local server. response_format accepts json (default), text and verbose_json. Transcription is deterministic: it decodes at temperature 0, and the request carries no sampler settings. A speech model mounts the transcription route and not the chat routes, and vice versa; each server reports what it mounts on GET / and GET /health.

Streaming

A speech server also mounts GET /v1/audio/stream, a WebSocket endpoint for live dictation — the same rolling-partial engine behind fermion listen, over a socket, so anything (an editor plugin, a web page, another machine on your LAN) can stream a microphone at it. Pick the build with --model, exactly as everywhere else: fermion serve --model phonon serves Phonon-1 (the default), --model FermionResearch/Phonon-1-Micro the smallest, --model FermionResearch/Phonon-1-Big the largest.

The protocol, in ten lines:

1. Open a WebSocket to ws://127.0.0.1:8000/v1/audio/stream.
2. Send one JSON text frame: {"sample_rate": 16000, "format": "pcm_f32le"} — both fields optional; pcm_s16le is also accepted. Audio must be 16 kHz mono.
3. Send binary frames of raw audio, any chunking (≈0.05–0.5 s works well).
4. Receive {"type":"partial","text":...} — the current whole hypothesis, each one replacing the last — while a phrase is in flight.
5. Receive {"type":"final","text":...,"segment":N} (N from 1) when a pause (~0.7 s of quiet) or the 30 s segment cap closes a phrase.
6. Send {"type":"end"} (or just close) when done.
7. Receive {"type":"done","text":...} — the full transcript, finals joined with single spaces: byte-identical to fermion transcribe on the same audio for a single-utterance clip.
8. Any refusal arrives in-band as {"type":"error","message":...} followed by a clean close.
9. One stream at a time. Decoding runs on one Metal queue, so a second concurrent stream is refused immediately with an error frame and a clean close — it is never queued behind the first. Retry after the active stream ends.
10. With --api-key, send Authorization: Bearer <key> — or ?api_key=<key> in the URL, for browser WebSocket() clients that cannot set headers.

A complete client in standard-library Python (no websockets dependency — the server end is hand-rolled RFC 6455 and this is its mirror):

import base64, json, os, socket, struct, sys, threading

HOST, PORT = "127.0.0.1", 8000

def send_frame(sock, opcode, payload):          # client frames are masked
    n, head = len(payload), bytes([0x80 | opcode])
    if n < 126:     head += bytes([0x80 | n])
    elif n < 65536: head += bytes([0x80 | 126]) + struct.pack(">H", n)
    else:           head += bytes([0x80 | 127]) + struct.pack(">Q", n)
    mask = os.urandom(4)
    sock.sendall(head + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(payload)))

def read_frame(f):                              # server frames are unmasked
    b1, b2 = f.read(2)
    n = b2 & 0x7F
    if n == 126:   (n,) = struct.unpack(">H", f.read(2))
    elif n == 127: (n,) = struct.unpack(">Q", f.read(8))
    return b1 & 0x0F, f.read(n)

sock = socket.create_connection((HOST, PORT))
f = sock.makefile("rb")
sock.sendall((f"GET /v1/audio/stream HTTP/1.1\r\nHost: {HOST}:{PORT}\r\n"
              "Upgrade: websocket\r\nConnection: Upgrade\r\n"
              f"Sec-WebSocket-Key: {base64.b64encode(os.urandom(16)).decode()}\r\n"
              "Sec-WebSocket-Version: 13\r\n\r\n").encode())
while f.readline() not in (b"\r\n", b""):       # skip the 101 response
    pass
send_frame(sock, 0x1, b'{"sample_rate": 16000, "format": "pcm_f32le"}')

def hear():                                     # print partials/finals/done
    while True:
        op, payload = read_frame(f)
        if op == 0x8: return                    # server closed
        msg = json.loads(payload)
        print(msg["type"], msg.get("text") or msg.get("message", ""))
        if msg["type"] in ("done", "error"): return
listener = threading.Thread(target=hear); listener.start()

import soundfile as sf                          # feed a file, 0.5 s at a time
audio, sr = sf.read(sys.argv[1], dtype="float32")
assert sr == 16000, "resample to 16 kHz mono first"
for i in range(0, len(audio), 8000):
    send_frame(sock, 0x2, audio[i:i + 8000].tobytes())
send_frame(sock, 0x1, b'{"type":"end"}')
listener.join()

Swap the file loop for a sounddevice input stream and it is live dictation. The decode configuration is transcribe's, untouched; partial cadence is the Mac app's (first at ~0.35 s of speech, then every ~0.5 s).

Quickstart (2 commands)

Recent releases in one breath — 0.1.6: batched prefill (4-5x) and download progress. 0.1.7: OpenAI tool calling on serve, persistent session runtime (agent turns up to 72x faster), tolerant tool-call parsing. 0.1.8: multi-conversation KV cache (agent frameworks no longer evict your session) and tool-name aliasing for MCP-style clients. 0.1.9: the 8B now downloads as a 2.56 GB coded transport and unpacks locally (sha-verified against the manifest), and serve streams prose even on tool-calling requests. 0.1.10: fp16 KV cache by default (halves KV memory; --kv-dtype) and experimental YaRN long context (--yarn-factor) on the native runtime. 0.1.11: free-disk preflight before every model download, full-disk errors that name the directory that filled, FERMION_CACHE_DIR to relocate the model cache, and the transport is deleted after its sha-verified decode (8B cache footprint ~6.4 GB → ~3.9 GB). 0.1.12: fermion inspect reports the int8 KV column (it shipped in 0.1.10 but the table only showed fp16/fp32), plus honest scoping on --yarn-factor and on which SKU emits tool calls. 0.1.13: honest device scoping on --draft — the token-identical guarantee is CPU/greedy, and on CUDA bf16 near-tie positions can flip (measured: 28 divergences in 232 tokens on an L4) with no speedup on that path, so the CLI now states both. New in 0.1.14: the native runner drafts in C (--draft passes straight through — no torch model is loaded), and --draft auto fetches the canonical distilled draft instead of requiring a path.

--yarn-factor: leave it off unless you need >40,960 tokens. YaRN extends addressable positions; it does not improve recall. It also applies a global attention-temperature term (0.1·ln(factor)+1) that is not gated by position, so it changes output at every length, short prompts included. Our needle probe measures retrieval falling off well inside the native window with YaRN off entirely, so the ceiling is the model, not the flag.

pip install fermion-research
fermion chat                # downloads fermionresearch/Neutrino-8B, opens REPL

That first run fetches 3.89 GB — the container, the tokenizer, the config and the native runner — and nothing else. The model repo also carries a GGUF build and a coded transport that this CLI never opens; they are filtered out. (FERMION_DOWNLOAD_ALL=1 fetches the whole repo if you want them.)

Disk space: what the quickstart needs, and where

"No space left on device" with hundreds of GB free almost always means the free space is on a different volume than the one being written. Everything the quickstart writes goes to directories on your root/home volume unless you say otherwise:

• Linux: pip install fermion-research pulls torch's CUDA wheel stack — ~8 GB for the install, and pip stages ~3 GB of it in /tmp (often a RAM-backed tmpfs). The 8B model itself is ~4 GB in ~/.cache/huggingface. CPU-only box? Install torch from the CPU index first — ~1 GB instead of ~7.5 GB, and the /tmp trap disappears: pip install torch --index-url https://download.pytorch.org/whl/cpu
• Apple Silicon: the pip install is small, but the first 8B fetch needs ~6.5 GB free in the cache volume during the one-time transport decode (2.6 GB download + 3.9 GB decoded container, both on disk for a moment). Once the decoded container passes its sha256 check the transport is deleted, so the steady-state footprint is ~3.9 GB (FERMION_KEEP_TRANSPORT=1 keeps the transport instead).

Point the caches at the big disk and both traps disappear:

export HF_HOME=/big/disk/hf              # the model cache (the standard HF knob)
export FERMION_CACHE_DIR=/big/disk/fermion   # or: this CLI's model downloads only
export TMPDIR=/big/disk/tmp              # pip's staging area, if `pip install` is what failed

Since 0.1.11 the CLI checks free space in the cache directory before downloading and prints the directory, the bytes needed and the bytes free when the model will not fit; a disk that fills mid-download or mid-decode gets the same named-path message instead of a traceback.

With a local file (no download):

fermion chat --model /path/to/neutrino-8b_v4.bin         # 8B container
fermion chat --model /path/to/qr_chatmax_artifact.bin    # the QR-code model
fermion generate "hello" --model ... --max-new 32        # one-shot
fermion info --model ...                                 # header + integrity check
fermion serve --model ...                                # OpenAI-compatible API

Sampling defaults: chat and serve are sampled, generate is deterministic

fermion chat and fermion serve default to the graded shipping config — temperature 0.01, top-p 1.0, repetition penalty 1.05 over a 256-token window — because that is the configuration the conversational surfaces were graded at. fermion generate defaults to deterministic greedy (temperature 0, no penalty) because it is the scriptable, pipeable path that our receipts, fermion verify and the token-identity gates depend on reproducing. Every knob is a flag on all three, so either default is one argument away.

fermion info — did my download actually work?

info prints the container header and proves the file is the whole file: it checks the length against the record structure the container itself describes, and against the sha256 the model repo's MANIFEST.json pins. It exits non-zero on a truncated or altered container, so it is safe to use in a script:

$ fermion info --model ./Neutrino-8B
./Neutrino-8B/neutrino-8b_v4.bin: TRTC v4 arch=3 layers=36 hidden=4096 vocab=151936 (3.61 GiB)
[fermion] integrity OK: 3,875,404,812 bytes · length matches its own record
          structure · length matches MANIFEST.json · sha256 matches MANIFEST.json

$ fermion info --model ./half-downloaded.bin ; echo $?
./half-downloaded.bin: INTEGRITY CHECK FAILED
  container is truncated: input embedding weights needs bytes up to 622,329,932
  but the file is only 67,108,864 bytes
1

--no-checksum skips the hash (the length checks always run).

fermion serve — OpenAI-compatible endpoint

fermion serve                                  # 127.0.0.1:8000, default model
fermion serve --model /path/to/neutrino-8b_v4.bin --port 8000

POST /v1/chat/completions    messages, temperature, top_p, max_tokens, stop, stream
POST /v1/completions         plain-text completion for older clients
GET  /v1/models              the loaded model id
GET  /health                 ok + the loaded container's sha256

Streaming is real SSE (data: {...}\n\n chunks with choices[].delta, terminated by data: [DONE]); non-streaming returns choices[].message plus a usage block. Errors come back OpenAI-shaped ({"error": {...}}).

Point any OpenAI client at it. Python:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")
print(client.chat.completions.create(
    model="neutrino-8b_v4.bin",
    messages=[{"role": "user", "content": "What is 2+2?"}],
).choices[0].message.content)

Open WebUI — Settings → Connections → OpenAI API:

Base URL: http://127.0.0.1:8000/v1
API key:  not-needed          # any non-empty string

(Open WebUI in Docker: use http://host.docker.internal:8000/v1.)

Continue (~/.continue/config.json):

{"models": [{"title": "Neutrino 8B", "provider": "openai",
             "model": "neutrino-8b_v4.bin", "apiKey": "not-needed",
             "apiBase": "http://127.0.0.1:8000/v1"}]}

curl:

curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"What is 2+2?"}],"max_tokens":32}'

Notes:

• Localhost only by default. The server is unauthenticated; pass --api-key SECRET to require Authorization: Bearer SECRET, and expect a printed warning if you bind a non-loopback --host. --cors adds Access-Control-Allow-Origin: * for browser-side clients (off by default).

• Sampler defaults are the graded shipping config (same as fermion chat, and unlike fermion generate, which stays greedy — see above): temperature 0.01, top-p 1.0, repetition penalty 1.05 over a 256-token window. A client that asks for temperature: 0 gets a deterministic argmax with the repetition penalty still applied on either backend — the C runtime no-ops its sampler flags at temp 0, so the native path emulates argmax on top of a live sampler rather than dropping the penalty. Send the vLLM-style "repetition_penalty": 1.0 extension to turn the penalty off.

• One request at a time (single resident model, serialised behind a lock); n > 1, embeddings and function-calling are not implemented and say so.

• Chat templating is the same code path as fermion chat, and served output is gated token-identical to fermion generate at matched settings.

• --draft PATH attaches a second container as a speculative-decoding draft model (assisted generation; the draft must share the tokenizer). The recommended draft for Neutrino-8B is the Neutrino-0.6B container (its weights are distilled for drafting as of 2026-07-31 — see that repo's changelog). Easiest is --draft auto (0.1.14), which fetches it for you:

  fermion serve --model /path/to/neutrino-8b_v4.bin --draft auto
  

  or download it explicitly:

  hf download FermionResearch/Neutrino-0.6B neutrino-0.6b_v4.bin --local-dir .
  fermion serve --model /path/to/neutrino-8b_v4.bin --draft neutrino-0.6b_v4.bin
  

  It is a distilled draft specialist (a draft, not a chat model); its acceptance table and receipts are on the Neutrino-8B card.

What backs it

Two decode backends, and the CLI tells you which one you got

Since 0.1.4 the CLI runs the prebuilt native runtime by default.

fermion info --model ...       # prints the active backend and why
fermion generate --backend torch ...   # force the reference path
fermion generate --backend native ...  # fail loudly instead of running slow
FERMION_THREADS=8 fermion chat ...     # override the thread count

fermion serve reports the same thing in GET /health as "backend".

Native runtimes exist for macOS arm64 and Linux x86-64 only. On Windows, Linux arm64, or with --device cuda/mps, you get the torch path: correct, and one to two orders of magnitude slower. The published speed figures are native figures and do not describe the torch path.

Two known differences on the native path, both deliberate and documented in fermion/native.py:

• The C runtime disables its whole sampler at --temp 0, so a greedy request that also carries a repetition penalty is mapped to --temp 0.01 --min-p 0.999 --seed 0 — argmax by construction, penalty intact, still byte-reproducible.
• Greedy output is token-identical to the torch reference at float32, not at the CLI's bfloat16 default, and identity is a near-tie property rather than a guarantee: two independent implementations pick different tokens when the top-2 logits are within measurement noise. Measured agreement and the divergence analysis are summarised on the model card.

What backs it

• TRTC v4 containers load through the hf_ternary integration (vendored verbatim, sha-recorded): native Qwen3ForCausalLM etc. with packed five-value planes resident (98.9% memory honesty), correctness-gated at 0 greedy mismatches over 768 tokens vs the expander reference.
• The torch path is the reference: correct everywhere torch runs, fast nowhere. It is what fermion verify and --draft (speculative decoding is a torch-graph feature) use, and what the native path is gated against. Measured numbers live in the model card and eval-receipts, each with venue+version+date.
• CLI activation dtype defaults to bfloat16 (--dtype to override): fp16 NaN-overflows at 8B scale (measured; receipts in packaging/acceptance/), while --dtype float16 reproduces the fp16-identity receipts on the small twin containers gated that way.
• Tiny models (QR 2.5 KB GRU / GIF 170 KB transformer) run through the float64 reference decoder the browser demo is bit-exactness-gated against.

import fermion
from transformers import AutoModelForCausalLM
fermion.write_transformers_config("neutrino-8b_v4.bin", "cfg-dir")
model = AutoModelForCausalLM.from_pretrained("cfg-dir")   # native Qwen3

Loading in plain transformers

hf download fermionresearch/Neutrino-8B --local-dir Neutrino-8B \
    --exclude "gguf/*" --exclude "*.tv4z"

import fermion          # <-- REQUIRED, and it must come first
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Neutrino-8B")
tokenizer = AutoTokenizer.from_pretrained("Neutrino-8B")

import fermion is what registers the trtc_v4 model type with the Transformers Auto classes. Pass a local directory, not a Hub id.

If you see this error, you forgot the import

ValueError: The checkpoint you are trying to load has model type `trtc_v4`
but Transformers does not recognize this architecture. ... You can update
Transformers with the command `pip install --upgrade transformers`.

Ignore that advice. Upgrading Transformers will never help and neither will installing it from source — trtc_v4 is our model type, registered at import time by this package, so no Transformers release will ever know it. trust_remote_code=True does not help either: these repos carry no auto_map. Add import fermion above the Transformers import and it loads.

Dev

pip wheel --no-deps -w dist .     # build the wheel
python -m venv /tmp/v && /tmp/v/bin/pip install dist/*.whl
/tmp/v/bin/fermion --version

License: Apache-2.0 (flagship is a Qwen3-8B derivative, Apache-2.0 upstream).