DecisionNode Edge runtime

The official runtime for the DecisionNode Edge models from Bynn Intelligence. Load a model, send a request in the DecisionNode API format, and receive typed, calibrated decisions. It runs on NVIDIA GPUs, Apple silicon and CPUs, entirely offline.

Models: DecisionNode Edge and DecisionNode Edge Mini on Hugging Face.

Install

pip install decisionnode-edge              # torch, transformers, safetensors
pip install "decisionnode-edge[images]"    # adds Pillow and torchvision, for pictures

The model folders carry a copy of this package in runtime/ for machines without network access: pip install ./runtime (or "./runtime[images]") from the model folder installs the same thing.

Optional, for speed on NVIDIA GPUs with the PyTorch backend: flash-linear-attention and causal-conv1d. transformers uses them when they are installed. Without them the model gives the same answers about twice as slowly on long inputs.

The fastest install for your device

The runtime reads the model with PyTorch unless a faster backend is installed. Install the one for your device:

Device	Install	Backend
NVIDIA GPU (Linux)	pip install "decisionnode-edge[vllm]"	vllm
x86 CPU	pip install "decisionnode-edge[openvino]", then build the OpenVINO files once (below)	openvino
Apple silicon	pip install "decisionnode-edge[mlx]"	mlx
Any device llama.cpp supports, including NVIDIA Jetson	llama.cpp's llama-server; the model folder ships the 8-bit GGUF file (below)	llamacpp

Every backend agrees with the PyTorch reference on at least 98% of top answers in our tests, and the remaining differences fall on near ties. A backend only computes the model's last hidden state; the runtime then applies the model's own answer table (readout.safetensors) and calibration (calibration.json) in float32, as it does on PyTorch.

Measured with one truth question per request, one request at a time, median time of the whole decide call:

Device (as measured)	Backend	Edge, about 500 / 2,000 tokens	Edge Mini	Answers kept (Edge, Edge Mini)	PyTorch on the same device (Edge; Edge Mini)
NVIDIA RTX PRO 6000 Blackwell	vllm	9.5 / 23 ms	7.3 / 15 ms	99.7%, 99.2%	14 / 28 ms; 14 / 16 ms (with the optional kernels)
AMD Ryzen 7 9800X3D (8 cores)	openvino	0.46 / 1.7 s	0.24 / 0.90 s	98.6%, 98.4%	bfloat16: 0.49 / 2.6 s; 0.28 / 1.4 s. float32, the default: 1.6 / 7.8 s; 0.66 / 4.0 s
Apple M5 Max	mlx	35 / 127 ms	17 / 65 ms	99.5%, 99.3%	device="mps": 0.30 / 1.3 s; 0.28 / 1.2 s
Apple M5 Max	llamacpp (Metal)	55 / 212 ms	30 / 112 ms	99.4%, 99.0%
NVIDIA RTX PRO 6000 Blackwell	llamacpp (CUDA)	14 / 44 ms	10.5 / 30 ms	99.2%, 98.4%
AMD Ryzen 7 9800X3D (8 cores)	llamacpp (CPU)	1.1 / 5.1 s	0.58 / 2.7 s	99.8%, 98.1% (412 questions up to 512 tokens)

At about 8,000 tokens: vllm 87 ms (Edge) and 50 ms (Edge Mini), mlx 625 and 297 ms, llamacpp 218 and 160 ms on CUDA and 0.92 and 0.55 s on Metal. OpenVINO hands prompts of that length to PyTorch.

"Answers kept" is how often the backend's top answer equals the PyTorch reference (bfloat16 on an NVIDIA GPU) on a fixed set of 2,056 held-out questions; for OpenVINO, on the 1,936 of them up to 2,048 tokens, the longest it is built for.

The GPU is an NVIDIA RTX PRO 6000 Blackwell workstation card, the CPU an AMD Ryzen 7 9800X3D (8 cores, 16 threads), the Mac an Apple M5 Max. Versions: vLLM 0.32.0, OpenVINO 2026.4.1, MLX 0.32.4 with mlx-lm 0.32.2, llama.cpp at commit f174553, PyTorch 2.13 and 2.14, transformers 5.18 and 5.19. All backends ran in bfloat16 except llama.cpp, which ran the 8-bit (Q8_0) file.

Use

from decisionnode_edge import EdgeModel

model = EdgeModel.load("path/to/decisionnode-edge")   # CUDA if present, else the CPU

response = model.decide({
    "state": {"ticket": "I was charged twice and nobody has replied for 3 days."},
    "questions": {
        "route": {
            "type": "choice",
            "instructions": "Where should `ticket` go?",
            "criteria": {"billing": "money", "bug": "something broken", "account": "login and access"},
        },
        "urgency": {
            "type": "score",
            "instructions": "How urgent is `ticket`?",
            "criteria": ["routine", "today", "urgent", "critical"],
        },
        "escalate": {"type": "truth", "instructions": "Should a human take over now?"},
        "days": {"type": "number", "instructions": "How many days has the customer waited?", "min": 0, "max": 30},
    },
})

response["answers"]["route"]       # {"type": "choice", "choice": ..., "confidence": ..., "probabilities": {...}}
response["answers"]["escalate"]    # {"type": "truth", "truth": ...}

model.predict(request) gives the unrounded calibrated probabilities per question instead: one per option or level in request order, [yes, no] for a truth question, one per grid value for a number question. Point and box questions are answered by decide only.

A response names the model that answered: decisionnode-edge-0.9.1 for DecisionNode Edge, decisionnode-edge-mini-0.9.1 for DecisionNode Edge Mini (the name in the model directory's decisionnode_edge.json).

From the command line:

python -m decisionnode_edge path/to/decisionnode-edge request.json
cat request.json | python -m decisionnode_edge path/to/decisionnode-edge -

Serving over HTTP

python -m decisionnode_edge serve path/to/decisionnode-edge --host 127.0.0.1 --port 8000
python -m decisionnode_edge serve path/to/decisionnode-edge --api-key key.txt     # callers send Authorization: Bearer <key>

A small server on Python's standard library, with nothing more to install. It loads the model with the backend EdgeModel.load picks (or --backend, --artifacts, --device, --dtype), answers one request at a time, and listens on 127.0.0.1 unless --host says otherwise.

Route	What
POST /v1/systemone	the /v1/systemone format used by public decision-model leaderboards: {"model", "state", "questions", "assets"} with noul, choice, score and box questions. A noul is answered {"type": "noul", "noul": p}, a choice and a score as on /v1/decide with probabilities that sum to exactly 1, a box as below. model is required, and any name is accepted: a local server has no model catalogue, and the answer names the model that answered. Number and point questions go to /v1/decide
POST /v1/decide	the request and answers of this runtime (below), number, truth, point and box included
GET /v1/models	{"models": [{"name", "description", "release_date"}]}
GET /healthz	{"status": "ok", "model", "backend"}, no key needed

Every answer carries usage: input_tokens and output_tokens (always 0). Pictures go as base64: the runtime's images with data (a local path is refused over HTTP, so a caller can never make the server read a file), or, on /v1/systemone, the format's own:

• an assets map, {"<ref>": {"type": "image", "format": "png", "data_base64": "..."}} with format png, jpg or jpeg (the bytes decide when they say otherwise). Wherever a reference appears in the state, an instruction or a criterion, the text reads Picture <n> and the asset becomes picture n (image when it is the only one, else image_<n>), numbered in the order the references first appear. An asset no text refers to is refused. A score's legend shows its levels as sent, references included.
• image_url content parts ({"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}) in the same places, and images as a list of data URLs.

This is the 0.9.2 behaviour, the same as the hosted /v1/systemone. In 0.9.1 each asset became a picture whose id was its reference, and the text stayed as sent. /v1/decide reads pictures as before.

A box question on /v1/systemone is {"type": "box", "instructions": "..."}. Its answer is {"type": "box", "box": {"top", "bottom", "left", "right"}, "confidence"}, the edges as fractions of the picture's height and width. When the request holds several pictures, the box looks in the one a path in backticks names in the state (`photo`, `cams.door`), else in the first picture of the state; with the runtime's own images, name it with image, as on /v1/decide. A box with max keeps the list answer of /v1/decide.

confidence is the probability that the box is right (an intersection over union of 0.5 or more with the true box) when the model folder holds a box calibration, box_calibration.json. Without one it is present, the probability that the thing is in the picture. The server reads the file once at start and refuses to start on a file it cannot check. --box-calibration FILE uses another file and --box-calibration none uses none; --box-calibration-sha256 (or box_calibration_sha256 in decisionnode_edge.json) pins the file's sha256. /v1/decide never reads it.

Errors come as {"detail": {"error_type": ..., "message": ...}} (a 422 as a list of field errors). A request longer than the model's limit is a 400 whose message says "maximum context length"; a choice with more than 255 options is a 400 that says "options per choice". On /v1/systemone these limits come in the hosted route's bodies, which benchmark harnesses read to record a request as beyond the model:

Limit	/v1/systemone body
longer than the model's limit	{"detail": {"error_type": "max_tokens_exceeded", "message": "The request does not fit the model's maximum context length (...)."}}
more than 255 options	{"detail": "Too many choices. Must have at most 255 choices (255 options per choice)."}
more than 10 levels	{"detail": "Too many score levels. Must have at most 10 levels (a score takes 2 to 10 levels; one level is answered without the model)."}

Options

EdgeModel.load(path, device=None, dtype=None, backend=None, artifacts=None, options=None):

• device: "cuda", "cuda:1", "cpu" or "mps". Default: CUDA when present, else the CPU. It also sets the device class the backend is chosen for.
• dtype: "bfloat16" or "float32", for PyTorch. Default: bfloat16 on a GPU (the reference precision) and float32 on a CPU. On CPUs with bfloat16 support (recent AMD and Intel parts) dtype="bfloat16" was about three times faster in our tests, with answers as close to the GPU's as float32 gives.
• backend: "auto" (the default), "torch", "vllm", "openvino", "mlx" or "llamacpp".
• artifacts: the directory the build tool wrote, for openvino and llamacpp.
• options: the backend's settings (below).

Choosing a backend

By default EdgeModel.load takes the fastest backend that is installed and can run on the machine:

Device class	Order
NVIDIA GPU	vllm, llamacpp, torch
Apple silicon	mlx, llamacpp, torch
Any other CPU	openvino, llamacpp, torch

openvino counts only when artifacts holds an OpenVINO build, and llamacpp only when artifacts holds a GGUF build and llama-server is found (or options names a running server). Asked for by name, openvino and llamacpp also find a build in the model folder itself, such as the GGUF file that ships there. model.backend says which one was taken.

To choose yourself:

model = EdgeModel.load("path/to/decisionnode-edge", backend="torch")
model = EdgeModel.load("path/to/decisionnode-edge", backend="openvino", artifacts="path/to/edge-openvino")
model = EdgeModel.load("path/to/decisionnode-edge", backend="llamacpp", options={"url": "http://127.0.0.1:8080"})

The same from the environment: DECISIONNODE_EDGE_BACKEND, DECISIONNODE_EDGE_ARTIFACTS, and DECISIONNODE_EDGE_OPTIONS (a JSON object). Arguments win over the environment. On the command line: python -m decisionnode_edge path/to/decisionnode-edge request.json --backend openvino --artifacts path/to/edge-openvino.

The faster backends read text. PyTorch reads what they do not take: pictures, and prompts longer than the backend's limit (an OpenVINO build's longest length, 2,048 tokens by default; a llama.cpp context, 16,384 tokens by default). It is loaded the first time it is needed, on device in dtype, and stays loaded. If most of your requests carry pictures, use backend="torch".

Building what a backend loads

The build tools read the model directory and write a new directory. They never change the model directory.

OpenVINO. One file per static input length, 512 and 2,048 tokens by default:

pip install "decisionnode-edge[openvino]"
python -m decisionnode_edge.build openvino path/to/decisionnode-edge --out path/to/edge-openvino
python -m decisionnode_edge.build openvino path/to/decisionnode-edge --out path/to/edge-openvino --lengths 512,1024,2048

The export runs on the GPU when there is one (--device cuda, then the default) and on the CPU otherwise. On our machine it took 14 minutes for 512 tokens and 19 minutes for 2,048, per length. The files keep the weights in float32 (exact copies of the bfloat16 weights) and OpenVINO runs them in bfloat16, so the build takes 6.2 GB for Edge Mini and 15.2 GB for Edge. Before it ends, the tool runs each file once against PyTorch.

A prompt is padded to the next length it fits, so a 600-token prompt costs about what a 2,048-token one does. Add a length (for example --lengths 512,1024,2048) if many of your prompts fall between two lengths. Lengths above a few thousand tokens take hours to export; longer prompts go to PyTorch.

GGUF for llama.cpp. The model folder ships the 8-bit file (decisionnode-edge-0.9.1-q8_0.gguf, decisionnode-edge-mini-0.9.1-q8_0.gguf) with its build record, so EdgeModel.load("path/to/decisionnode-edge", backend="llamacpp") uses it with no build step. To make it yourself (8-bit keeps the answers; 4-bit does not, so the tool does not offer it):

git clone https://github.com/ggml-org/llama.cpp
cmake -S llama.cpp -B llama.cpp/build -DCMAKE_BUILD_TYPE=Release        # add -DGGML_CUDA=ON on an NVIDIA GPU or a Jetson
cmake --build llama.cpp/build --target llama-server -j
pip install -r llama.cpp/requirements/requirements-convert_hf_to_gguf.txt
python -m decisionnode_edge.build gguf path/to/decisionnode-edge --out path/to/edge-gguf --llama-cpp llama.cpp

The conversion takes seconds and writes one file: 2.0 GB for Edge, 0.8 GB for Edge Mini. The same file serves a CPU, an NVIDIA GPU, a Jetson and a Mac; llama.cpp itself is built for each.

The runtime then starts llama-server on 127.0.0.1 itself (from your PATH, or options={"server": "path/to/llama-server"}) and stops it with the model. To run the server yourself, start it with the embedding endpoint and the last-position pooling, with a batch as large as the context:

llama-server -m edge-gguf/model-q8_0.gguf --embeddings --pooling last -c 16384 -b 16384 -ub 16384 -ngl 999 --port 8080

and pass options={"url": "http://127.0.0.1:8080"}. The runtime sends token ids, so llama.cpp's own tokenizer is never used.

Backend options

Options can be flat ({"threads": 16}) or under a backend's name ({"openvino": {"threads": 16}}), so one set of options can serve whichever backend is taken.

Backend	Option	Default
vllm	any vLLM engine argument, such as gpu_memory_utilization or enforce_eager	a 2 GiB KV cache; the share of the GPU that covers the weights, that cache and 4 GiB for activations
openvino	threads	OpenVINO's choice
openvino	precision: "bf16" or "f32"	"bf16" (OpenVINO falls back to f32 on CPUs without bfloat16)
openvino	config: other OpenVINO CPU properties
llamacpp	url: a running llama-server	start one
llamacpp	server, gguf: the program and the model file	llama-server on PATH, the build's file
llamacpp	gpu_layers, threads	999 (all layers on the GPU when there is one), llama.cpp's choice
llamacpp	context: the longest prompt llama.cpp takes	16,384

vLLM runs its engine inside your process and uses the first visible GPU (choose it with CUDA_VISIBLE_DEVICES). Its first load takes about half a minute while it compiles. MLX uses the Apple GPU.

Pictures

Send pictures beside the state, up to 16, each with an id:

{"images": [{"id": "receipt", "data": "<base64>", "media_type": "image/jpeg"}]}   # as with the hosted API
{"images": [{"id": "receipt", "path": "/data/receipt.jpg"}]}                       # a local file, Edge only

A picture is turned upright from its EXIF orientation, converted to RGB and scaled so its longest side is at most 2,048 pixels. Every question sees every picture. The prompt names a picture Picture <n> (<id>), so instructions can refer to "Picture 2" or to "the receipt".

Points and boxes

A point question asks where something is in a picture, a box question asks for a rectangle around it. Add max to get every match as a list:

response = model.decide({
    "images": [{"id": "belt", "path": "/data/belt.jpg"}],
    "questions": {
        "pick": {"type": "point", "instructions": "The yellow package closest to the robot arm."},
        "damage": {"type": "box", "instructions": "The damaged package.", "units": "normalized"},
        "parcels": {"type": "box", "instructions": "Every parcel on the belt.", "max": 10},
    },
})

response["answers"]["pick"]
# {"type": "point", "x": ..., "y": ..., "present": ..., "confidence": ..., "image": "belt", "width": 1920, "height": 1080}
response["answers"]["parcels"]
# {"type": "box", "count": ..., "present": ..., "complete": ..., "boxes": [{"x1": ..., "y1": ..., "x2": ..., "y2": ..., "confidence": ...}, ...],
#  "image": "belt", "width": 1920, "height": 1080}

Field	Notes
instructions	what to find, required: a phrase ("the yellow package"), a description with conditions, or a question ("Where is the crack?")
image	the id of the picture to look in; needed when the request has two or more pictures
units	"pixels" (the default): whole pixels of the picture as sent, after its EXIF orientation. "normalized": fractions of the width and height with 4 decimals
max	2 to 16: list up to that many places in place of one

In an answer, x and y are measured from the top-left corner, x to the right and y down. A box is its top-left corner (x1, y1) and its bottom-right corner (x2, y2). present is the probability that the thing is in the picture at all; when it is low, the place is only the best guess. confidence says how sure the model is of the place: the product of the peak probabilities of its coordinate readings. It is not calibrated yet, so use it to compare answers, not as a probability. image, width and height name the picture and its size.

With max, count is the number of places listed, complete the probability that there are no more than max, and the places come from the left, ties from the top, each with its own confidence. Two boxes that overlap with an intersection over union of 0.7 or more are one thing: the less confident one is left out.

A point or box question takes a few short prompts on the same picture: whether the thing is there, then where it is across the width, then down the height at that spot, and for a box its width and height around that centre. A point takes 3 prompts, a box 5, a list of n boxes 2 + 4n. Each prompt reads the picture again, so a box costs about five times a truth question on the same picture. As with every question that carries pictures, PyTorch reads them whichever backend is chosen.

The request

The request and the answers follow the DecisionNode API's /v1/decide:

Field	Notes
state	a string (used as it is), an object or an array (compact JSON, key order kept); missing, null and "" are the empty state
questions	id to question, at least one; the id is never shown to the model
images	optional, see above
model	ignored

Question type	Fields
choice	instructions (optional), criteria: {option: description}, 1 to 255 options; a description may be a string, an object, an array or null
score	instructions (optional), criteria: [level, ...] low to high, 1 to 10 levels
truth	instructions and/or criteria: {"true": ..., "false": ...}
number	instructions, min, max, step (defaults 0, 255, 1), up to 4,096 values; more than 256 values are answered in pages of 256 and composed
point, box	instructions, image (with two or more pictures), units (pixels or normalized), max (2 to 16, a list); see Points and boxes

Answers are rounded as the API rounds them: probabilities to 2 decimals (3 for number questions), with the largest entries moved by 0.01 until a list sums to 1.

A request the model cannot answer raises decisionnode_edge.RequestError, whose status is the HTTP status the hosted API gives the same fault (400 or 422). For point and box questions, errors lists every faulty field as the hosted API's 422 does. The runtime checks what the model needs; the hosted API's error bodies are not reproduced word for word. Requests longer than the model's limit (limits.max_input_tokens in decisionnode_edge.json, 16,384 tokens) are refused, never cut.

Not on Edge: videos, boosters, image URLs and uploads, sessions and batch jobs. These are features of the hosted API.

How it works

Each question becomes one prompt: the state block (pictures first, then the state text), then the question block (the type, the instructions, then one line per option, each led by an answer code token), then the word Answer and an answer marker. The codes of a question are drawn from a hash of the question alone, so the same question always gets the same codes. The model reads the prompt in one forward pass, and the last hidden state is scored against the rows of readout.safetensors that belong to the question's codes (or to its yes and no tokens). calibration.json turns those scores into probabilities. The model never generates text.

A point or box question is a short series of such prompts. Its coordinate prompts list no options: they are answered by the 256 code tokens in order, code k meaning step k of 256 along a side of the picture, and the answer is the probability-weighted mean of the steps around the most probable one. A later prompt is given what the earlier ones read: the y is asked at the x already chosen, and a box's size around its centre.

The code is a few short modules: template.py (the prompt and the codes), request.py (reading a request), answers.py (calibration and the answer shapes), places.py (point and box questions), backends.py (what reads the backbone: PyTorch, vLLM, OpenVINO, MLX or llama.cpp), build.py (the OpenVINO and GGUF builds), server.py (the HTTP server) with systemone.py (the /v1/systemone format), and model.py tying them to the weights.

License

The runtime code, like the model weights, is licensed under the GNU Affero General Public License, version 3 only (SPDX: AGPL-3.0-only). For use in closed products and services, Bynn Intelligence Inc. offers a commercial license: licensing@bynn.com.