Decisio

A decision server: typed questions about a piece of text, a probability for every option, one forward pass per question.

Decisio serves a frozen open model, Qwen3.6-35B-A3B, on vLLM and reads its answer to a closed question from the option letters' logits. No text is generated. A request carries a state (a ticket, a document, a record) and one or more questions of three types: yes/no, a choice among named options, or a score on an ordered scale. The answer to each question is a distribution over its options, and the request format is TypeSafe's System One wire format, so existing clients work unchanged.

What ships here is code, recipes and evaluation records. No trained weights: every measured answer comes from the official checkpoint. EVAL_CARD.md states every number, what it was measured on, and what was fitted on what.

What it does

Latency on one RTX PRO 6000, server time, from EVAL_CARD.md (medians; a state the server has never seen, unless marked):

Each further question costs about 17 ms in the served default and 2.7 to 5.4 ms batched when many share a state; at $1.50 per card-hour that is about $0.012 per 1,000 single-question decisions, about $0.007 per 1,000 further questions, and $0.0012 to $0.0022 batched.

--pad-policy row (opt-in) cuts a fresh single question by about 10 to 12 ms: the whole row, state and question, is padded to end on the block boundary, so vLLM prefills it in one engine step instead of two. Measured against the served default on one card in one session: 48.8 to 38.1 ms at 300 tokens, 52.1 to 40.9 at 1,000, 91.0 to 79.2 at 3,000; multi-question requests are unchanged. Accuracy and calibration stayed within noise on the suite, JevBench, the four Decision Index benchmarks and the intent heads. Its cost is repeatability: a repeated identical request can move by up to 0.015 on long prompts (2 of 1,400 suite items in the gate, no answer changed), where the default padding returns the same probabilities every time, so it stays off by default (runs/2026-10-02_pad-policy-row/).

Quickstart

Requirements: Linux, one NVIDIA card with 96 GB (measured on an RTX PRO 6000 Blackwell) and a driver that supports CUDA 13.0 (the runtime uv.lock pins), Python 3.12, uv.

git clone https://github.com/aminry/decisio
cd decisio
uv sync --extra serve --frozen
uv run python -m decisio.serve.vllm_engine --model Qwen/Qwen3.6-35B-A3B-FP8 --model-class hidden-readout

The first start downloads the checkpoint (about 36 GB) and warms the engine. The decisio plugin registers its model classes with vLLM through an entry point, so vLLM 0.30.0 loads them without any patch. patches/ holds an optional latency patch series, off by default; see patches/README.md.

Ask a question:

curl http://127.0.0.1:8000/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": "Hi, since this morning none of our 40 staff can log in to the dashboard. We get \"session expired\" right after entering the password. Payroll is due today.",
  "questions": {
    "urgent":   {"type": "noul",   "instructions": "Does this ticket need a response within the hour?"},
    "category": {"type": "choice", "instructions": "Which team should handle this ticket?",
                 "criteria": {"billing": "Invoices, payments, refunds", "access": "Login, passwords, permissions", "bug": "The product behaves wrongly", "other": null}},
    "impact":   {"type": "score",  "instructions": "Rate the business impact using only the reported facts.",
                 "criteria": ["No function impaired", "One user impaired, with a workaround", "Many users blocked from a core function", "Data loss or legal exposure"]}
  }
}'

Response, abridged (rounded; the same in 30 of 30 repeats on one server, runs/2026-10-02_readme-example/):

{"answers": {
  "urgent":   {"type": "noul",   "noul": 0.90},
  "category": {"type": "choice", "choice": "access", "probabilities": {"billing": 0.01, "access": 0.97, "bug": 0.01, "other": 0.00}},
  "impact":   {"type": "score",  "score": 2.0, "probabilities": {"0": 0.00, "1": 0.02, "2": 0.94, "3": 0.04}}}}

Repeat the request and you get the same probabilities: each question is scored in its own engine call after the state is prefilled once, so its answer equals the question sent on its own. With --multi-question warm the three questions are scored in one batch, faster, and their probabilities vary with that batch: by about 0.12 on this example (the probability of access ranged from 0.79 to 0.91, runs/2026-10-01_docker-first-gpu-start/repeat_variability/), and when two options are close the choice can change too.

Teach it your question in ten examples

The model is frozen, but the server can learn one recurring question from your own labelled examples: it fits a per-task calibration and, for 10 or more options, a small head on the model's hidden state, each kept only if cross-validation on your examples shows a gain. With 10 labelled examples per intent, accuracy on held-out test items rose from 0.740 to 0.847 on BANKING77 (77 intents) and from 0.820 to 0.893 on CLINC150 (150 intents), against the same model unregistered.

curl http://127.0.0.1:8000/v1/tasks -H 'Content-Type: application/json' -d @examples/tasks/examples.json

docs/tasks.md walks through it and states what each number was measured on; examples/tasks/ runs it on a laptop against the CPU stand-in.

Run with Docker

uv build --wheel                   # the image installs this wheel
docker compose up --build          # needs the NVIDIA Container Toolkit and a 96 GB card
curl http://127.0.0.1:8000/health  # answers once the first start has fetched the checkpoint

The first start downloads the checkpoint (about 36 GB) into the decisio-data volume; it is never part of the image. The container runs as a non-root user, compose publishes the port on 127.0.0.1 only, and Dockerfile and compose.yaml explain the rest. Release images go to ghcr.io/aminry/decisio, with their digest in the release notes. First GPU start (2026-10-01, one RTX PRO 6000 Blackwell, image built on the machine from the Dockerfile): healthy in 651 s including the checkpoint download and in 206 s from the cached volume; the example request below and the conformance gates C2 to C4 pass against the container (runs/2026-10-01_docker-first-gpu-start); the image published with v0.1.0 repeated the start from its digest (healthy in 223 s with the checkpoint cached, example and conformance pass, pushed_image_0.1.0 in that run).

Without a GPU

The unit tests and the CPU stand-in engine run on any machine:

uv sync --extra dev --frozen
uv run pytest

--backend hf serves a small Hugging Face model on the CPU for development of the routes; it is not the measured system.

On a Mac with Apple silicon, --backend mlx serves the text route with every feature (tasks, the intent head, the temperature, abstention) from an MLX conversion of the same checkpoint; the image route is not served by it:

uv sync --extra mlx
uv run python -m decisio.serve.vllm_engine --backend mlx --model mlx-community/Qwen3.6-35B-A3B-6bit

The 6-bit conversion is the Mac default and the 4-bit one the option for 32 GB machines (docs/design/mlx-backend.md). It does not pad a state (--pad-policy none, the MLX default; its cache needs no padding), so its prompts are the served ones without the padding, and a question asked alone and inside a request is one prompt. At 6 bits it passes the served default's gates against the FP8 records (suite accuracy -0.1 [-1.4, +1.1] points, the intent heads reproduced, 30.7 GB at 32k tokens). A single question takes about 0.23 s on an M5 Pro, about 0.09 s when its state was seen before (a cross-request prefix cache, exact by construction), and about 0.1 s per question when 100 share a state (docs/design/mlx-backend.md, runs/2026-10-02_mlx-backend).

A laptop version of the same model, packaged for Ollama's decision route, exists as ollama pull aminroudaki/decisio. Ollama builds its own prompt and applies no calibration, so its numbers are the ones on that page, not the ones here.

How it performs

All numbers are on the served default described in EVAL_CARD.md, measured privately with the public harnesses; nothing here is a leaderboard score.

Where it stands: on the public harnesses this frozen model is behind TypeSafe's Jev on hard knowledge questions by several points and on intent taxonomies without labelled examples. With 10 labelled examples per intent, registered heads reach accuracy 0.847 on BANKING77 (150 held-out items) and 0.893 on CLINC150 (100 held-out items), means of three draws. They use labelled examples, so those figures are not comparable with zero-shot systems. Calibration trails too: hard-tier ECE on JevBench is 0.069, above the 0.05 we aimed for. EVAL_CARD.md has the full tables, the calibration figures, and the three disclosures about what was fitted on what.

Repository layout

src/decisio/serve/      the server: /v1/systemone, tasks, abstention, temperature, image route, conformance
src/decisio/readout/    the letters readout, calibration and the intent head (reference implementations)
src/decisio/vllm_plugin/  the vLLM entry point and the two model classes
src/decisio/bench/      benchmark scoring (JevBench v1.5 open-set reading, Decision Index reports)
benchmarks/             scripts that run the public harnesses against a server
patches/                optional vLLM patch series, off by default
Dockerfile  compose.yaml  docker/   the server image (vLLM 0.30.0 release image plus the wheel)
tests/unit/  tests/gpu/ CPU tests run on every pull request; GPU tests run nightly (pytest -m gpu)
runs/                   evaluation records: a manifest, per-item results and hashes per run
docs/                   the task-registration guide and specification, design notes
examples/tasks/         a runnable task-registration walk-through (CPU stand-in)
EVAL_CARD.md            what was measured, on what, and what was fitted on what

Contributing

See CONTRIBUTING.md. Pull requests need a DCO sign-off (git commit -s), pass the CPU tests and lint, and are reviewed by a maintainer before merge. Security issues go through GitHub's private vulnerability reporting, as described in SECURITY.md.

Licence

Apache-2.0 (LICENSE, NOTICE). The model weights are Alibaba's Qwen3.6-35B-A3B under Apache-2.0 and are downloaded, not redistributed. THIRD-PARTY.md lists everything else this project builds on.