decidr

Typed decisions from local LLMs, in one forward pass. No generated sentence, no JSON to parse, no decoding loop.

from decidr import Client

client = Client(model="qwen3.5:4b")

d = client.decide({
    "id": "route-1",
    "state": "Customer cannot access an account after a password reset. The reset email never arrived.",
    "question": "Which queue should handle this request?",
    "options": [
        {"id": "access",  "description": "Account access and authentication support."},
        {"id": "billing", "description": "Billing and payment support."},
        {"id": "sales",   "description": "Sales and product evaluation."},
    ],
})

d.choice          # 'access'
d.confidence      # 0.9779
d.probabilities   # {'access': 0.9779, 'billing': 0.0133, 'sales': 0.0088}

Works with any model already pulled in Ollama. No fine-tuning, no extra runtime.

Install

pip install decidr

Python 3.10+, a running Ollama. Zero dependencies.

How it works

A model computes a probability over its whole vocabulary before it samples anything. Reading that distribution directly, instead of generating text and parsing it back into a decision, skips the entire decoding step. decidr sends the options, runs one forward pass, and reads the probability of each option straight out of the model's logits.

In prefix mode (the default), option ids are scored by their own text rather than a stand-in letter. Since there's no way to ask Ollama how a string tokenizes, boundaries are discovered by reading what the model actually returns and matching it against each option's id, one piece at a time; options that share a prefix get resolved together over more than one request. Full mechanism, including why every option is walked to the end of its own id rather than stopped as soon as it's distinguishable from the others: docs/PREFIX_MATCHING.md.

Two modes

Client(model="qwen3.5:4b")                    # prefix, the default
Client(model="qwen3.5:4b", id_mode="letters")  # fixed cost, capped option count

letters is the older, simpler path: each option is shown to the model as A, B, C, and the meaning stays in the description. It's a fixed one request per decision, always, but the option count is capped at 16 because there are only so many letters.

Both modes read scores from top_logprobs, capped at 20 on stock Ollama. If an option's real next token doesn't land in that window, it's reported in decision.unscored rather than assigned a guessed value. That cap is Ollama's own request validation, not a limit either backend actually has — checked directly, not assumed, by building a server with it raised and confirming a full 100-alternative response comes back correctly. A change proposed upstream raises it from 20 to 100: ollama/ollama#18590, ollama/ollama#18591. Not merged yet.

Client asks for 100 alternatives by default. Against stock Ollama that request is rejected once, and the client falls back to 20 and remembers it for the rest of that Client's life — one failed request the first time a decision is made, never repeated. Against a server with the cap raised, it works on the first try. Either way, nothing needs to be configured by hand.

Calibration

Probabilities are conditional on the option set supplied, and are not calibrated by default. choice is usually right. confidence on its own is not a true probability until it's checked against real outcomes.

decidr.calibrate fits one temperature to labeled data and rescales:

from decidr.calibrate import fit_temperature, evaluate_out_of_fold

labeled = [(client.decide(row), row["correct_id"]) for row in labeled_rows]

fit_temperature(labeled)
# CalibrationResult(temperature=1.8, n=200, ece_before=0.15, ece_after=0.06, accuracy=0.83)

evaluate_out_of_fold(labeled, folds=5)
# fits T on 4 folds, scores it on the 5th, repeated for every fold.
# use this number, not fit_temperature's own -- a temperature can look
# better than it is by overfitting to the exact sample it was fit on.

Rescaling by a constant can't change which option wins, so choice stays fixed and only confidence moves. Without labeled data, confidence still ranks options correctly relative to each other; treat the absolute number as approximate.

API

Client(model, host="http://127.0.0.1:11434", temperature=1.0, id_mode="prefix")

id_mode is "prefix" or "letters", see above. temperature scales the softmax over option logprobs, not sampling.

client.decide(row) -> Decision

row needs id, state, question, and 2+ options (max 16 in letters mode), each with an id and a description. state may be a string, dict, or list. A malformed row raises DecisionError naming the problem.

Decision

client.decide_all(rows) runs a list sequentially.

What's new

v0.4 — removed the exact/logprob_tokens code path. It depended on a change proposed upstream to Ollama that was later withdrawn in favor of a simpler one (ollama/ollama#18590/#18591, replacing the closed #18579/#18580), so no server would ever produce it. mode is now always "prefix" or "ranked".

v0.3 — real option ids by default (prefix mode), replacing the single-letter scheme as the default path. Removes the 16-option cap for the common case. Adds decidr.calibrate for fitting confidence against labeled data.

Limitations

• Sequential, no batching.
• prefix mode's request count is variable: 1 for options that don't collide, more when they do, bounded by a depth cap.
• letters mode is capped at 16 options and always costs exactly 1 request.
• Both modes lose an option to unscored if its real next token falls outside top_logprobs's top-20 window, which gets more likely as option count or ambiguity grows.
• Confidence uncalibrated by default; fixable with decidr.calibrate and labeled data.
• Tested against qwen3.5:4b. Small models (under 1B parameters) often won't treat a bare option id as a plausible next token at all.

What this is not

• Not a new model. It's a way of reading models already running.
• Not a fine-tune. Nothing is trained; taxonomies change per request.
• Not novel as a technique. Reading option logits in a single pass is established; TypeSafe's Jev and SemIf (an open-source alternative with far more benchmarking and calibration work than this project has) both do it. decidr's claim is narrow: the smallest version that runs against an Ollama already installed, with zero dependencies.

References

• docs/PREFIX_MATCHING.md — how prefix matching and chain-rule scoring work, in full
• ollama/ollama#18590 — proposed upstream change raising the top_logprobs cap from 20 to 100
• ollama/ollama#18591 — the pull request implementing it
• TheoLeeCJ/SemIf — open-source single-pass decision engine with calibration benchmarking
• convaiinnovations/laya — a trained decision model doing a related version of this

License

MIT