decidr

Typed decisions from local LLMs, in one forward pass.

Most decisions an application asks an LLM to make are small: route this ticket, is this evidence sufficient, how angry is this customer. A chat model can answer them, but it generates a sentence, or JSON, which your code then parses back into an if statement.

decidr skips that. It gives the model your options, runs one forward pass, and reads the probability of each option directly out of the model's own logits. No answer sentence. No JSON to repair. No decoding loop.

from decidr import Client

client = Client(model="qwen3.5:4b")

decision = client.decide({
    "id": "route-1",
    "state": "Customer cannot access an account after a password reset. The reset email never arrived.",
    "question": "Which queue should handle this request?",
    "options": [
        {"id": "access",  "description": "Account access and authentication support."},
        {"id": "billing", "description": "Billing and payment support."},
        {"id": "sales",   "description": "Sales and product evaluation."},
    ],
})

decision.choice          # 'access'
decision.confidence      # 0.9999
decision.probabilities   # {'access': 0.9999, 'billing': 0.0001, 'sales': 0.0}
decision.is_reliable()   # True

Works with any model you already have in Ollama. No fine-tuning, no extra runtime, no separate model to download.

Install

pip install decidr

Requires Python 3.10+ and a running Ollama. Zero dependencies — it's stdlib urllib and math.

How it works

Three things, in order:

1. Options become letters. Each option is presented to the model as A, B, C… rather than by its own name. This is not cosmetic: real labels like billing or SYS_OUTAGE are usually several tokens, and you cannot read a single-token probability for a multi-token string. Letters are single tokens in every vocabulary. The meaning lives in the descriptions, which is where the model actually reads it.

2. One forward pass, no generation. The prompt ends where the answer begins, and generation is capped at a single token. Reasoning mode is explicitly disabled — on a hybrid-reasoning model, a <think> preamble would put thinking tokens in the answer slot, and the next token would stop being the decision.

3. The scores are read, not sampled. Instead of taking whichever letter the model emitted, decidr reads the log probability of every option letter and normalizes over them. You get a distribution, not just a pick — so you can threshold on confidence, route ambiguous cases to a human, or log calibration over time.

Two modes, picked automatically

How completely decidr can read those scores depends on your Ollama build. It probes once per process and tells you which mode you're in via decision.mode.

Mode	When	Behavior
ranked	Stock Ollama (what you have today)	Falls back to top_logprobs (capped at 20 by the API). Options whose letters don't surface in that window are reported in decision.unscored.
exact	Ollama with logprob_tokens	Every option's probability is read directly from the full distribution, regardless of rank.

Why this distinction exists: stock Ollama can only tell you about tokens that rank in the model's top 20 guesses. For a 3-option decision, the letters almost always make that cut and ranked is fine. For a 12-option decision, they often don't:

12 options, same model, same prompt:
  exact    ->  scored 12/12   reliable=True
  ranked   ->  scored 11/12   reliable=False   unscored: ['cat9']

decidr never invents a number for an option it couldn't measure. It reports it as unscored, is_reliable() returns False, and the remaining probabilities are normalized over what was actually observed. A fabricated floor value would be indistinguishable from a real measurement, which is the one thing a probability API must never do.

Getting exact mode

logprob_tokens is a change I've proposed upstream to Ollama — it is not merged yet:

• Issue: ollama/ollama#18579
• PR: ollama/ollama#18580

Until it lands (if it lands), decidr works today in ranked mode against stock Ollama. Nothing here depends on that PR being accepted — exact mode is an upgrade, not a requirement. If you want it now, you can build from the branch.

API

Client(model, host=..., timeout=..., temperature=..., force_mode=None)

• model — any Ollama model name, e.g. "qwen3.5:4b".
• host — defaults to http://127.0.0.1:11434.
• temperature — applied to the softmax over option logprobs, not to sampling. Higher values flatten confidence. Use this to calibrate against a labeled set (see below).
• force_mode — skip the capability probe and pin "exact" or "ranked".

client.decide(row) -> Decision

row needs id, state, question, and 2–16 options, each with an id and a description. state may be a string, dict, or list. Malformed rows raise DecisionError naming the problem rather than silently producing a confident wrong answer.

Decision

Field
choice	option id with the highest probability
confidence	probability of choice
probabilities	option id → probability, sums to 1 over scored options
logprobs	raw log probabilities before normalizing
mode	"exact" or "ranked"
unscored	options the server could not report
is_reliable()	False when anything went unscored
raw_answer	the letter the model actually emitted

client.decide_all(rows) runs a list sequentially.

Calibration

Probabilities are conditional on the option set you supplied and are not calibrated by default. choice is usually right. confidence on its own is not a true probability until you check it against real outcomes.

decidr.calibrate fits one temperature to your own labeled data and rescales the probabilities:

from decidr import Client
from decidr.calibrate import fit_temperature, evaluate_out_of_fold

client = Client(model="qwen3.5:4b")
labeled = [(client.decide(row), row["correct_id"]) for row in labeled_rows]

fit_temperature(labeled)
# CalibrationResult(temperature=1.8, n=200, ece_before=0.15, ece_after=0.06, accuracy=0.83)

evaluate_out_of_fold(labeled, folds=5)
# fits T on 4 folds, scores it on the 5th, repeated for every fold.
# use this number, not fit_temperature's, since a temperature can overfit
# to the exact sample it was fit on.

Rescaling by a constant can't change which option wins, so choice stays fixed and only confidence moves. Without labeled data, confidence still ranks options correctly relative to each other, just treat the absolute number as approximate.

What this is not

• Not a new model. It's a way of reading models you already run.
• Not a fine-tune. Nothing is trained; taxonomies change per request.
• Not novel. Reading option logits in a single pass is an established technique — TypeSafe's Jev, Laya, and SemIf all do versions of it, and SemIf in particular has done far more rigorous benchmarking and calibration work than this has. decidr's only claim is a narrow one: it's the smallest possible version that runs against an Ollama you already have, with zero dependencies and no model downloads.

Limitations

• Sequential — no batching yet. One decision, one request.
• 2–16 options (letters A–P).
• ranked mode's completeness degrades as option count grows.
• Probabilities uncalibrated by default; fixable with decidr.calibrate and your own labeled data (see above).
• Tested against qwen3.5:4b. Small models (<1B) often won't treat a bare letter as a plausible next token; decidr raises a clear error rather than returning noise if none of the letters are scored.

License

MIT