decidr

Typed decisions from local LLMs, in one forward pass.

Most decisions an application asks an LLM to make are small: route this ticket, is this evidence sufficient, how angry is this customer. A chat model can answer them, but it generates a sentence, or JSON, which your code then parses back into an if statement.

decidr skips that. It gives the model your options, runs one forward pass, and reads the probability of each option directly out of the model's own logits. No answer sentence. No JSON to repair. No decoding loop.

from decidr import Client

client = Client(model="qwen3.5:4b")

decision = client.decide({
    "id": "route-1",
    "state": "Customer cannot access an account after a password reset. The reset email never arrived.",
    "question": "Which queue should handle this request?",
    "options": [
        {"id": "access",  "description": "Account access and authentication support."},
        {"id": "billing", "description": "Billing and payment support."},
        {"id": "sales",   "description": "Sales and product evaluation."},
    ],
})

decision.choice          # 'access'
decision.confidence      # 0.9999
decision.probabilities   # {'access': 0.9999, 'billing': 0.0001, 'sales': 0.0}
decision.is_reliable()   # True

Works with any model you already have in Ollama. No fine-tuning, no extra runtime, no separate model to download.

Install

pip install decidr

Requires Python 3.10+ and a running Ollama. Zero dependencies — it's stdlib urllib and math.

How it works

Three things, in order:

1. Options become letters. Each option is presented to the model as A, B, C… rather than by its own name. This is not cosmetic: real labels like billing or SYS_OUTAGE are usually several tokens, and you cannot read a single-token probability for a multi-token string. Letters are single tokens in every vocabulary. The meaning lives in the descriptions, which is where the model actually reads it.

2. One forward pass, no generation. The prompt ends where the answer begins, and generation is capped at a single token. Reasoning mode is explicitly disabled — on a hybrid-reasoning model, a <think> preamble would put thinking tokens in the answer slot, and the next token would stop being the decision.

3. The scores are read, not sampled. Instead of taking whichever letter the model emitted, decidr reads the log probability of every option letter and normalizes over them. You get a distribution, not just a pick — so you can threshold on confidence, route ambiguous cases to a human, or log calibration over time.

Two modes, picked automatically

How completely decidr can read those scores depends on your Ollama build. It probes once per process and tells you which mode you're in via decision.mode.

Why this distinction exists: stock Ollama can only tell you about tokens that rank in the model's top 20 guesses. For a 3-option decision, the letters almost always make that cut and ranked is fine. For a 12-option decision, they often don't:

12 options, same model, same prompt:
  exact    ->  scored 12/12   reliable=True
  ranked   ->  scored 11/12   reliable=False   unscored: ['cat9']

decidr never invents a number for an option it couldn't measure. It reports it as unscored, is_reliable() returns False, and the remaining probabilities are normalized over what was actually observed. A fabricated floor value would be indistinguishable from a real measurement, which is the one thing a probability API must never do.

Getting exact mode

logprob_tokens is a change I've proposed upstream to Ollama — it is not merged yet:

• Issue: ollama/ollama#18579
• PR: ollama/ollama#18580

Until it lands (if it lands), decidr works today in ranked mode against stock Ollama. Nothing here depends on that PR being accepted — exact mode is an upgrade, not a requirement. If you want it now, you can build from the branch.

API

Client(model, host=..., timeout=..., temperature=..., force_mode=None)

• model — any Ollama model name, e.g. "qwen3.5:4b".
• host — defaults to http://127.0.0.1:11434.
• temperature — applied to the softmax over option logprobs, not to sampling. Higher values flatten confidence. Use this to calibrate against a labeled set (see below).
• force_mode — skip the capability probe and pin "exact" or "ranked".

client.decide(row) -> Decision

row needs id, state, question, and 2–16 options, each with an id and a description. state may be a string, dict, or list. Malformed rows raise DecisionError naming the problem rather than silently producing a confident wrong answer.

Decision

client.decide_all(rows) runs a list sequentially.

Calibration, honestly

The probabilities are conditional on the option set you supplied and are not calibrated out of the box. A raw instruction-tuned model's logits are not a calibrated probability distribution — they're systematically overconfident, and binary yes/no framings in particular carry a strong prior-driven skew.

What that means in practice: choice is usually trustworthy; confidence is directional, not a true probability. Before thresholding on it (e.g. "auto-route above 0.9"), fit temperature against a labeled sample from your own workload and measure whether the numbers track observed accuracy. Purpose-built decision models are trained against proper scoring rules precisely because this step matters.

If you need calibrated confidence out of the box more than you need "runs on anything you already have," use a trained decision model instead. decidr's bet is that zero setup is worth more for most use cases, and that you should be told plainly where the limits are.

What this is not

• Not a new model. It's a way of reading models you already run.
• Not a fine-tune. Nothing is trained; taxonomies change per request.
• Not novel. Reading option logits in a single pass is an established technique — TypeSafe's Jev, Laya, and SemIf all do versions of it, and SemIf in particular has done far more rigorous benchmarking and calibration work than this has. decidr's only claim is a narrow one: it's the smallest possible version that runs against an Ollama you already have, with zero dependencies and no model downloads.

Limitations

• Sequential — no batching yet. One decision, one request.
• 2–16 options (letters A–P).
• ranked mode's completeness degrades as option count grows.
• Probabilities uncalibrated by default (see above).
• Tested against qwen3.5:4b. Small models (<1B) often won't treat a bare letter as a plausible next token; decidr raises a clear error rather than returning noise if none of the letters are scored.

License

MIT