decidr

Typed decisions from local LLMs, in one forward pass. No generated sentence, no JSON to parse, no decoding loop.

from decidr import Client

client = Client(model="qwen3.5:4b")

d = client.decide({
    "id": "route-1",
    "state": "Customer cannot access an account after a password reset. The reset email never arrived.",
    "question": "Which queue should handle this request?",
    "options": [
        {"id": "access",  "description": "Account access and authentication support."},
        {"id": "billing", "description": "Billing and payment support."},
        {"id": "sales",   "description": "Sales and product evaluation."},
    ],
})

d.choice          # 'access'
d.confidence      # 0.9779
d.probabilities   # {'access': 0.9779, 'billing': 0.0133, 'sales': 0.0088}

Works with any model already pulled in Ollama. No fine-tuning, no extra runtime.

Install

pip install decidr

Python 3.10+, a running Ollama. Zero dependencies.

How it works

A model computes a probability over its whole vocabulary before it samples anything. Reading that distribution directly, instead of generating text and parsing it back into a decision, skips the entire decoding step. decidr sends the options, runs one forward pass, and reads the probability of each option straight out of the model's logits.

In prefix mode (the default), option ids are scored by their own text rather than a stand-in letter. Since there's no way to ask Ollama how a string tokenizes, boundaries are discovered by reading what the model actually returns and matching it against each option's id, one piece at a time; options that share a prefix get resolved together over more than one request. Full mechanism, including why every option is walked to the end of its own id rather than stopped as soon as it's distinguishable from the others: docs/PREFIX_MATCHING.md.

Two modes

Client(model="qwen3.5:4b")                    # prefix, the default
Client(model="qwen3.5:4b", id_mode="letters")  # fixed cost, capped option count

letters is the older, simpler path: each option is shown to the model as A, B, C, and the meaning stays in the description. It's a fixed one request per decision, always, but the option count is capped at 16 because there are only so many letters.

Both modes read scores from top_logprobs, capped at 20 by Ollama's API. If an option's real next token doesn't land in that top-20 window, it's reported in decision.unscored rather than assigned a guessed value. A separate change proposed upstream to Ollama removes that cap by letting a request name the exact tokens it wants scored, regardless of rank: ollama/ollama#18579, ollama/ollama#18580. Not merged yet. Both modes work today without it and pick it up automatically if it lands, since decidr probes for the capability once per process.

Calibration

Probabilities are conditional on the option set supplied, and are not calibrated by default. choice is usually right. confidence on its own is not a true probability until it's checked against real outcomes.

decidr.calibrate fits one temperature to labeled data and rescales:

from decidr.calibrate import fit_temperature, evaluate_out_of_fold

labeled = [(client.decide(row), row["correct_id"]) for row in labeled_rows]

fit_temperature(labeled)
# CalibrationResult(temperature=1.8, n=200, ece_before=0.15, ece_after=0.06, accuracy=0.83)

evaluate_out_of_fold(labeled, folds=5)
# fits T on 4 folds, scores it on the 5th, repeated for every fold.
# use this number, not fit_temperature's own -- a temperature can look
# better than it is by overfitting to the exact sample it was fit on.

Rescaling by a constant can't change which option wins, so choice stays fixed and only confidence moves. Without labeled data, confidence still ranks options correctly relative to each other; treat the absolute number as approximate.

API

Client(model, host="http://127.0.0.1:11434", temperature=1.0, id_mode="prefix", force_mode=None)

id_mode is "prefix" or "letters", see above. temperature scales the softmax over option logprobs, not sampling. force_mode skips the capability probe and pins "exact" or "ranked".

client.decide(row) -> Decision

row needs id, state, question, and 2+ options (max 16 in letters mode), each with an id and a description. state may be a string, dict, or list. A malformed row raises DecisionError naming the problem.

Decision

client.decide_all(rows) runs a list sequentially.

What's new

v0.3 — real option ids by default (prefix mode), replacing the single-letter scheme as the default path. Removes the 16-option cap for the common case. Adds decidr.calibrate for fitting confidence against labeled data.

Limitations

• Sequential, no batching.
• prefix mode's request count is variable: 1 for options that don't collide, more when they do, bounded by a depth cap.
• letters mode is capped at 16 options and always costs exactly 1 request.
• Both modes lose an option to unscored if its real next token falls outside top_logprobs's top-20 window, which gets more likely as option count or ambiguity grows.
• Confidence uncalibrated by default; fixable with decidr.calibrate and labeled data.
• Tested against qwen3.5:4b. Small models (under 1B parameters) often won't treat a bare option id as a plausible next token at all.

What this is not

• Not a new model. It's a way of reading models already running.
• Not a fine-tune. Nothing is trained; taxonomies change per request.
• Not novel as a technique. Reading option logits in a single pass is established; TypeSafe's Jev and SemIf (an open-source alternative with far more benchmarking and calibration work than this project has) both do it. decidr's claim is narrow: the smallest version that runs against an Ollama already installed, with zero dependencies.

References

• docs/PREFIX_MATCHING.md — how prefix matching and chain-rule scoring work, in full
• ollama/ollama#18579 — proposed upstream change removing the top-20 cap
• ollama/ollama#18580 — the pull request implementing it
• TheoLeeCJ/SemIf — open-source single-pass decision engine with calibration benchmarking
• convaiinnovations/laya — a trained decision model doing a related version of this

License

MIT