Decision-Jef-0.1

Answer several runtime-defined questions about one state, in a single forward pass, with calibrated probabilities over exactly the options you supply.

307M parameters. 12.12 ms for four decisions in one forward pass.

The answer space is built from the request, so a value you did not offer is not representable — not merely unlikely. There is no classification head over a fixed label set.

Results

Typed-decisions test set, 2,000 decisions: 600 choice, 600 noul, 800 score. Same set and same split as the published competitors.

Ahead of Laya by 0.40 global and 1.75 on score. Behind Lex by 1.15 global, with choice and noul within a third of a point and the gap concentrated in score.

Soft accuracy is the label mass on the answer we pick, which matters on a benchmark whose labels are annotator averages rather than single verdicts. Brier is reported both ways because published figures use both conventions and they differ by roughly the option count.

Temperature scaling per (type, cardinality) bucket ships with the model and is applied by default, taking ECE from 0.138 to 0.060. It never changes an argmax, so accuracy is 77.00 either way. Pass calibrated=False to decide() for the raw distribution.

Question isolation is exact: adding a question moves another question's logits by at most 3e-07.

Latency

NVIDIA H100 NVL, bf16, median of 30 calls after warm-up. End to end: packing, encoder and readout.

Going from one decision to four costs 0.39 ms. The state is encoded once and the question branches are masked apart, so a request carrying four questions is not four requests. Throughput at batch 64 and 1,024 tokens is 2.66 ms per decision.

Usage

pip install decision-jef

The weights are published separately from the package. Authenticate with hf auth login if the model repository is not yet public, or point from_pretrained at a local directory holding model.pt.

from decision_jef import Decider, Question

d = Decider.from_pretrained("BarraHome/Decision-Jef-0.1")

state = """from: user@acme.com
subject: Duplicate charge on invoice #4411
body: We were billed twice for March. Please refund the duplicate today
or we will cancel our plan."""

answers = d.decide(state, {
    "department": Question("choice", "Which department should handle this?", {
        "billing": "invoices, payments, refunds",
        "technical": "bugs, outages, system errors",
        "sales": "pricing, new contracts",
        "other": "everything else",
    }),
    "urgency": Question("score", "How urgent is this request?",
                        ["not urgent", "soon", "critical or blocking"]),
    "churn_risk": Question("noul", "Does the user threaten to leave?", {
        "false": "The user makes no threat to stop using the service.",
        "true": "The user threatens to cancel, churn or leave.",
    }),
})

for qid, a in answers.items():
    print(qid, a.choice or a.p("true"), a.confidence, a.probabilities)

department  billing  0.94  {'billing': 0.9383, 'technical': 0.0147, 'sales': 0.0301, 'other': 0.017}
urgency     2        0.58  {'0': 0.156, '1': 0.2662, '2': 0.5778}    score=1.42
churn_risk  0.87     0.87  {'false': 0.1292, 'true': 0.8708}

d.to_wire(answers) returns the same content as a JSON-ready response body.

Give every yes/no outcome a description

This is a requirement, not a style note. The model scores the option text, so a bare yes/no gives it nothing to compare. On the example above:

The three question types

Option order is part of the question. The same options in a different order are a different request, and the model is sensitive to it — include a permutation check in any evaluation.

How it works

<bos> state ... <eos>                                    shared, encoded once
[Q] instructions ... [OPT] opt0 [OPT] opt1 ... [DEC]     question 1
[Q] instructions ... [OPT] opt0 [OPT] opt1 ... [DEC]     question 2

The query is read at [DEC], after the whole option list, so the decision sees every option. Keys come from each [OPT] in the same pass, so the options are read together rather than scored in isolation. Each question attends to the state and to itself only; the state attends to neither.

Limitations

• ECE is 0.060 only with the shipped temperatures applied, 0.138 without. They are on by default; do not disable them unless you are recalibrating.
• score is the weakest type at 74.00, 2.38 behind Lex.
• Trained and measured on English typed decisions. The backbone is multilingual and the tokenizer covers 256k tokens, but no non-English benchmark has been run — treat multilingual use as untested.
• The guardrails and moderation tags reflect coverage of toxicity and hate-speech decisions. Neither capability has been benchmarked.
• Option order changes the answer on 19.3% of choice questions. Measured over 1,800 permutations of the benchmark. Fix the order you present options in, or average over permutations if you need stability.
• Keep choice questions at ten options or fewer. Training never showed more than ten, so beyond that the model is extrapolating: measured zero-shot it holds about 1.9x random up to twenty options and falls toward chance at forty. Nothing is truncated — the packer fits eighty options in 898 tokens — so this is a coverage limit, not a budget one.
• Long states are truncated to the window with the questions reserved first.

License and provenance

MIT, following the jhu-clsp/mmBERT-base backbone. No weights, gradients or private data from any third-party decision service are used or claimed; the design follows publicly documented API behaviour and public benchmark splits.