Sharada

Typed decisions about text in one forward pass — options in the request, calibrated probabilities out.

A small encoder that makes typed decisions about text in one forward pass: the options come with the request, and what comes back is a probability for each of them. Nothing is generated, nothing is parsed, and the same model answers a question it has never seen because the labels are part of the input rather than part of the weights.

from sharada import DecisionModel

model = DecisionModel.from_pretrained("lenabarretta/sharada-base")

d = model.decide(
    text="My card still hasn't arrived and I ordered it two weeks ago.",
    question="Which team should handle this?",
    options=["billing", "card delivery", "technical support", "account closure"],
)

d.answer          # 'card delivery'
d.confidence      # 0.86
d.probabilities   # {'billing': 0.07, 'card delivery': 0.86, ...}

Three kinds of question are typed, so the model knows what the options mean:

from sharada import Request

model.decide_many([
    Request(text=review, question="How positive is this review?",
            options=["very negative", "negative", "neutral", "positive", "very positive"],
            kind="scale", task="review-tone"),
    Request(text=review, question="Does this mention a refund?",
            options=["no", "yes"], kind="binary", task="refund-flag"),
])

Install

pip install git+https://github.com/LenaBarretta/sharada

Python 3.10+, torch and transformers; CPU is enough to run it.

Models

Weights are stored in half precision: the encoder was trained under a float16 autocast, so the bits below that were never signal, and the file halves for a shift in the probabilities of about 2e-4. Loading casts back to float32, so fine-tuning is unaffected — this is a storage format, not quantisation, and nothing about the model you get is approximate.

Accuracy is over 38 label sets with every label offered at once — banking on all 77 intents, clinc on all 151 — and ECE is how far the stated probability lands from how often it turns out right. The gap between the two models is widest where it matters most: on label sets neither was trained on, large reads arXiv categories at 0.456 against base's 0.317.

A checkpoint carries its own encoder, limits and temperatures in config.json, so a bigger model — or a multilingual one, built on a multilingual encoder — is another repository rather than another version of the library. It also means a multilingual model cannot be a flag on an English one: this encoder is English down to its tokenizer, and another language means other weights.

Accuracy and calibration per label set are in each model card, measured on label sets the model was not trained on as well as on the ones it was.

What it will not do

• It does not generate. No free-form answers, no extraction, no reasoning out loud. Options or nothing.
• 256 tokens of text by default (Limits), with 48 for the question and 12 per option. Longer documents need chunking or a larger limit, and the limit costs quadratic attention.
• One of them is English-only. base and large are built on an English encoder and were trained on English label sets; the multilingual checkpoints cover far more, but pay about two and a half points of English accuracy for it, and no checkpoint has been measured on a language outside the forty-odd in the mix.
• No medicine, no law, no code. Nothing of the sort is in the training mix. The two medical label sets are measured only, and one of them — verifying public-health claims — lands at the majority-class baseline, which is to say it does not work. Those domains need fine-tuning on your own labelled examples.
• It is small. Where a frontier model knows a fact that is not in the text, it wins. This answers questions about the text in front of it.

Seen and unseen

The two numbers worth separating. A label set the model was trained on is the ordinary case; one it has never seen is the claim that makes this architecture interesting at all, since the labels live in the input rather than in the weights.

Both are weighted by how many examples each label set contributed, so they are the same kind of number as the overall accuracy. The gap is the honest cost of asking a question nobody trained for — and the calibration error roughly doubles across it, which is the part to watch: the model is not only less right on unfamiliar label sets, it is also less aware of being wrong.

Three things the architecture guarantees

The request is laid out as one sequence — text, question, then every option as a parallel branch — and a mask decides who may read whom. Both are in layout.py and masking.py, and they buy three properties that hold by construction, not because training got them approximately right:

1. The order of the options cannot matter. Every option branch starts at the same position id, and the encoder's positions are rotary, so no option is earlier or later than another. Reshuffle them and the probabilities follow their options exactly.
2. An option's score does not depend on which other options are offered. An option reads the text, the question and itself, nothing else. Drop two options from a list of four and the remaining two keep their scores to the last bit — so the probabilities are a renormalisation, and a long list of options does not make each one noisier.
3. The text is read once. Text tokens read only text tokens, so their states do not depend on the question. Ten questions about one document are ten cheap read-outs over one encoding of it.

These are the tests in tests/test_model.py, checked on an untrained model.

Fine-tune it on your own labels

This is what the package is built around: a few hundred labelled examples and a few minutes.

from sharada import DecisionModel, Example

examples = [
    Example(text="the invoice is wrong again", question="Which team should handle this?",
            options=["billing", "technical", "sales"], label=0, task="routing"),
    ...
]

model = DecisionModel.from_pretrained("lenabarretta/sharada-base")
report = model.fit(examples)        # holds out 20%, stops when held-out log loss stops improving
model.save("my-router")

report        # Report(820 examples, accuracy 0.914, log loss 0.287, calibration error 0.031)

fit keeps a part of the examples out, trains on the rest, early-stops, then fits one temperature per task on the held-out part and writes a calibration passport. Useful arguments:

Mixing several tasks in one fit is the normal case — give each one its own task name and each gets its own temperature. See examples/finetune_your_own.py, which trains on a CSV.

The number next to the answer is supposed to be true

0.86 should mean right about 86% of the time. That is a property of a distribution, not of a model, so it is fitted and it expires:

from sharada import check_passport, save_passport

passport = model.calibrate(recent_examples)     # one temperature per task
save_passport(passport, "passport.json")

check_passport(passport, recent_examples, model)
# ['the calibration expired on 2026-04-01; fit it again on recent answers',
#  'routing: the options have changed since the calibration']

check_passport returns an empty list when it finds nothing wrong — run it in the deployment pipeline and fail the build on anything it returns. evaluate gives accuracy, log loss, Brier, expected calibration error with a bootstrap interval, a reliability curve and a risk–coverage curve.

What to do with 0.86

A probability is not a decision. Policy turns one into the action that costs the least, including handing the request to a person:

from sharada import Policy, escalation_budget

policy = Policy(options=["approve", "reject"],
                costs={("fraud", "approve"): 10_000, ("clean", "reject"): 100},
                escalate=30)

policy.act({"fraud": 0.1, "clean": 0.9})     # Action('answer', 'reject', expected_loss=90.0)
policy.act({"fraud": 0.5, "clean": 0.5})     # Action('escalate', ...)

# a person can look at 5% of the traffic, no more:
policy = escalation_budget(policy, probabilities, budget=0.05)

Every label set, one by one

Rows marked unseen were kept out of training entirely. A dash means the label set is not in that model's mix. Trained on is what the set contributed to training, measured on how many held-out examples the accuracy rests on — read the second one first, because a row measured on ninety-six examples moves by a full point when a single answer changes.

Per-label-set calibration error and the fitted temperatures are in each model card.

Train any of them yourself

training/ has the whole run. The mix of label sets is declared in sources.py — intents, topics, review scores, emotion, toxicity, spam, entailment — each one asked through several wordings of its question, with the options shuffled and long label sets often shown as a sampled handful, so the model learns to read the options rather than their positions. Some label sets are kept out of training entirely and only measured: those are the zero-shot numbers above.

Every published checkpoint comes from one of these four commands, with the settings they share written out in full so that nothing about them has to be guessed:

SHARED="--steps 15000 --cap-scale 3 --variants 2 --max-hours 11 \
        --eval-every 1000 --eval-examples 300 --checkpoint-every 500 --keep best"

python training/run.py --encoder answerdotai/ModernBERT-base  --out runs/base  \
    --batch-size 16 --accumulate 2 --lr 3e-5 $SHARED
python training/run.py --encoder answerdotai/ModernBERT-large --out runs/large \
    --batch-size 8 --accumulate 4 --lr 2e-5 $SHARED
python training/run.py --encoder jhu-clsp/mmBERT-base  --out runs/multilingual-base  --multilingual \
    --batch-size 8 --accumulate 4 --lr 3e-5 $SHARED
python training/run.py --encoder jhu-clsp/mmBERT-small --out runs/multilingual-small --multilingual \
    --batch-size 16 --accumulate 2 --lr 3e-5 $SHARED

The effective batch is 32 in all four; the micro-batch differs only because a wider model and a larger vocabulary need more memory to hold. --multilingual adds the label sets that come in many languages and keeps the English ones, so a multilingual model gains languages rather than trading English for them; without the flag they are skipped, which is what keeps the English checkpoints reproducible.

Measured on one free T4: about three hours for base, four to five for large, five for multilingual-base. The multilingual runs also spend twenty minutes assembling the mix before the first step, because it downloads sixty-four language editions.

A multilingual encoder costs more to train than its body suggests. mmBERT-base has exactly the same twenty-two layers and hidden size as ModernBERT-base, so a forward pass is the same work — but the optimiser updates every parameter on every step, and 197M of its 287M are the embedding table. That is twice the optimiser traffic for the same arithmetic, which is where the extra two hours go. At inference the table costs nothing: looking a row up is not a matrix multiply.

The run checkpoints every five hundred steps and resumes from a checkpoint if it finds one, which is what lets a Kaggle session that ran out of time be restarted rather than lost; training/kaggle.ipynb is the notebook wrapper, and a word in its first cell picks which of the four to train. What resuming cannot do is extend a run that already finished: the learning-rate schedule is laid out over the total number of steps, so asking for more lifts it back up, and training/README.md has what happened when that was tried.

Where it came from

The design, the experiments behind it and what each training signal did are written up in RLCR from Scratch, with a runnable lab.

Apache 2.0.