sysone

Train calibrated "system one" models on your own labels with a single fit call.

A system one model is a small, fast model that reads a piece of text, such as a support thread, an invoice or an agent trace, and answers a fixed kind of question about it with a probability for each answer. It doesn't generate text or reason step by step; it makes the quick, routine call that you would otherwise send to a large LLM, at a fraction of the cost and latency.

Building one usually means writing a prompt format, tokenizing, freezing layers, attaching adapters and a classification head, writing a training loop, picking when to stop, and then correcting over-confident probabilities. sysone does all of that for you. You describe the questions, hand it labelled examples, and get back a model whose "80% sure" means right about 80% of the time.

Documentation: sysone.readthedocs.io

Install

pip install sysone

Python 3.10+. Trains on an NVIDIA GPU, Apple Silicon or the CPU.

Quickstart

Train a model to triage customer-support conversations from the public typed-decisions dataset (300 training and 100 test conversations), then check it against simple baselines.

1. Train and evaluate

import json

from datasets import load_dataset
from sysone import SystemOneModel, baselines, compare

# The questions: what the model should answer about each conversation.
questions = {
    "category": {"type": "choice",
                 "instructions": "What is this customer conversation primarily about?",
                 "criteria": ["account", "billing", "delivery", "refund", "technical"]},
    "action":   {"type": "choice",
                 "instructions": "What should the assistant do next with this conversation?",
                 "criteria": ["answer_directly", "close_no_action", "escalate_to_human",
                              "execute_refund", "request_information"]},
}

# Labelled rows: the conversation (state), the questions, and the right answers (expected).
def load_rows(split):
    ds = load_dataset("LocalLLaMA/typed-decisions", "customer_service", split=split)
    return [{"state": r["state"], "questions": questions,
             "expected": {q: json.loads(r["gold"])[q]["label"] for q in questions}}
            for r in ds]

train, test = load_rows("train"), load_rows("test")

# Fine-tunes every layer of ModernBERT, then calibrates: run this on a decent GPU (e.g. a Colab L4).
model = SystemOneModel("answerdotai/ModernBERT-base", tuning="full", seed=0).fit(train)

results = baselines(test, train=train)   # random guess and average answer
results["full-fine-tuning"] = model.evaluate(test)
compare(results, by="overall")           # accuracy, log loss, Brier score, calibration error

On a L4 Colab GPU, this gave:

                  accuracy  distance  log_loss     brier       ece
random guess          0.20  0.800000  1.609438  0.800000  0.000000
average answer        0.35  0.701567  1.362123  0.709700  0.056667
full-fine-tuning      0.73  0.305703  0.659549  0.360719  0.111332

The fine-tuned model picks the right answer 73% of the time, against 35% for the average answer in the training data, and halves the log loss. Results may vary a little with seed.

2. Predict and save

from pprint import pprint

out = model.predict("I was charged twice for the same invoice. Please fix it.", questions)
pprint(out)

{'answers': {'action': {'answer_confidence': 0.4786,
                        'choice': 'escalate_to_human',
                        'confidence': 0.3483,
                        'probabilities': {'answer_directly': 0.2042,
                                          'close_no_action': 0.0,
                                          'escalate_to_human': 0.4786,
                                          'execute_refund': 0.0296,
                                          'request_information': 0.2876},
                        'type': 'choice'},
             'category': {'answer_confidence': 0.9963,
                          'choice': 'billing',
                          'confidence': 0.9954,
                          'probabilities': {'account': 0.0034,
                                            'billing': 0.9963,
                                            'delivery': 0.0001,
                                            'refund': 0.0,
                                            'technical': 0.0001},
                          'type': 'choice'}},
 'usage': {'input_tokens': 109, 'output_tokens': 0}}

model.save("sysone-support")             # reload with SystemOneModel.load("sysone-support")

Questions are plain dicts, and rows can come from lists of dicts, .jsonl, .csv or .parquet files, DataFrames or Hugging Face datasets. The quickstart guide walks through each step.

Serve it as an API

The model you just saved can answer requests over HTTP. sysone serve puts it behind the System One API (POST /v1/systemone), the request and response shape of TypeSafe's Jev, so Jev clients and the typesafe-sdk work with it by changing only the base URL:

pip install "sysone[serve]"
sysone serve sysone-support                        # the folder saved above, or a Hugging Face repo id

curl http://127.0.0.1:8000/v1/systemone -H "Content-Type: application/json" -d '{
  "state": "I was charged twice for the same invoice. Please fix it.",
  "questions": {
    "category": {"type": "choice", "instructions": "What is this customer conversation primarily about?",
                 "criteria": ["account", "billing", "delivery", "refund", "technical"]},
    "action":   {"type": "choice", "instructions": "What should the assistant do next with this conversation?",
                 "criteria": ["answer_directly", "close_no_action", "escalate_to_human",
                              "execute_refund", "request_information"]}}}'

The response holds the same answers as model.predict, plus the model's name and token usage. To share the model first, model.push_to_hub("your-username/sysone-support") uploads it with a model card, and sysone serve your-username/sysone-support serves it on any machine.

• Local by default: the server listens on 127.0.0.1; add --host 0.0.0.0 to accept requests from other machines.
• API key: --api-key-env SYSONE_API_KEY requires Authorization: Bearer <key>, with the key read from that environment variable.
• Abstention: --abstain 0.8 marks answers below 0.8 confidence as "abstained": true.
• Batching: requests that arrive together share one forward pass.

See Serving for every option, the typesafe-sdk example and deployment notes.

What fit does for you

1. Builds the prompt: the text, the question and its options are laid out in one input; you never write a template.
2. Sizes everything to the model and data: token budgets, which layers train, the head's size, the batch size that fits in memory, and the learning rates.
3. Trains: holds back 10% of the rows for validation, stops early when validation loss stops improving, and keeps the best weights. A run stopped with Ctrl-C resumes where it left off.
4. Calibrates: fits a temperature on the validation rows, so probabilities match how often answers are right. Accuracy is unchanged; log loss, Brier score and calibration error improve.
5. Reports: evaluate and compare give accuracy, log loss, Brier score, calibration error and more, per question type, next to simple baselines.

Questions it can answer

Each question is one of three types, and one model can answer any mix of them. Questions are passed at prediction time, so new questions don't need a new model class, and training rows can mix question sets.

type	example question	the answer
choice	Which team should handle it?	one label from a list, with a probability for each
score	How urgent is it, from none to critical?	a level on an ordered scale
noul	Does this need a human?	the probability of yes

Labels can be one right answer, or a probability per option when several raters disagree.

Models

Any checkpoint of these families works by name. EncoderModel fully tunes every layer by default; DecoderModel trains QLoRA adapters (LoRA without an NVIDIA GPU).

family	class	example checkpoints	notes
ModernBERT	EncoderModel	answerdotai/ModernBERT-base (default), answerdotai/ModernBERT-large	English
Ettin	EncoderModel	jhu-clsp/ettin-encoder-150m, jhu-clsp/ettin-encoder-400m, jhu-clsp/ettin-encoder-1b	English; best measured so far
mmBERT	EncoderModel	jhu-clsp/mmBERT-small, jhu-clsp/mmBERT-base	1,800+ languages
EuroBERT	EncoderModel	EuroBERT/EuroBERT-210m, EuroBERT/EuroBERT-610m, EuroBERT/EuroBERT-2.1B	European and widely used languages
Qwen3	DecoderModel	Qwen/Qwen3-0.6B (default), Qwen/Qwen3-1.7B, Qwen/Qwen3-4B	multilingual
Llama	DecoderModel	meta-llama/Llama-3.2-1B, meta-llama/Llama-3.2-3B, HuggingFaceTB/SmolLM2-1.7B	Meta's checkpoints are gated
Mistral	DecoderModel	mistralai/Mistral-7B-v0.3, mistralai/Ministral-8B-Instruct-2410	7B and up: use a large GPU
Gemma 3	DecoderModel	google/gemma-3-1b-pt, google/gemma-3-1b-it	gated; text-only 1B sizes

SystemOneModel picks the class for you, so you don't need to know whether a model is an encoder or a decoder:

from sysone import SystemOneModel

model = SystemOneModel("jhu-clsp/ettin-encoder-400m")   # an EncoderModel, with its defaults
model = SystemOneModel("Qwen/Qwen3-1.7B")               # a DecoderModel

Fully tuned on the typed-decisions benchmark (1,200 training records), Ettin-encoder-400m reached 0.8025 accuracy, the smallest model at 0.80 or above on its leaderboard of fine-tuned models; ModernBERT-large reached 0.788 and mmBERT-base 0.776 (one run each, the leaderboard's scoring). Gated checkpoints need their licence accepted on Hugging Face and a login. More families are coming; another can be added with one architecture class.

When you need more control

The defaults are meant to work without tuning, but every choice can be overridden:

from sysone import DecoderModel

model = DecoderModel(
    "Qwen/Qwen3-1.7B",
    tuning="lora",            # "head", "lora", "qlora" or "full"
    num_layers=0.7,           # keep the bottom 70% of layers
    lora_layers="all",        # adapters on every kept layer
    lora_rank=16,
    backbone_lr=1e-4,
    head_lr=1e-3,
    epochs=10,
    n_iter_no_change=2,       # early-stopping patience, in epochs
    target_loss=0.4,          # or stop once validation loss reaches this
    loss="soft-ce",
    ordinal_weight=1.0,       # how much score questions respect level order
    balance="category",       # equal weight per category of rows
    calibrate=True,
    checkpoint_dir="runs/",   # autosave during long runs
    checkpoint_every=500,     # optimizer steps between autosaves
    precision="bf16",
    device="cuda",
)

Choosing how to train explains when each setting matters, and the API reference lists every parameter.

Learn more

• Quickstart: questions, labels, training and predictions step by step.
• User guide: data formats, training options, predictions, evaluation and calibration, saving and sharing.
• Examples: notebooks on public datasets, runnable on Google Colab.
• Changelog: sysone is in early development (0.5), and the API may change between minor versions.

Contributing

Issues and pull requests are welcome. See Development for running the tests and building the docs, and the design document for how sysone works.

Acknowledgements

sysone builds on Laya, the tutorials and code of Neural AVB, ModernBERT, Qwen3, the Typed Decisions benchmark, the System One format of TypeSafe AI and many open-source libraries; see Acknowledgements.

License

MIT