sys1bench 0.3.2

A benchmark for typed System One decision models
Jev · Laya · Kev · and whatever comes next

System One models read a block of state plus typed questions and return a probability distribution over the answers you defined — one of N options, a level on an ordinal scale, or yes/no — in a single forward pass. No text is generated, so nothing has to be parsed. sys1bench measures what that buys: whether the probabilities are calibrated (against a noise floor), how much the answer depends on how the question is worded, how confidence behaves out of scope, how accuracy scales with options and state length, whether batched questions interfere, and what it all costs per decision. Headline numbers come from generated items whose labels follow a stated policy, so they cannot be memorised.

pip install sys1bench
sys1bench suite all results/my-model --config my_model.yaml        # A–I on generated, policy-labelled data
sys1bench report results --html results/dashboard.html            # tables + interactive dashboard

Results at a glance

_{Jev 1.13.0 (hosted), Laya 0.3.4 and Kev 0.8B / 4B / 9B (local, GB10) on identical generated manifests, n = 500, September 2026. Left to right: accuracy across five paraphrases and criteria variants of the same question (red = canonical wording); reliability on the same question; accuracy as the option count grows from 2 to 255; median latency as more questions share one request. Hosted latency includes the network path and is not comparable to local compute time.}

Why this benchmark

Add your model

A hosted model with a conventional JSON decisions API needs a config file, no code:

sys1bench configs example_future_vendor > my_model.yaml   # fill in url, auth env var, field names

Anything else subclasses BaseAdapter (declare capabilities, implement decide) and registers with @register("my_model") or the sys1bench.adapters entry-point group from your own package. Kev, which speaks the TypeSafe API, runs with --config kev_4b and no code at all.

Documentation

• Specification — contract, data tiers, suites A–I, statistics, audits, rules for future models
• Design review — how the design was derived from the public state of the art
• Changelog · Issues

_{Apache 2.0 · Rahul Sharma · numbers change with model versions, so every row carries the version string the provider returned}