sys1bench 0.3.1

sys1bench benchmarks System One decision models: non-autoregressive models that read a block of state plus typed questions (choice, score, noul) and return a calibrated probability distribution in one forward pass, instead of generating text. The first two such models are TypeSafe's hosted Jev and Convai's open-weight Laya; the harness is model-agnostic so the next ones plug in through an adapter or a YAML file.

It answers the questions a vendor scorecard does not: whether the model beats trivial baselines on the same items, whether its accuracy depends on how the question is worded, whether its probabilities mean anything and in which direction they are wrong, whether confidence can gate actions, how it degrades with option count, state length, noise and out-of-scope inputs, whether batching questions changes answers or only cost, and what all of it costs per 100k decisions.

Why another benchmark

Install

pip install sys1bench                 # core: numpy, scipy, pandas, pydantic, httpx, typer
pip install "sys1bench[plots]"        # + matplotlib for reliability / risk-coverage / framing plots
pip install "sys1bench[laya]"         # + the laya package (needs torch; GPU recommended)
pip install "sys1bench[local]"        # + torch, transformers, sentence-transformers baselines

From source: git clone https://github.com/rssr25/system-one-bench && cd system-one-bench && uv pip install -e ".[dev]".

Quickstart (offline, two minutes)

sys1bench generate support_tickets tickets.jsonl --n 200 --seed 42
sys1bench run tickets.jsonl preds.jsonl --adapter mock --framings support_tickets --permutations 3 --corruption --short-circuit
sys1bench run tickets.jsonl prior.jsonl --adapter majority_prior
sys1bench score preds.jsonl prior.jsonl --out summary.json

| model             | acc (median) | acc range      | ECE/floor | ECE15 | Brier | AURC  | cov@5% | T     | perm JSD | p50 ms |
|-------------------|--------------|----------------|-----------|-------|-------|-------|--------|-------|----------|--------|
| mock-v1 / queue   | 0.783        | [0.783, 0.795] | 2.736     | 0.124 | 0.387 | 0.214 | 0.000  | 1.070 | 0.182    | 5.9    |
| mock-v1 / priority| 0.750        | [0.750, 0.752] | 3.614     | 0.096 | 0.438 | 0.195 | 0.004  | 1.271 | –        | 5.9    |

Run it on a real model

# Jev (hosted, first-party API)
echo 'TypeSafe_API_KEY=...' > .env
sys1bench canary jev_typesafe --model jev-1.13.0                        # 200 fixed items; drift baseline
sys1bench suite all results/jev --config jev_1.13 --n 500 --concurrency 4      # packaged config; `sys1bench configs` lists them

# Laya (local; same manifests as Jev so the comparison is item-for-item)
sys1bench suite all results/laya --config laya_en --manifests-from results/jev --budget

# Report, LaTeX tables, HTML dashboard and plots for everything under results/
sys1bench report results --latex results/tables.tex --html results/dashboard.html
sys1bench plots results

suite takes any subset of A C D E F G I (or all) and writes into results/<name>/ and results/<name>_sweeps/; every request is cached, so re-running is free, and a local model that lands on CPU when CUDA was requested aborts and is retried rather than silently measured. The scripts/ directory keeps shell equivalents.

results/REPORT.md contains, per model: the headline table above, ordinal fidelity for score questions, the framing table (paraphrases, label-only, with negatives, vague, swapped descriptions), decomposition, control arms and audits, then the sweep tables (cardinality, length, interference, robustness, prior shift, option budget), and cost. results/plots/ overlays every model on the same reliability, risk-coverage and framing-range axes.

Adding your model

Capability limits are declared, then measured: a request outside them is recorded as a capability_issue or rejected_by_vendor row, never a crash.

Suites

First results

Jev 1.13.0 and Laya 0.3.4 (english and typed-decisions) on identical Tier G manifests, n=500, 22 September 2026. Findings F1 to F9 with caveats: docs/RESULTS_2026-09-22_narrative.md; generated tables: docs/RESULTS_2026-09-22.md.

Headline: both models saturate the easy choice and noul questions; the policy-following score questions are where they separate. Jev is more accurate on every choice and noul question, reads lookup tables and numbered policies at 1.00, is language-independent, and keeps answers independent of co-asked questions, but it is strongly over-confident on scores (refit temperature 3.8 to 4.1), its ticket-priority accuracy moves from 0.39 to 0.53 with wording alone, and under a 20:1 cost matrix acting on its probabilities is worse than argmax. Laya is under-confident on scores, beats Jev on 4-level urgency (typed-decisions 0.79 vs 0.48), has a cleaner out-of-scope signal, but collapses with option count, on JSON records, on Devanagari, and under negation, and its latency grows linearly with batched questions. A fine-tuned DistilBERT trained on a disjoint seed scores 1.00 on every question, so every current label is learnable from surface features; the System One models' failures are failures to apply stated policies zero-shot. Sixteen findings with numbers are in the narrative.

Documentation

• docs/SPEC_v2.md: the benchmark specification (contract, data tiers, suites, statistics, audits, future-model rules)
• docs/REVIEW_v1.md: review of the original plan against the public state of the art
• CHANGELOG.md

Layout

src/sys1bench/
  schemas.py      manifest v2, DecisionRequest/Response contract, ModelCapabilities
  adapters/       mock, jev_typesafe, jev_openrouter, laya_local, generic_http, hybrid_router, encoder_finetuned, baselines
  metrics/        calibration, selective, ordinal, consistency, robustness, efficiency, decision_value
  generators/     support_tickets, phishing_email, rag_relevance, log_triage, policy_compliance, guardrail_intent,
                  multilingual_tickets (paired control arms, regex rules, cost matrices)
  framing/        paraphrase / criteria expansion, corruption, permutation, short-circuit, decomposition, perturbations, prior shift
  runners/        cached runner (reparse-from-raw), canary, sweeps, robustness
  analysis/       paired bootstrap, McNemar, Holm; audits; decomposition; control arms
  report/         scorecards, results document, plots
  data/           packaged framing sets and the 200-item drift canary
  cli.py          suite | generate | run | score | canary | sweep-* | interference | robustness | noul-consistency |
                  ordinal-probes | decision-value | hybrid-sweep | report (--latex, --html) | plots

Citing

If you use sys1bench, please cite the repository and the results document with the model versions and date; the vendor models move, so the version string returned by the provider is part of every row.

License

Apache 2.0. Generated data, framing sets and results are released under the same license.