s1ops — Dedicated System 1 LLMOps Platform & SDK
The dedicated System 1 LLMOps SDK for sub-100ms reflexive inference (
noul/choice/score).Install:
pip install s1ops· Python: 3.11+,import s1ops/import s1· CLI:s1/s1ops
s1ops is an offline-first, vendor-neutral evaluation, calibration, threshold compilation, blind label audit, and observability platform purpose-built for Typed Decision Models (System 1):
- TypeSafe
/v1/systemone:jev-1.13.0,kev-1.0,laya,von(noul,choice,score) - OpenAI
/v1/decisions:gpt-6-luna(predicate,choice) - Local / Custom Heads: DeBERTa, ModernBERT, BGE rerankers, and constrained logprob classifiers emitting normalized probability distributions.
Why System 1 Models Need Dedicated LLMOps
Traditional LLMOps platforms (Langfuse, Arize Phoenix, LangSmith) are architected for System 2 generative workflows (free-text completions, prompt templates, multi-turn RAG chains, token streaming latency). When applied to System 1 decision models, seven critical gaps emerge:
| Dimension | System 2 LLMOps (Langfuse / Phoenix) | s1ops (System 1 LLMOps) |
|---|---|---|
| Primary Output | Unstructured text (string) |
Calibrated probability vector (noul: p in [0, 1]; choice: a distribution over K options; score: ordinal 0..L-1) |
| Aggregation Unit | Trace / Prompt Name | Strict (question_id, model_served, primitive) group — never pools across wordings or models (drift's per-question report pools models on purpose, so a model change shows) |
| Calibration & Small-n Bias | None (only scalar scores) | Equal-width/mass ECE, Monte Carlo Calibrated Noise Floor (p95), Tie-Mass rounding detector, Temp/Platt/Isotonic post-hoc fit |
| Decision Thresholds & Routing | Hardcoded 0.5 in app code |
Tie-safe Error-Budget Threshold (select_threshold + Clopper-Pearson), System 1 → System 2 Escalation Band ([lo, hi)), & DecisionGate Runtime Evaluator |
| Prompt / Question QA | Free-form prompt playground | Static Linter (JL000–JL060) + Probe Coherence (Noul vs Yes/No Choice twin gap & inverse sum abs(p + p_inv - 1)) |
| Distribution & Model Drift | Token/latency or generic embedding drift | Label-Free Probability PSI & Jensen-Shannon Divergence, Silent model_served Upgrade Alert, & Escalation Band Rate Shift |
| Data Privacy Default | Logs full prompt/response bodies by default | safe-v1 Zero-State Default: never writes raw item state into records, probe ledgers, or exported spans/scores |
s1ops bridges both worlds: you run rigorous System 1 calibration, linting, drift monitoring, and policy compilation locally (s1 CLI + s1 web), execute sub-millisecond runtime gating via s1.DecisionGate, and export safe-v1 body-free scores, annotation queues, and OpenInference spans as local files for Langfuse and Arize Phoenix; s1 export uploads nothing.
Architecture Overview
Inputs (local files; everything except a live probe runs offline): questions.json; decision records records.jsonl / records_canary.jsonl (jev-record/1, s1-record/1); pre-declared slice features features.jsonl; an optional local state store states_local.jsonl.
Questions file. A JSON object mapping question names to questions, or that mapping under one key, {"questions": {...}}. s1 lint check, s1 probe and s1 labelaudit sheet --questions read it the same way. An entry that is not a question, a repeated name, an empty file, or nesting past 100 levels stops s1 probe and s1 labelaudit sheet before anything is sent or written; s1 lint check reports each as a JL000 finding and exits 1.
Fix the wording before probing. question_id hashes the exact wording, option order included. Data that permutes a question's options per row gives every permutation its own question_id, and so its own fits and policies: pick one wording per question first.
A rating is a judgment. "How many stars did this reviewer give?" with levels "1 star: very negative" .. "5 stars: very positive" gets JL012 (counting) and JL031 (numeric levels), which are written for quantities the model would have to count. Suppress them on that question with "x-s1-lint": {"ignore": ["JL012", "JL031"]}; question_id does not change.
Core engine (import s1ops / import s1):
- Lint & Probe Runner — rules
JL000–JL060, budget-guarded probe runner, Noul/Choice twin and inverse coherence. - Calibration & Noise Floor — log loss, Brier, ECE (equal-width and equal-mass), Monte Carlo noise floor, tie-mass detector.
- Improvement Compiler — fit/apply (temperature, Platt, isotonic), error-budget threshold and escalation band tuner, $/1k cost simulator.
- Slice & Blind Label Audit — answerability (Wilson CI), Benjamini-Hochberg FDR error slice miner, blind review sheet with control lift.
- Shadow & Judge Audit — Noul
|Δp|and Choice TVD, exact McNemar-Holm test, rubric quadratic-weighted κ and bias. - Drift Monitor & Runtime Gate — PSI and Jensen-Shannon drift, silent
model_servedguard, sub-millisecondDecisionGate.
Outputs: compiled fits.json and policy.json, a blind review sheet, safe-v1 Langfuse runs/scores/queues, Phoenix / OpenInference GUARDRAIL / EVALUATOR / RERANKER spans (noul / choice / score), and a local 6-tab web workbench on 127.0.0.1:8765. The state store is read by two commands only: s1 labelaudit sheet writes the state text of every item under review (the top-ranked flags plus random controls) into the review sheet, which is meant for a human reviewer, and s1 probe sends each state to the selected backend (a live API for --backend systemone or openai). s1ops never writes that state text into records, ledgers, probe output or exports; s1 apply, which copies its input rows, leaves a top-level state out.
Quickstart
1. Installation
pip install s1ops # or: uv add s1ops
pip install "s1ops[langfuse]" # optional: the Langfuse client, for your own upload of `s1 export` files
pip install "s1ops[typesafe]" # optional: live `s1 probe --backend systemone`
pip install "s1ops[openai]" # optional: live `s1 probe --backend openai`
2. Generate the Deterministic Synthetic Demo Workspace
s1 demo s1-demo
s1 demo writes a complete, seeded offline workspace into a new (or empty) directory and refuses anything else, so it never overwrites a real workspace. --seed N changes the generated data.
questions.json: Canonical questions + intentionalJL010/JL011/JL014/JL023lint cases.records_train.jsonl: 120 calibration-fold records (jev-1.13.0).records.jsonl: 160 held-out evaluation records (jev-1.13.0, with calibrated probabilities fromfits.json, 6 planted gold-label errors, and a concentrated failure slice onlang=ko_slang).records_kev.jsonl: 160 paired shadow records (kev-1.0) on the same items.records_canary.jsonl: 120 production canary records simulating distribution drift (PSI > 0.25) + silent backend upgrade on a third of traffic (jev-1.13.0→jev-1.14.0-rc1). Thejev-1.13.0records carrycalibfromfits.json, ass1 applywrites it; the upgraded ones have no fit.probe_records.jsonl: Noul vs Yes/No Choice twin & inverse polarity probes.features.jsonl: Pre-declared slice tags (answerable,unanswerable,adversarial_trap,lang,has_code).states_local.jsonl: Synthetic item states for the blind review sheet and the probe runner. One row per item,{"dataset", "item_id", "state"}; both commands checkitem_id,state,datasetandsplitthe same way (s1 probealso checks alabel) and read a row withoutdatasetasdataset: "default", the values1 probewrites into its records. To match records that carry nodataset, write"dataset": null.judge_grades.jsonl: 90 paired rubric judge vs human grades across 3 criteria (faithfulness,policy_compliance,tone).fits.json+policy.json: A compiled calibration fit and routing policy.records_categorical_train.jsonl/records_categorical.jsonl: 150 + 150 choice (support_intent) and score (urgency) records per fold (jev-1.13.0, overconfident on purpose), the held-out fold with calibrated probabilities fromfits_categorical.json.fits_categorical.json+policy_categorical.json: their temperature fits and error-budget policies, tuned on calibrated probabilities. The web workbench reads the noul files only.
3. Run the End-to-End CLI Pipeline
# 1) Static Question Lint & Probe Coherence Audit
s1 lint rules
s1 lint check s1-demo/questions.json
s1 lint probe-audit s1-demo/probe_records.jsonl
# 2) Post-Hoc Calibration Fit, Out-of-Sample Apply, and Noise-Floor Audit
s1 fit s1-demo/records_train.jsonl -o s1-demo/fits.json --method temperature
s1 apply s1-demo/records.jsonl --fit s1-demo/fits.json -o s1-demo/records_calibrated.jsonl
s1 audit s1-demo/records.jsonl --fit s1-demo/fits.json
# 3) Error-Budget Threshold & System 1->2 Escalation Band Compilation
s1 policy tune s1-demo/records.jsonl -o s1-demo/policy.json --error-budget 0.10 --use-calib
# For a consumer that acts only on BLOCK: the smallest cutoff at which at most 2 % of label-false items pass
s1 policy tune s1-demo/records.jsonl -o s1-demo/policy_false_pass.json --max-false-pass 0.02 --use-calib
# Route Noul groups by the error-budget threshold, so the budget bounds the accepted answers' error
s1 policy tune s1-demo/records.jsonl -o s1-demo/policy_threshold.json --error-budget 0.10 --noul-routing threshold --use-calib
s1 policy curve s1-demo/records.jsonl --use-calib
# 3b) The same for choice and score answers
s1 fit s1-demo/records_categorical_train.jsonl -o s1-demo/fits_categorical.json --method temperature
s1 apply s1-demo/records_categorical.jsonl --fit s1-demo/fits_categorical.json -o s1-demo/records_categorical_calibrated.jsonl
s1 audit s1-demo/records_categorical.jsonl --fit s1-demo/fits_categorical.json
s1 policy tune s1-demo/records_categorical.jsonl -o s1-demo/policy_categorical.json --error-budget 0.10 --use-calib
# 4) Answerability Pre-Flight & Benjamini-Hochberg FDR Error Slice Mining
s1 slice answerable s1-demo/records.jsonl --features s1-demo/features.jsonl
s1 slice mine s1-demo/records.jsonl --features s1-demo/features.jsonl --fdr-q 0.10
# 5) Calibrated Label Disagreement Ranking & Blind Review Sheet
s1 labelaudit rank s1-demo/records.jsonl --fit s1-demo/fits.json --top 10
# (a sheet asks one question: pass --question when the records hold several; its options are the
# question's classes, and --options may only reorder them; --questions prints the wording)
s1 labelaudit sheet s1-demo/records.jsonl \
--states s1-demo/states_local.jsonl \
--questions s1-demo/questions.json \
--fit s1-demo/fits.json \
-o s1-demo/blind_sheet.md \
-k s1-demo/blind_key.json \
--top 6 --controls 6
# 6) Multi-Backend Shadow Compare (McNemar-Holm) & Rubric Judge Agreement (Weighted κ)
s1 compare s1-demo/records.jsonl s1-demo/records_kev.jsonl
s1 judge audit s1-demo/judge_grades.jsonl
# 7) Distribution & Silent Model-Served Drift Monitor (PSI on raw probabilities, alerting only when a permutation test finds it significant;
# --calibrated compares calibrated ones and needs `calib` on every answered record of both files, from one fit per question and model;
# --raw-only, the 0.1.0 name of the default, cannot be combined with it; a stored `calib` is validated either way)
s1 drift s1-demo/records.jsonl s1-demo/records_canary.jsonl --policy s1-demo/policy.json
# (a live canary has no `calib` yet: pass `--fit s1-demo/fits.json`, which calibrates each file in memory, as
# `s1 apply canary.jsonl --fit s1-demo/fits.json -o canary_calibrated.jsonl --pass-through-unfitted` writes it,
# keeping the records of models without a fit)
# 8) Budget-Guarded Probe Runner (built-in deterministic mock backend) & Runtime DecisionGate
s1 probe s1-demo/questions.json --states s1-demo/states_local.jsonl \
-o s1-demo/probe_live.jsonl --ledger s1-demo/probe_ledger.jsonl --cap-usd 0.50
s1 gate s1-demo/policy.json --fit s1-demo/fits.json \
--question q-30076be2728be682 --model jev-1.13.0 --p 0.52
# a choice or score question takes its distribution instead of --p
s1 gate s1-demo/policy_categorical.json --fit s1-demo/fits_categorical.json \
--question q-a70bd97f0b7f1055 --model jev-1.13.0 \
--probs '{"billing": 0.97, "technical": 0.02, "account": 0.01}' # SELECT:billing
s1 gate s1-demo/policy_categorical.json --fit s1-demo/fits_categorical.json \
--question q-5c05e56162b3eaa0 --model jev-1.13.0 \
--probs '{"0": 0.1, "1": 0.45, "2": 0.4, "3": 0.05}' # ESCALATE
# 9) Safe-v1 Body-Free Export to Langfuse & Arize Phoenix (OpenInference)
s1 export langfuse s1-demo/records.jsonl -o s1-demo/langfuse_export.json --policy s1-demo/policy.json
s1 export phoenix s1-demo/records.jsonl -o s1-demo/phoenix_spans.jsonl --policy s1-demo/policy.json
s1 export phoenix s1-demo/records_categorical.jsonl -o s1-demo/phoenix_categorical.jsonl --policy s1-demo/policy_categorical.json
Live probes. s1 probe --backend systemone sends each state and question to POST /v1/systemone (key in TYPESAFE_API_KEY, endpoint in TYPESAFE_BASE_URL or --base-url); --backend openai sends them to OpenAI's POST /v1/decisions (OPENAI_API_KEY, OPENAI_BASE_URL) with prompt caching disabled. Both need --model; only the offline mock has a default. A response that reports a cache write has no known price and is settled at its full reservation; cached reads, which are discounted, are charged at the full input price, an upper bound. A question that API cannot take as written — a Noul with criteria, object (rather than string or input-part list) instructions, or option and level counts outside its limits — is refused before anything is reserved, rather than sent in a different form under the same question_id. A refused answer is recorded with error_code: "refusal". /v1/systemone resolves an alias to the version that answered (jev-latest came back as jev-1.14.0), so a silent upgrade shows in model_served; /v1/decisions returned the alias gpt-6-luna itself, so a model change behind it is visible only as distribution drift. model_served is whatever the server reports: a compatible server that echoes the request records the name you passed, and one that names every checkpoint alike leaves a checkpoint change visible only as distribution drift too, since drift's model comparison and the exit 3 of s1 apply and s1 gate compare names. The APIs report billable input tokens, not dollars, so --usd-per-input-token is required. Every attempt reserves its worst case first (--worst-usd-per-token, plus --reserve-overhead-tokens for the prompt tokens a server bills on top of the state and question: 300 for systemone and 200 for openai by default: a short state was billed up to 216 and 98 tokens more than estimated, and about 236 and 118 extrapolated to an empty one) and is settled as measured, or, when the request may have been billed without a usable reported cost (timeout, connection error, HTTP error), at its full reservation. Timeouts, connection errors, 408, 429 and 5xx are retried up to --max-retries times. An item that still fails is written as a record with model_served: null and an error_code, which s1 drift counts as a failure. Records are written as they arrive, so a run stopped by --cap-usd or by a measured cost above its reservation keeps what it paid for. Error messages carry only an error code, never server text, since a server may echo the request.
Probe coherence. Unless --no-expand-probes is given, s1 probe sends each Noul with string instructions, or with a supplied inverse (below), three ways: as written (probe_role: "noul"), as a yes/no Choice with the same instructions (twin), and as its inverse (inverse). s1 lint probe-audit then reports, per question and served model, the twin gap |p - p_yes| and the inverse error |p + p_inv - 1|. Write the inverse yourself when you can, under the question's lint extension:
{"positive": {"type": "noul", "instructions": "Is this movie review positive?",
"criteria": {"true": "positive", "false": "negative"},
"x-s1-lint": {"inverse": {"instructions": "Is this movie review negative?",
"criteria": {"true": "negative", "false": "positive"}}}}}
The extension is stripped before hashing, so the question keeps the question_id s1 lint check gives it. Without one, the inverse is generated as "Is it false that: <instructions>" with the true/false criteria swapped; a model that reads that sentence literally answers it like the original, so its error can measure the generated wording more than the model (on a local Kev, s1ops 0.6.0's generated inverse, which also dropped the criteria, gave an inverse flip rate of 1.0 next to a twin gap of 0.012). A Noul with object or list instructions and no supplied inverse is sent once, as written, and the audit skips it. Each inverse record says which wording it got (probe_inverse: supplied or generated), and the audit reports it as inverse_source (null for records from before the tag, or a mix). A question and model whose inverse records carry more than one question_id are refused rather than pooled, so audit the output of one wording at a time. A question carries x-s1-lint or x-jev-lint, not both, and s1 probe refuses a malformed extension, as s1 lint check reports it, before the ledger or the output is opened.
4. Launch the 6-Tab Local Web Workbench
s1 web --workspace s1-demo --host 127.0.0.1 --port 8765
Open http://127.0.0.1:8765 to explore:
- Calibration & Noise Floor: SVG Reliability Diagram (Before vs After Fit), Monte Carlo 95th-percentile ECE Noise Floor, and Tie-Mass rounding detector.
- Threshold & Escalation Simulator: Interactive
band_lo,band_hi,cost_sys2, anderror_budgetsliders with live System 1 Auto vs System 2 Escalation split and Pareto frontier table. - Question Lint & Probe Coherence: Static
JL000–JL060diagnostics + Noul/Choice twin gap and polarity inverse sum audit. - Slices & Blind Label Audit: Wilson 95% CI Answerability table, Benjamini-Hochberg FDR (
q <= 0.10) error slices, and top calibrated label disagreement candidates. - Shadow Compare & Judge Audit: Paired
jev-1.13.0vskev-1.0accuracy, flip rate, exact McNemar Holm-adjusted p-value badges, and per-criterion quadratic-weighted κ. - Distribution & Model Drift: Baseline vs Canary PSI with its permutation p-value and Jensen-Shannon Divergence on raw probabilities, mean/entropy/escalation rate shift, and silent
model_servedupgrade alerts (CRITICAL/WARN/OK).
Python SDK Usage (import s1 / import s1ops)
The outputs in the comments are for a fresh s1 demo s1-demo workspace; the CLI pipeline above rewrites fits.json and policy.json, which changes the gate's decision.
import s1 resolves every export at once, the fitting stack (scipy.optimize) included. import s1ops resolves exports on first use, and a process that only gates, such as a per-call hook, can import the gate directly: from s1ops.runtime.gate import DecisionGate (with load_policies / load_fits) loads numpy, scipy.special and pydantic, not scipy's optimizers, scikit-learn, the CLI or the web app.
import s1
# 1. Define a question and compute its canonical wire-form question_id
q = s1.NoulQuestion(instructions="Does the user message request a financial wire transfer?")
qid = s1.question_id(q) # -> 'q-30076be2728be682'
# 2. Load and strictly group decision records by (question_id, model_served, primitive);
# the demo has one noul group per file
train = s1.select(s1.read_records("s1-demo/records_train.jsonl")).groups[0]
group = s1.select(s1.read_records("s1-demo/records.jsonl")).groups[0]
# 3. Fit temperature scaling on the calibration fold, then audit the held-out fold
# before/after calibration against the Monte Carlo noise floor
fit = s1.fit_group(train, method="temperature", split="cal")
after_P = s1.transform(fit.method, fit.primitive, fit.params, group.P)
report = s1.audit_group(group, bins=10, reps=500, after=after_P)
print("Raw ECE (mass):", report["before"]["ece_mass"], "-> Calibrated:", report["after"]["ece_mass"])
# 4. Sub-millisecond Runtime Gating (DecisionGate)
gate = s1.DecisionGate.from_files("s1-demo/policy.json", "s1-demo/fits.json")
decision = gate.evaluate_noul(
question_id=group.question_id,
model_served=group.model_served,
p_raw=0.52,
)
print(decision.action, decision.escalated, decision.p_calibrated) # -> 'ESCALATE', True, ...
# Choice and score questions: evaluate_choice / evaluate_score take the distribution;
# the action is SELECT:<label> or ESCALATE, and a score decision also reports expected_score.
# gate.evaluate_choice(question_id=..., model_served=..., probs_raw={"A": 0.7, "B": 0.3})
# gate.evaluate_score(question_id=..., model_served=..., probs_raw={"0": 0.1, "1": 0.2, "2": 0.7})
Safety & Statistical Guardrails
- Zero-State Privacy (
safe-v1): record files, the ledgerss1 probewrites, probe outputs, and Langfuse/Phoenix exports never store raw itemstatetext. A record'sstate_sha256, which both exporters copy out, is refused when it is read unless it is absent,null, or the 64 lowercase hex characters of a SHA-256 digest, so a records file cannot carry text to an export under that name; the error says the value's type and length and does not show it.s1 applycopies its input rows, so it leaves a top-levelstateout of every row it writes (state_sha256and the other fields are kept) and adds"state_dropped": Nto its JSON summary when any input record had one.Ledger.reserve/settlerefuse astatemetadata key (ValueError, before anything is written), ands1 ledger recoverrefuses a journal whose entries carry one rather than copy it into a new file; a journal written that way by s1ops 0.2.0 or earlier still opens. - One Routing Rule:
DecisionGate,s1 drift's escalation rate, and thepolicy_actionof both exporters apply a policy through the same rule. Every one of them routes the distribution the policy was tuned on: the calibrated one exactly when the policy was compiled with--use-calib(calibrated: true), whatever a record carries. The reports take it from the record'scalib, so a calibrated policy needss1 applyoutput, or--fitons1 driftands1 export, and refuses a record withoutcalib;s1 driftapplies to each record only the policy of its own(question_id, model_served, primitive)group, so records from another model (the demo canary'sjev-1.14.0-rc1third) are left out of the escalation rate, which isnullwhen no record has a policy. The distributionss1 driftcompares (PSI, JS divergence, quantiles, entropy) are a separate choice: raw on both sides, or calibrated on both with--calibrated, which refuses an answered record withoutcalibrather than compare it raw, and records of one(question_id, model_served, primitive)group calibrated through different fits (calib.fit_sha256), since a refit would show as drift; the report says which indistribution. s1ops 0.1.0 and earlier compared each record'scalibwhen present and its raw probabilities otherwise, and their drift reports have nodistribution. A Noulerror_budgetpolicy accepts whenmax(p, 1 - p)reaches the threshold (else escalates); a band accepts at or aboveband_hi, blocks belowband_lo, and escalates between.s1 policy tune --max-false-pass RATEwrites a Noul band withband_lo == band_hi, which never escalates: its cutoff is the smallest observed probability (or1.0) at which at mostRATEof the group's label-false items reach it (with--use-cp-upper, its Clopper-Pearson upper bound), andmetrics_on_splitrecords the counts it was chosen on (every other policy records what it does on the split:coverage,accepted,errors,accepted_error); it replaces the F1 band, so--max-escalationis refused with it, and choice and score groups keep the error budget. A choice or score policy selects the top answer when its probability reaches the threshold, and escalates on a tie at the top or when no threshold met the error budget (threshold: null); since a tie escalates at any threshold,s1 policy tune,s1 policy curveand the web simulator leave it out of every accepted error and count it only among the items (coverage, delegation). Retuning choice or score data that has such ties can therefore change the threshold, and with itpolicy_sha256, from what s1ops 0.8.0 and earlier wrote; a policy already compiled routes as before; a policy tuned with--use-calibmust be gated with itsfits.json. Exports of policy-routed records carryrouting_rule(s1.routing_ruleon Phoenix spans,metadata.routing_ruleon Langfuse observations), currently"3"; exports from s1ops 0.1.0 and earlier have none, and theirpolicy_actionandpredictedmay differ for the same policy. Choice and score probabilities are renormalized with an exactly rounded sum, so retune choice or score policies compiled by 0.1.0 or earlier on Python 3.11 (runs1 applyfirst for--use-calibones). - A Noul Band Is Not Bound by the Budget: every policy records in
metrics_on_splitwhat it does on the split it was tuned on, by the routing rule itself. For a Noul groups1 policy tunepicks an escalation band that maximizes F1 under--max-escalation, and the band then routes;--error-budgetand--use-cp-uppergovern the separatethreshold, not the band. Soaccepted_errorcan exceed the budget: on the demo records, raw, 0.238 at a budget of 0.10.--max-escalationdoes not help: at 0 the band is empty, one cutoff that sends nothing to System 2 and accepts every item (0.275 there), not the threshold.--noul-routing thresholdroutes a Noul group by the threshold instead (mode: error_budget), so the budget boundsaccepted_erroron the split: 74 accepted, 7 wrong (0.095) there, with the other 86 escalated; it counts an item atp == 0.5as the rule answers it, true, and refuses--max-escalationand--max-false-pass.--max-false-passreplaces the band with a cutoff that bounds the label-false items that pass. - No Cross-Group Pooling: Metrics, fits, thresholds, and slice reports are keyed strictly to
(question_id, model_served, primitive);s1 slice answerable/mineand the workbench's Slices tab report each group separately. Two commands are exceptions. Drift reports perquestion_idand pools its models, so a changedmodel_servedshows (CRITICAL); its policy lookup and calibrated fit check stay per group, and it refuses a question whose records name more than one primitive (aquestion_idnames one wire question).s1 comparerefuses a question whose paired records on one side come from more than one model or primitive; split such a file bymodel_served. - Drift Alerts Need Evidence (
s1 drift): PSI of two samples from one distribution grows with smaller samples (about 0.33 at 54 records per side, over theCRITICALthreshold of 0.25), so a PSI over a threshold raises the status only when a permutation test puts it outside sampling noise:CRITICALneeds PSI >= 0.25 and p < 0.01,WARNneeds PSI >= 0.10 and p < 0.05. The p-value (psi_p_value) is a Monte Carlo estimate from 1,000 resamples of the answered records' bin counts (seeded per question; the draws can differ under another numpy, which matters only for a p near alpha). A PSI over a threshold that is not significant is listed innotes, notalerts; each question also reportsn_ref_answered/n_cur_answered, the records the test used.OKdoes not establish stability: small samples miss real shifts (in one measurement, 50 of 120 comparisons with a moderate shift at about 54 records per side stayedOK). The test treats records as exchangeable, so it does not fit autocorrelated windows, and its alpha holds per question and per comparison, not across questions or repeated runs. Model changes,model_requestedmismatches and failure-rate jumps alert as before. The latency summary is informational: a statistic isnullwhen no record has a numericlatency_ms, or when it is not finite (latency_msis not validated, so very large values can overflow it).audit_drift(..., psi_resamples=0)restores the threshold-only rule. s1ops 0.6.0 and earlier alerted on PSI alone. - Out-of-Sample & Model-Mismatch Enforcement (
s1 apply&DecisionGate):- Attempting to apply a calibration fit to any
(dataset, item_id)used durings1 fitexits with code2. - With
--training-manifest, fitting or evaluating a checkpoint on an item it trained on also exits with code2(Fine-Tuning). - Attempting to apply a calibration fit or runtime
DecisionGateto a record whosemodel_serveddiffers from the fitted/compiled model exits with code3(ModelServedMismatchError). For a production file that mixes models,s1 apply --pass-through-unfittedcalibrates the records it has a fit for, writes the others unchanged (except a top-levelstate, as above), and prints how many of each (with the reason per group); class mismatches and in-sample items are still refused. s1 drift --fitands1 export langfuse|phoenix --fitdo the same in memory, so no intermediate file is needed: reports and export files equal the ones made throughs1 apply --pass-through-unfitted, a storedcalibis replaced on a record the fit file has a fit for and kept on any other, in-sample items exit2(--allow-in-sample), and class mismatches exit1. A refusal to apply the fit, and an input line the record reader refuses, name the input file and its own line.s1 drift's report and the printed export summaries gain afitblock with what was calibrated and what was passed through, per file; an export file is unchanged.s1 drift --fitneeds--policyor--calibrated, since a raw comparison without a policy compares and routes no calibrated probabilities. The web workbench does not take a fit. In the SDK:apply_fits(read_records(path), fits, pass_through_unfitted=True).records.
- Attempting to apply a calibration fit to any
- Policy Fingerprint (
policy_sha256): everyPolicyEntrycarries a canonical SHA-256 fingerprint of its routing fields. Verification is opt-in: pass the expected fingerprint toDecisionGate.evaluate_noul(expected_policy_sha256=...)ors1 gate --expected-sha256, and a mismatch raisesPolicyIntegrityError. Loadingpolicy.jsonalso compares each row's stored fingerprint with one recomputed from its fields and refuses a mismatch (PolicyFingerprintError, CLI exit 1, web 400), so a threshold or band edited afters1 policy tunewrote the file is caught at load. To accept a deliberate hand edit, remove that row'spolicy_sha256; a row without one is only recomputed. - Fit Provenance (
fit_sha256):s1 applywrites the fit'sfit_sha256into each record'scalib.s1 policy tune --use-calibrefuses a selected record without calibrated probabilities, a non-nullcalib.fit_sha256that is not a string, and a group whose records were calibrated through different fits, and records the group's hash in the policy (nullwhen every record's hash is missing ornull).s1 policy curve --use-calibrefuses the same records in the groups it selects (exit1), and so does the web workbench's policy simulator (/api/simulate-policy, whoseuse_calibdefaults to true) for the group it simulates (HTTP 400 naming the records file); records with nocalibat all simulate withuse_calib=false. A calibrated policy that records a hash accepts only that fit, always, independent of the opt-in fingerprint check:DecisionGateraisesPolicyIntegrityErrorfor anotherfits.json, ands1 driftand both exporters refuse a record whosecalib.fit_sha256differs, isnull, or is missing (runs1 applyagain with the policy'sfits.json). A policy withfit_sha256: nulltakes any fit. - Strict Inputs: records with NaN or infinite values (in a records file,
NaN,Infinityor a literal that overflows to infinity, such as1e400, anywhere in a row, whether anything reads that field or not; a row already decoded and passed toparse_recordhas only the fields it reads checked), a lone surrogate (half of a UTF-16 pair, which JSON can escape on its own and no UTF-8 file can hold; refused under the same two conditions, so no command writes part of afits.jsonorpolicy.jsonand stops at one), JSON booleans as numbers, or probabilities that do not sum to1 ± 0.02are refused with their file and line number; score keys must be 1–20 ASCII digits naming distinct levels up to2**53("0"and"00"together are refused), and wherever the calibrated distribution is read (s1 drift, both exporters,--use-calib) a calibrated choice or score distribution must have exactly the raw keys.s1 probeandrun_batchhold a states row'slabelto the record rule of every question it goes onto (onelabelgoes onto each question's record, so a row cannot label a noul and a choice in one file), and refuse it naming the row before anything is sent; a lone surrogate in a states row'sitem_id,datasetorsplit, in the model name, or in a question's name or wording is refused there too, since those strings are copied into the records (a question's name into its probe records) and a wording with one cannot be given aquestion_id. A customBackend'sanswercontributes only the answer keys (p;choice,probabilities,confidence;score,probabilities,confidence), and a response that would not read back as a record is recorded asinvalid_response, once its cost is settled, so a run leaves only records the reader takes (acost_usdthe ledger refuses stops the run, with no record of that call).s1 gate --p/evaluate_noul(p_raw=...)takes a finite number in[0, 1](not a boolean), and--probs/probs_rawat least two classes summing to1 ± 0.02, renormalized before routing.policy.jsonandfits.jsonhold one entry per group; anescalation_bandneeds0 <= band_lo <= band_hi <= 1. Malformed input raisesValueErrorin the SDK and exits1in the CLI. - Reserve-Then-Settle Budget Ledger (
s1 ledger&s1 probe): Uses exclusiveflockon<journal>.lock, rejects hard-linked ledgers, and halts immediately if a measured cost exceeds its reservation (LedgerUnusable).
Fine-Tuning Around an External Trainer
s1ops starts no training. It writes down what a checkpoint will train on and refuses to fit or evaluate that checkpoint on those items. The trainer is yours; Kev's kev.train --data takes the rows the recipe below writes.
# These run as written on the offline mock, which answers the same whatever the name, so its results judge nothing. Against a
# real server, add to every probe: --backend systemone --base-url URL --usd-per-input-token RATE (see Live probes).
#
# questions.json holds the one question these labels answer, here {"tickets": {"type": "noul", "instructions": "..."}}. One
# `label` goes onto every question in the file, so `s1 probe` refuses a label that does not suit them all, which the demo's
# questions.json, mixing noul, choice and score, would be.
# 1. Give each states row a `split` (train, cal, eval); `s1 probe` copies it into its records, as it does `label`:
# {"dataset": "tickets", "item_id": "t1", "state": "...", "split": "train", "label": true}
s1 probe questions.json --no-expand-probes --model current-1 --states states_local.jsonl -o current.jsonl --ledger ledger.jsonl
# 2. Name the new checkpoint and write what it trains on. `--parent previous_manifest.json` adds what a checkpoint it
# starts from trained on.
s1 trainset export current.jsonl --split train --checkpoint tickets-r1 -o training_manifest.json
# 3. Build train.jsonl with the recipe below, train outside s1ops, then serve the checkpoint under that name and probe the
# held-out items, with the cal and eval rows of the states file in files of their own. Probe the current model on the
# same eval rows for step 4's drift.
python3 -c 'import json, sys; sys.stdout.writelines(r for r in sys.stdin if r.strip() and json.loads(r).get("split") == sys.argv[1])' cal < states_local.jsonl > states_cal.jsonl
python3 -c 'import json, sys; sys.stdout.writelines(r for r in sys.stdin if r.strip() and json.loads(r).get("split") == sys.argv[1])' eval < states_local.jsonl > states_eval.jsonl
s1 probe questions.json --no-expand-probes --model current-1 --states states_eval.jsonl -o current_eval.jsonl --ledger ledger.jsonl
s1 probe questions.json --no-expand-probes --model tickets-r1 --states states_cal.jsonl -o candidate_cal.jsonl --ledger ledger.jsonl
s1 probe questions.json --no-expand-probes --model tickets-r1 --states states_eval.jsonl -o candidate_eval.jsonl --ledger ledger.jsonl
# 4. Promotion check. A new name starts with no fit and no policy: the current model's `s1 apply` exits 3 on it, and `DecisionGate`
# raises `ModelServedMismatchError`.
s1 compare current.jsonl candidate_eval.jsonl --training-manifest training_manifest.json
s1 fit candidate_cal.jsonl --training-manifest training_manifest.json -o fits.json
s1 apply candidate_eval.jsonl --fit fits.json -o candidate_eval_calibrated.jsonl
s1 policy tune candidate_eval_calibrated.jsonl --use-calib --training-manifest training_manifest.json -o policy.json
s1 drift current_eval.jsonl candidate_eval.jsonl
s1 drift across a rename reports every question CRITICAL with the alert that model_served changed, and --fail-on-critical exits 4, so read its PSI and quantiles here rather than its status. A later run against the promoted checkpoint's own records is the monitor. A probe's states row holds one label, and it goes onto every question in the questions file, so when the labels differ give each question its own questions file and states file.
The manifest (s1-training-manifest/1) holds (dataset, item_id), question_id and label for each row, plus the checkpoint's name, the split and the records file's SHA-256. It holds no state text and no question wording, so s1 probe and s1 labelaudit sheet stay the only readers of a states file. The trainer joins them from its own local files, checking the wording by question_id:
import json
from s1ops import question_id
from s1ops.lint.loader import read_question_entries
from s1ops.monitor.runner import probe_tasks
manifest = json.load(open("training_manifest.json", encoding="utf-8"))
entries, errors = read_question_entries("questions.json") # read as `s1 probe` reads the file
if errors:
raise SystemExit("; ".join(errors))
tasks = probe_tasks(entries, probe_mode=False)
wording = {question_id(t.question): t.question.wire_dict() for t in tasks}
states = {}
for line in open("states_local.jsonl", encoding="utf-8"):
row = json.loads(line)
states[(row.get("dataset", "default"), str(row["item_id"]))] = row["state"]
rows = {}
for item in manifest["items"]:
key = (item["dataset"], item["item_id"])
row = rows.setdefault(key, {"state": states[key], "questions": {}})
row["questions"][item["question_id"]] = {**wording[item["question_id"]], "label": item["label"]}
with open("train.jsonl", "w", encoding="utf-8") as f:
for row in rows.values():
f.write(json.dumps(row, ensure_ascii=False) + "\n")
The refusal. --training-manifest is repeatable on s1 fit, s1 audit, s1 policy tune, s1 policy curve and s1 compare. It exits 2 when a group those filters select is served under the manifest's checkpoint and holds an item it trained on; s1 compare checks each pair against the model of its side, and a candidate's training item the other file lacks is not compared. An item is its (dataset, item_id) across questions, because the model saw the state and rewording a question changes its question_id. Another model's answers on those items pass. A manifest whose checkpoint answered no record in the input file exits 1, which catches a misspelled name; it does not promise that the rows selected include that model, so --model current-1 on a mixed file checks no candidate row.
What it does not cover:
- The check runs only in a command given the flag.
s1 apply,s1 drift,s1 slice,s1 labelauditand the workbench do not take it, ands1 policy tune --use-calibdoes not look behind a storedcalib: ans1 fiton trained items without the flag, followed bys1 applyon clean items, passes a checkedtune.fits.jsondoes not record that a check ran. --parentcovers ancestors exported through s1ops. A released base checkpoint's training set is unknown here.- Identity is exact. A near-duplicate or another item from a trained source is not refused, though it can inflate a score the same way.
--modelis your claim about what answers. A server such askev.serveechoes the requested name without checking it, so name every checkpoint differently: fits, policies and drift's model comparison key on that name and see two checkpoints under one name as one model. A name is 1-128 characters, an ASCII letter or digit and then ASCII letters, digits or._:/@+-, the form a live backend accepts asmodel_served.s1 trainset exportreportsanswered_as_checkpointwhen the source file already holds records under the name.
CLI Exit Codes
| Code | Meaning |
|---|---|
0 |
Success |
1 |
Usage error, missing or malformed input, no matching groups, lint check --strict findings, policy fingerprint mismatch, ledger error, s1 demo target not new or empty |
2 |
In-sample refusal: s1 apply, or s1 drift or s1 export with --fit, on an item the fit used (override with --allow-in-sample), or --training-manifest on an item the checkpoint trained on |
3 |
model_served mismatch against the fit or policy (s1 apply without --pass-through-unfitted, s1 gate) |
4 |
s1 drift --fail-on-critical found a CRITICAL question |
Errors print a single s1: <message> line on stderr; non-printable characters in it (a newline or an escape sequence inside an id or a path) are written escaped, as \n or \x1b, while printable text, non-ASCII included, is kept. When an input that another command produces is missing, the message names that command (for example run \s1 fit` first). Set S1_DEBUG=1to re-raise errors raised by the library, with their traceback; checks the CLI makes itself (a missing input file, an empty grades file) still print one line. A missing input path is an error everywhere, includings1 ledger status, except s1 lint check, which reports an unreadable questions file as a JL000 finding in its normal (--format json`) report.
Development
From a source checkout (the sdist includes the test suite), all tests run without API keys or calls to external services (a few backend tests talk to an HTTP server on 127.0.0.1):
uv sync --all-extras
uv run pytest -v
Metadata
Release files for s1ops 0.10.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| s1ops-0.10.0.tar.gz | 337.6 kB | Without provenance | Details for s1ops-0.10.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| s1ops-0.10.0-py3-none-any.whl | Python 3 | none | any | Details for s1ops-0.10.0-py3-none-any.whl |
Total release size: 511.2 kB
Release files / s1ops-0.10.0.tar.gz
| Download URL | s1ops-0.10.0.tar.gz |
|---|---|
| Size | 337.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
013498c05a3c5531270e42c1e23f642a31b2877e761de3f570797e52e86dc90c
|
|
BLAKE2b-256 checksum How to use checksums |
4cda2f78d6c54d99caa91acc09ad5edb18fa2baea6f9289c0069c5a052235550
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / s1ops-0.10.0-py3-none-any.whl
| Download URL | s1ops-0.10.0-py3-none-any.whl |
|---|---|
| Size | 173.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bd2a533d8ef80bae3cd645caccabe761c9bbaef0f41156f97ae51639cbd36079
|
|
BLAKE2b-256 checksum How to use checksums |
93fd380aba9b6f6460abb38f7dcccb6c3ce75f9870048ea060fe9310dfb287d8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Security for s1ops 0.10.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.