Skip to main content

RSI-Jev

A recursively self-improving research loop that builds Jev-style System One decision models.

🌐 shanghua-gao.github.io/RSI-Jev

🤗 Models · Docs · Every experiment · Release notes

MIT Apache-2.0 weights Jev-compatible API

AI agents propose the hypotheses, register their predictions before spending GPU time, run the experiments, and retire their own champions when the evidence says to. Every release, and every experiment that failed on the way, is published with its numbers. The loop is the next version of AutoScientists.

  • 9 releases in 41 days, from v1.0 to v7.0-VL, each trained, evaluated and documented by the loop.
  • 801 experiments, every one written up, failures included: EXPLORE.md.
  • v6.1-VL 27B has the highest score on the Decision Index 0.3 public set: 65.64, with --effort auto. The official score is pending.
  • v7.0-VL 4B scores 60.63 on the public Decision Index 0.3, up from 50.98 for v6.1-VL 4B.
  • v7.0-VL 3B, the 4B cut at layer 20, scores 60.01 at two thirds of its latency.

Decision Index 0.3 public set: RSI-Jev v6.1-VL 27B 65.64, Torchcast Decision 27B 65.10, Perplexity Decider v1.1 (27B) 62.25, Cloudflare clef 61.71, Jev 57.96, RSI-Jev v6.1-VL 4B 50.98

What it builds

Models that answer a yes/no, pick-one-of-k or rate-on-a-rubric question about a document, a chat or an image. One forward pass returns a calibrated probability for every option. Nothing is generated, so there are no reasoning tokens to spend.

What v7.0-VL can spend is depth. It has answer heads at layers 16, 20 and 32, and effort picks how many layers a request uses:

effort layers median / mean latency use it for
low 16 24 / 65 ms intent, routing, retrieval
medium 20 28 / 78 ms classification and judgement
high 32 42 / 117 ms knowledge, multi-step reasoning, code
auto 16, 20 or 32 41 / 85 ms mixed traffic: stops at the first layer that is confident enough

v7.0-VL 3B is the same model cut at layer 20: low stops at 16, medium and high at 20. v6.1-VL 27B has the same levels at layers 48, 56 and 64.

Fast when it's easy, deep when it's hard. Which tasks need which depth: docs/effort-depth.md.

A tray of gel capsules, one leaking: defective, 68% sure A delete-file dialog; the user said keep the file: Cancel, 99% sure
Capsules photo from VisA (Zou et al. 2022, CC BY 4.0, resized).

Quickstart

pip install "rsi-jev[fast,vision] @ git+https://github.com/Shanghua-Gao/RSI-Jev"
rsi-jev serve v7.0-vl-4b --effort auto --port 8000

rsi-jev serve v7.0-vl-3b serves the 3B (6.8 GB). On a 96 GB GPU, rsi-jev serve v6.1-vl-27b --effort auto serves the 27B.

On an Apple Silicon Mac (the 3B runs in about 2 GB) or on Linux, nothing else needs to be installed first: curl -fsSL https://github.com/Shanghua-Gao/RSI-Jev/releases/latest/download/install.sh | sh, then git diff | decide "Does this change touch authentication?". Building on it: decideit; on a Mac, see also the Mac guide.

curl localhost:8000/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "jev-latest",
  "state": "I was charged twice. Please refund.",
  "questions": {"refund": {"type": "noul", "instructions": "Does the user request a refund?"}}
}'

Or in Python, with no server:

from rsijev import Decider

d = Decider("v7.0-vl-4b")
d.decide("Customer photo: <image>\nThe customer says it arrived damaged.",
         {"damaged": {"type": "noul", "instructions": "Does the photo show visible damage?"}},
         images=["photo.jpg"])

The server speaks Jev's API, so an existing Jev client works once its base URL points here. Every response reports which layer answered and how confident it was. Options and limits: serve/README.md; setups and what each costs: docs/inference.md.

Releases

Model Input Decision Index 0.3 Decision Index 0.2.1 Held-out set
v7.0-VL-4B text, images 60.63 59.63 0.811 🤗
v7.0-VL-3B text, images 60.01 58.84 0.797 🤗
v6.1-VL-27B, auto text, images 65.64 64.92 0.808 🤗
v6.1-VL-4B text, images 50.98 50.74 0.729 🤗
v6.0-VL-4B text, images 46.23 46.24 0.698 🤗
v5.0-VL-3B text, images – 38.38 0.689 🤗
v4.0-VL-2B text, images – – 0.653 🤗
v3.0-2B text – – 0.649 🤗
v2.1-2B text – – 0.633 🤗
v2.0-2B text – – – 🤗
v1.0-2B text – – 0.604 🤗
v1.0-0.8B text – – – 🤗

Decision Index is the public benchmark, full runs of the default configuration (the 27B with --effort auto, which we recommend): the 0.3 public index (public set; v6.1-VL 4B scored with the pre-release 0.3 kit) and 0.2.1. The held-out set is our zero-shot check: no release trained on it. The 4B and smaller are fine-tuned from Qwen3.5 Base; v6.0-VL and v6.1-VL 4B run the whole Qwen3.5-4B-Base and can answer at layer 16, 20 or 32 of it. v7.0-VL 4B starts from the instruction-tuned Qwen3.5-4B, repeats the options and reads them with attention that runs both ways over the prompt; it answers at layer 16, 20 or 32 the same way. v7.0-VL 3B is v7.0-VL 4B cut at layer 20, with no retraining. v6.1-VL 27B is fine-tuned from Qwen3.8-27B and answers at layer 48, 56 or 64. Every number, caveat and per-benchmark score is in the release records.

  • 2026-11-06 · v7.0-VL 3B is v7.0-VL 4B cut at layer 20, no retraining: the 4B's answers at medium, 3.30B parameters. Decision Index 0.3 60.01.
  • 2026-11-05 · v7.0-VL 4B reads the options in a new way (option echo, prefix-bidirectional attention) and is trained once on 2.76M examples. Decision Index 0.3 50.98 → 60.63.
  • 2026-10-08 · v6.1-VL 27B brings the same method to Qwen3.8-27B. Decision Index 0.3 public set 65.64 with auto.
  • 2026-10-07 · v6.1-VL 4B averages v6.0-VL with a second fine-tune of the same base. Decision Index 0.3 46.23 → 50.98.
  • 2026-10-06 · v6.0-VL 4B chooses its depth per question. Decision Index 38.38 → 46.24.
  • 2026-10-02 · v5.0-VL 3B cuts the model to the first 20 of 32 layers and says "unknown" when a question has no answer.
  • 2026-10-01 · v4.0-VL reads images, with text held at v3.0's level.
  • 2026-09-28 · v3.0 is the first release where reinforcement learning helps.

How it got here

The champion line climbs from v1.0 (0.622) to v2.0 (0.709), v2.1 (0.736) and v3.0 (0.756) on the 15-benchmark suite; grey dots are the experiments that did not clear it
The loop's first three releases: each grey dot is an experiment that did not beat the champion.

The experiments that made a release are in docs/history.md; all the others, with why each failed, are in EXPLORE.md; the reinforcement-learning arms are in docs/rl.md.

Each experiment that moved a release, from v1.0 to v7.0-VL
where it went result what it established
v1.0 typed-decisions 0.662 the bar to clear, and it had never seen a benchmark's train split
train on the benchmark's own train split typed-decisions 0.7794 worth +0.12 — and it costs general knowledge, so it was rejected
the same + 15% general-knowledge replay typed-decisions 0.796 the replay pays that cost back. The first keeper
the other benchmarks' train splits too, 3k per source suite 0.7049 the lever is not specific to one benchmark
a per-question confidence head, forbidden to sharpen score questions suite ECE 0.0736 confidence that means something, and it never changes which option wins. Shipped
v2.0 suite 0.7056, ECE 0.0546 capability was v1.0's lever; what it trains on, and calibrating it afterwards, is a second one — and free at inference. But the gain was on the benchmarks it had trained for
the replay weighted toward science, half of it recast to ten options MMLU-Pro 0.331 more and better-shaped data barely moves general knowledge: +0.008
freeze the tower's lower third instead MMLU-Pro 0.339 holding those layers still does recover some of it — and breaks three of the decision benchmarks, because they do need to adapt
the bottom 8 layers at one tenth the learning rate MMLU-Pro 0.359 it was never a data problem. Fitting decisions through the whole tower at one rate was overwriting what the knowledge sat in — frozen layers cannot adapt, slow ones can
v2.1 suite 0.7291, MMLU-Pro 0.383 and it travels: ahead of v1.0 on all three benchmarks it never trained on, where v2.0 was not
wider coverage (data-scale-2): +84k questions, emotion/SST-5/ANLI and more tasksource and Open-Jev train items suite +0.011 / +0.013 on two seeds coverage is still the lever that moves the suite
a two-layer MLP combining the option and cross-attention features suite 0.7589 on the new 15-benchmark suite the readout had a little left in it: +0.002
a listwise reranking reward (NDCG@5 over 16 candidates) on top, against a KL anchor hippo R@1 0.192 → 0.308, suite −0.003 the first RL stage kept: a reward over the order of candidates carries what per-item labels cannot
v3.0 suite 0.756, ECE 0.066 the first release where RL works: +60% reranking R@1 over its supervised parent, the rest held level
image training on the train splits of twelve public datasets, with text replay held-out image top-1 0.809 against 0.796 for a text-only control the model reads images through the base model's frozen vision tower
generated reasoning questions with soft targets where the model is at chance held-out ECE after calibration, BBH excluded, 0.096 → 0.075 the image rounds had made it overconfident; this repairs most of it
an RL stage charging a confident mistake 4x a timid correct answer held-out ECE before calibration 0.200 → 0.082 against a matched supervised stage: better accuracy and calibration, no detectable ranking difference (one seed)
v4.0-VL image top-1 0.803 held out, suite 0.756, ECE 0.043 reads images, with text held at v3.0's level
the same recipe on Qwen3.5-4B, read at layer 16, 20 or 24 of its 32 suite 0.744 · 0.760 · 0.760 a System One model doesn't need the deep layers: quality flattens by layer 20
images on the layer-20 model, plus questions whose right answer is "unknown" KoBBQ unknown-when-ambiguous 0.679 → 0.891 images cost the model its "unknown" answer until the data asked for it
pool each option over its own tokens, not the separator that follows the previous one CLINC150 0.383 → 0.753 a readout bug found by an outside benchmark; the same weights read correctly once each option is pooled over its own tokens
retrain only the decision head for that readout, tower frozen, 600 steps final ECE after calibration 0.080 → 0.042, Decision Index 38.38 the head had learned the old pooling; a short head-only stage recovers the calibration
v5.0-VL MMLU-Pro 0.429, Decision Index 38.38, KoBBQ unknown 0.932 3B: the first 20 of 32 layers
early-exit heads that read a detached copy of their layer, so no gradient from them reaches the trunk MMLU-Pro 0.443 at layer 32 retuning heads on a trunk trained with attached exits had not recovered depth: the trunk had lost it
10,000 steps on a cleaned corpus instead of 40,000 on the old one held-out +0.026, MMLU-Pro +0.043 longer training overfits the suite; shorter and cleaner wins held out
one temperature per exit and an exit policy fitted on held-out proxies of the test mix every release bar passes at 20.9 layers on average policies tuned on in-distribution rows stop too early on hard questions
v6.0-VL Decision Index 46.24, MMLU-Pro 0.440 4B: answers at layer 16, 20 or 32, whichever is confident first
v6.0-VL averaged, weight 0.5 each, with a second fine-tune of the same base trained on other data suite 0.793 and held-out 0.728 at layer 32, against 0.770 and 0.695 the average beats both members; the gain is in the tower, the heads add nothing
one temperature per exit and both thresholds chosen again for the average auto at 0.85 / 0.50 matches layer 32 at 19.5 layers averaging shrinks the logits: at v6.0-VL's temperatures it was underconfident; calibration ends worse than v6.0-VL's (final ECE 0.048 vs 0.036)
v6.1-VL Decision Index 0.3 50.98, held-out 0.729 4B: v6.0-VL averaged with a second fine-tune
two LoRA fine-tunes of Qwen3.8-27B, averaged weight 0.5 each, chosen from 9 full Decision Index 0.3 reads Decision Index 0.3 65.59, against 64.83 and 65.30 for the two members averaging carries over to 27B; uneven weights and a third ingredient read 65.37 to 65.58
short inputs right-padded to a multiple of 64 tokens and replayed as CUDA graphs short one-question requests in about half the time at one request a time the 27B is bound by kernel launches, not compute
v6.1-VL 27B Decision Index 0.3 public set 65.64 with auto, held-out 0.808 27B: the same method on Qwen3.8-27B, answers at layer 48, 56 or 64
repeat the option list once and read the second copy; let the prompt and the options attend both ways our 4B pick set +3.5 and +2.6, +3.8 over v6.1-VL together with an option-order consistency loss what the readout sees mattered more than ~150 later loss and schedule variants
start from the instruction-tuned model, keep the base model's reasoning traces as a training signal held level at 4B, +1.78 at 2B reasoning stays a training-time teacher; nothing is generated at inference
data rounds 7 to 12: each round's weakest families get training data built from public train splits Decision Index 0.3 53.06 → 60.62 targeted data moves its targets and dilutes the rest; rounds are judged per benchmark
v7.0-VL Decision Index 0.3 60.63, held-out 0.811 4B: option echo and prefix-bidirectional attention, instruction-tuned start, one end-to-end run
v7.0-VL 3B Decision Index 0.3 60.01, held-out 0.797, median latency 19.6 ms against 29.4 ms 3B: v7.0-VL 4B cut at layer 20, no retraining

Documentation

Inference guide setups, measured latency, speed-ups
Mac (MLX) the 8-bit and 4-bit Apple-silicon builds and what each costs
Effort and depth which tasks and questions need the deep layers
HTTP API the Jev-compatible server
Benchmarks what each number means and how to reproduce it
Release records how each release was built, what it scores, its limitations
Reinforcement learning where RL beat supervised training, and where it didn't
Speed how inference was made faster
Code training and the files the loop may rewrite

Contributing

A case where the model is confidently wrong, or a variant you tried that failed, is the most useful thing you can send: report a wrong answer · report an experiment. Each becomes a registered prediction tested in the next version. See CONTRIBUTING.md. To collaborate or support the work with compute, contact Shanghua Gao.

Acknowledgements

Thanks to HP and NVIDIA for providing the HP ZGX Nano AI Station, powered by the NVIDIA GB10 Grace Blackwell Superchip.

License

Code: MIT. Weights: Apache-2.0, following the base model; some image training sources are non-commercial, listed on each model card. Not affiliated with TypeSafe AI. To cite, use CITATION.cff.

Metadata

Release files for decideit 7.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for decideit 7.0.1
File Size Uploaded Provenance Actions
decideit-7.0.1.tar.gz 523.9 kB Without provenance Details for decideit-7.0.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for decideit 7.0.1
File Interpreter ABI Platform Actions
decideit-7.0.1-py3-none-any.whl Python 3 none any Details for decideit-7.0.1-py3-none-any.whl

Total release size: 906.6 kB

Release files / decideit-7.0.1.tar.gz

Download URL decideit-7.0.1.tar.gz
Size 523.9 kB
Tags Source
SHA-256 checksum
How to use checksums
94f3718e36a3b1738661b5d58da3db04282edd24030fe2e92f0e3f79af9e063d
BLAKE2b-256 checksum
How to use checksums
2e3fccd73c13276167c5d527e1155824de52b3b53705dafcc3d9bfb167841ceb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Release files / decideit-7.0.1-py3-none-any.whl

Download URL decideit-7.0.1-py3-none-any.whl
Size 382.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
6b6d74a37835076ec9b057db360401f8c426c77fc3fc867dd71530046aac7474
BLAKE2b-256 checksum
How to use checksums
bf626720bb4b204455eda5983c851a3e2fb9e0b8f026963b44f486bc2f78d879
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Security for decideit 7.0.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

7.0.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page