RSI-Jev
A recursively self-improving research loop that builds Jev-style System One decision models.
🌐 shanghua-gao.github.io/RSI-Jev
🤗 Models · Docs · Every experiment · Release notes
AI agents propose the hypotheses, register their predictions before spending GPU time, run the experiments, and retire their own champions when the evidence says to. Every release, and every experiment that failed on the way, is published with its numbers. The loop is the next version of AutoScientists.
- 9 releases in 41 days, from v1.0 to v7.0-VL, each trained, evaluated and documented by the loop.
- 801 experiments, every one written up, failures included: EXPLORE.md.
- v6.1-VL 27B has the highest score on the Decision Index 0.3 public set: 65.64, with
--effort auto. The official score is pending. - v7.0-VL 4B scores 60.63 on the public Decision Index 0.3, up from 50.98 for v6.1-VL 4B.
- v7.0-VL 3B, the 4B cut at layer 20, scores 60.01 at two thirds of its latency.
What it builds
Models that answer a yes/no, pick-one-of-k or rate-on-a-rubric question about a document, a chat or an image. One forward pass returns a calibrated probability for every option. Nothing is generated, so there are no reasoning tokens to spend.
What v7.0-VL can spend is depth. It has answer heads at layers 16, 20 and 32, and effort picks
how many layers a request uses:
| effort | layers | median / mean latency | use it for |
|---|---|---|---|
low |
16 | 24 / 65 ms | intent, routing, retrieval |
medium |
20 | 28 / 78 ms | classification and judgement |
high |
32 | 42 / 117 ms | knowledge, multi-step reasoning, code |
auto |
16, 20 or 32 | 41 / 85 ms | mixed traffic: stops at the first layer that is confident enough |
v7.0-VL 3B is the same model cut at layer 20: low stops at 16, medium and high at 20.
v6.1-VL 27B has the same levels at layers 48, 56 and 64.
Fast when it's easy, deep when it's hard. Which tasks need which depth: docs/effort-depth.md.
Capsules photo from VisA (Zou et al. 2022, CC BY 4.0, resized).
Quickstart
pip install "rsi-jev[fast,vision] @ git+https://github.com/Shanghua-Gao/RSI-Jev"
rsi-jev serve v7.0-vl-4b --effort auto --port 8000
rsi-jev serve v7.0-vl-3b serves the 3B (6.8 GB). On a 96 GB GPU, rsi-jev serve v6.1-vl-27b --effort auto serves the 27B.
On an Apple Silicon Mac (the 3B runs in about 2 GB) or on Linux, nothing else needs to be installed first:
curl -fsSL https://github.com/Shanghua-Gao/RSI-Jev/releases/latest/download/install.sh | sh, then
git diff | decide "Does this change touch authentication?". Building on it: decideit;
on a Mac, see also the Mac guide.
curl localhost:8000/v1/systemone -H 'Content-Type: application/json' -d '{
"model": "jev-latest",
"state": "I was charged twice. Please refund.",
"questions": {"refund": {"type": "noul", "instructions": "Does the user request a refund?"}}
}'
Or in Python, with no server:
from rsijev import Decider
d = Decider("v7.0-vl-4b")
d.decide("Customer photo: <image>\nThe customer says it arrived damaged.",
{"damaged": {"type": "noul", "instructions": "Does the photo show visible damage?"}},
images=["photo.jpg"])
The server speaks Jev's API, so an existing Jev client works once its base URL points here. Every
response reports which layer answered and how confident it was. Options and limits:
serve/README.md; setups and what each costs: docs/inference.md.
Releases
| Model | Input | Decision Index 0.3 | Decision Index 0.2.1 | Held-out set | |
|---|---|---|---|---|---|
| v7.0-VL-4B | text, images | 60.63 | 59.63 | 0.811 | 🤗 |
| v7.0-VL-3B | text, images | 60.01 | 58.84 | 0.797 | 🤗 |
v6.1-VL-27B, auto |
text, images | 65.64 | 64.92 | 0.808 | 🤗 |
| v6.1-VL-4B | text, images | 50.98 | 50.74 | 0.729 | 🤗 |
| v6.0-VL-4B | text, images | 46.23 | 46.24 | 0.698 | 🤗 |
| v5.0-VL-3B | text, images | – | 38.38 | 0.689 | 🤗 |
| v4.0-VL-2B | text, images | – | – | 0.653 | 🤗 |
| v3.0-2B | text | – | – | 0.649 | 🤗 |
| v2.1-2B | text | – | – | 0.633 | 🤗 |
| v2.0-2B | text | – | – | – | 🤗 |
| v1.0-2B | text | – | – | 0.604 | 🤗 |
| v1.0-0.8B | text | – | – | – | 🤗 |
Decision Index is the public benchmark, full runs of the default configuration (the 27B with
--effort auto, which we recommend): the 0.3 public
index (public set; v6.1-VL 4B scored with the pre-release 0.3 kit) and 0.2.1. The held-out set
is our zero-shot check: no release trained on it. The 4B and smaller are fine-tuned from Qwen3.5
Base; v6.0-VL and v6.1-VL 4B run the whole Qwen3.5-4B-Base and can answer at layer 16, 20 or 32
of it. v7.0-VL 4B starts from the instruction-tuned Qwen3.5-4B, repeats the options and reads
them with attention that runs both ways over the prompt; it answers at layer 16, 20 or 32 the same
way. v7.0-VL 3B is v7.0-VL 4B cut at layer 20, with no retraining. v6.1-VL 27B is fine-tuned from Qwen3.8-27B and answers at layer 48, 56 or 64. Every
number, caveat and per-benchmark score is in the release records.
- 2026-11-06 · v7.0-VL 3B is v7.0-VL 4B cut at layer 20, no retraining: the 4B's answers at
medium, 3.30B parameters. Decision Index 0.3 60.01. - 2026-11-05 · v7.0-VL 4B reads the options in a new way (option echo, prefix-bidirectional attention) and is trained once on 2.76M examples. Decision Index 0.3 50.98 → 60.63.
- 2026-10-08 · v6.1-VL 27B brings the same method to Qwen3.8-27B. Decision Index 0.3 public set 65.64 with
auto. - 2026-10-07 · v6.1-VL 4B averages v6.0-VL with a second fine-tune of the same base. Decision Index 0.3 46.23 → 50.98.
- 2026-10-06 · v6.0-VL 4B chooses its depth per question. Decision Index 38.38 → 46.24.
- 2026-10-02 · v5.0-VL 3B cuts the model to the first 20 of 32 layers and says "unknown" when a question has no answer.
- 2026-10-01 · v4.0-VL reads images, with text held at v3.0's level.
- 2026-09-28 · v3.0 is the first release where reinforcement learning helps.
How it got here
The loop's first three releases: each grey dot is an experiment that did not beat the champion.
The experiments that made a release are in docs/history.md; all the others, with why each failed, are in EXPLORE.md; the reinforcement-learning arms are in docs/rl.md.
Each experiment that moved a release, from v1.0 to v7.0-VL
| where it went | result | what it established |
|---|---|---|
| v1.0 | typed-decisions 0.662 | the bar to clear, and it had never seen a benchmark's train split |
| train on the benchmark's own train split | typed-decisions 0.7794 | worth +0.12 — and it costs general knowledge, so it was rejected |
| the same + 15% general-knowledge replay | typed-decisions 0.796 | the replay pays that cost back. The first keeper |
| the other benchmarks' train splits too, 3k per source | suite 0.7049 | the lever is not specific to one benchmark |
| a per-question confidence head, forbidden to sharpen score questions | suite ECE 0.0736 | confidence that means something, and it never changes which option wins. Shipped |
| v2.0 | suite 0.7056, ECE 0.0546 | capability was v1.0's lever; what it trains on, and calibrating it afterwards, is a second one — and free at inference. But the gain was on the benchmarks it had trained for |
| the replay weighted toward science, half of it recast to ten options | MMLU-Pro 0.331 | more and better-shaped data barely moves general knowledge: +0.008 |
| freeze the tower's lower third instead | MMLU-Pro 0.339 | holding those layers still does recover some of it — and breaks three of the decision benchmarks, because they do need to adapt |
| the bottom 8 layers at one tenth the learning rate | MMLU-Pro 0.359 | it was never a data problem. Fitting decisions through the whole tower at one rate was overwriting what the knowledge sat in — frozen layers cannot adapt, slow ones can |
| v2.1 | suite 0.7291, MMLU-Pro 0.383 | and it travels: ahead of v1.0 on all three benchmarks it never trained on, where v2.0 was not |
| wider coverage (data-scale-2): +84k questions, emotion/SST-5/ANLI and more tasksource and Open-Jev train items | suite +0.011 / +0.013 on two seeds | coverage is still the lever that moves the suite |
| a two-layer MLP combining the option and cross-attention features | suite 0.7589 on the new 15-benchmark suite | the readout had a little left in it: +0.002 |
| a listwise reranking reward (NDCG@5 over 16 candidates) on top, against a KL anchor | hippo R@1 0.192 → 0.308, suite −0.003 | the first RL stage kept: a reward over the order of candidates carries what per-item labels cannot |
| v3.0 | suite 0.756, ECE 0.066 | the first release where RL works: +60% reranking R@1 over its supervised parent, the rest held level |
| image training on the train splits of twelve public datasets, with text replay | held-out image top-1 0.809 against 0.796 for a text-only control | the model reads images through the base model's frozen vision tower |
| generated reasoning questions with soft targets where the model is at chance | held-out ECE after calibration, BBH excluded, 0.096 → 0.075 | the image rounds had made it overconfident; this repairs most of it |
| an RL stage charging a confident mistake 4x a timid correct answer | held-out ECE before calibration 0.200 → 0.082 | against a matched supervised stage: better accuracy and calibration, no detectable ranking difference (one seed) |
| v4.0-VL | image top-1 0.803 held out, suite 0.756, ECE 0.043 | reads images, with text held at v3.0's level |
| the same recipe on Qwen3.5-4B, read at layer 16, 20 or 24 of its 32 | suite 0.744 · 0.760 · 0.760 | a System One model doesn't need the deep layers: quality flattens by layer 20 |
| images on the layer-20 model, plus questions whose right answer is "unknown" | KoBBQ unknown-when-ambiguous 0.679 → 0.891 | images cost the model its "unknown" answer until the data asked for it |
| pool each option over its own tokens, not the separator that follows the previous one | CLINC150 0.383 → 0.753 | a readout bug found by an outside benchmark; the same weights read correctly once each option is pooled over its own tokens |
| retrain only the decision head for that readout, tower frozen, 600 steps | final ECE after calibration 0.080 → 0.042, Decision Index 38.38 | the head had learned the old pooling; a short head-only stage recovers the calibration |
| v5.0-VL | MMLU-Pro 0.429, Decision Index 38.38, KoBBQ unknown 0.932 | 3B: the first 20 of 32 layers |
| early-exit heads that read a detached copy of their layer, so no gradient from them reaches the trunk | MMLU-Pro 0.443 at layer 32 | retuning heads on a trunk trained with attached exits had not recovered depth: the trunk had lost it |
| 10,000 steps on a cleaned corpus instead of 40,000 on the old one | held-out +0.026, MMLU-Pro +0.043 | longer training overfits the suite; shorter and cleaner wins held out |
| one temperature per exit and an exit policy fitted on held-out proxies of the test mix | every release bar passes at 20.9 layers on average | policies tuned on in-distribution rows stop too early on hard questions |
| v6.0-VL | Decision Index 46.24, MMLU-Pro 0.440 | 4B: answers at layer 16, 20 or 32, whichever is confident first |
| v6.0-VL averaged, weight 0.5 each, with a second fine-tune of the same base trained on other data | suite 0.793 and held-out 0.728 at layer 32, against 0.770 and 0.695 | the average beats both members; the gain is in the tower, the heads add nothing |
| one temperature per exit and both thresholds chosen again for the average | auto at 0.85 / 0.50 matches layer 32 at 19.5 layers |
averaging shrinks the logits: at v6.0-VL's temperatures it was underconfident; calibration ends worse than v6.0-VL's (final ECE 0.048 vs 0.036) |
| v6.1-VL | Decision Index 0.3 50.98, held-out 0.729 | 4B: v6.0-VL averaged with a second fine-tune |
| two LoRA fine-tunes of Qwen3.8-27B, averaged weight 0.5 each, chosen from 9 full Decision Index 0.3 reads | Decision Index 0.3 65.59, against 64.83 and 65.30 for the two members | averaging carries over to 27B; uneven weights and a third ingredient read 65.37 to 65.58 |
| short inputs right-padded to a multiple of 64 tokens and replayed as CUDA graphs | short one-question requests in about half the time | at one request a time the 27B is bound by kernel launches, not compute |
| v6.1-VL 27B | Decision Index 0.3 public set 65.64 with auto, held-out 0.808 |
27B: the same method on Qwen3.8-27B, answers at layer 48, 56 or 64 |
| repeat the option list once and read the second copy; let the prompt and the options attend both ways | our 4B pick set +3.5 and +2.6, +3.8 over v6.1-VL together with an option-order consistency loss | what the readout sees mattered more than ~150 later loss and schedule variants |
| start from the instruction-tuned model, keep the base model's reasoning traces as a training signal | held level at 4B, +1.78 at 2B | reasoning stays a training-time teacher; nothing is generated at inference |
| data rounds 7 to 12: each round's weakest families get training data built from public train splits | Decision Index 0.3 53.06 → 60.62 | targeted data moves its targets and dilutes the rest; rounds are judged per benchmark |
| v7.0-VL | Decision Index 0.3 60.63, held-out 0.811 | 4B: option echo and prefix-bidirectional attention, instruction-tuned start, one end-to-end run |
| v7.0-VL 3B | Decision Index 0.3 60.01, held-out 0.797, median latency 19.6 ms against 29.4 ms | 3B: v7.0-VL 4B cut at layer 20, no retraining |
Documentation
| Inference guide | setups, measured latency, speed-ups |
| Mac (MLX) | the 8-bit and 4-bit Apple-silicon builds and what each costs |
| Effort and depth | which tasks and questions need the deep layers |
| HTTP API | the Jev-compatible server |
| Benchmarks | what each number means and how to reproduce it |
| Release records | how each release was built, what it scores, its limitations |
| Reinforcement learning | where RL beat supervised training, and where it didn't |
| Speed | how inference was made faster |
| Code | training and the files the loop may rewrite |
Contributing
A case where the model is confidently wrong, or a variant you tried that failed, is the most useful
thing you can send: report a wrong answer
· report an experiment.
Each becomes a registered prediction tested in the next version. See
CONTRIBUTING.md. To collaborate or support the work with compute,
contact Shanghua Gao.
Acknowledgements
Thanks to HP and NVIDIA for providing the HP ZGX Nano AI Station, powered by the NVIDIA GB10 Grace Blackwell Superchip.
License
Code: MIT. Weights: Apache-2.0, following the base model; some image training sources are
non-commercial, listed on each model card. Not affiliated with TypeSafe AI. To cite, use
CITATION.cff.
Metadata
Release files for decideit 7.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| decideit-7.0.1.tar.gz | 523.9 kB | Without provenance | Details for decideit-7.0.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| decideit-7.0.1-py3-none-any.whl | Python 3 | none | any | Details for decideit-7.0.1-py3-none-any.whl |
Total release size: 906.6 kB
Release files / decideit-7.0.1.tar.gz
| Download URL | decideit-7.0.1.tar.gz |
|---|---|
| Size | 523.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
94f3718e36a3b1738661b5d58da3db04282edd24030fe2e92f0e3f79af9e063d
|
|
BLAKE2b-256 checksum How to use checksums |
2e3fccd73c13276167c5d527e1155824de52b3b53705dafcc3d9bfb167841ceb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.7
|
Release files / decideit-7.0.1-py3-none-any.whl
| Download URL | decideit-7.0.1-py3-none-any.whl |
|---|---|
| Size | 382.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6b6d74a37835076ec9b057db360401f8c426c77fc3fc867dd71530046aac7474
|
|
BLAKE2b-256 checksum How to use checksums |
bf626720bb4b204455eda5983c851a3e2fb9e0b8f026963b44f486bc2f78d879
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.7
|
Security for decideit 7.0.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.