AnyJev

English | 简体中文

Turn any LLM into a Jev-style decision model. Typed decisions, real probabilities, no training.

Give it a state and typed questions, get back a decision and a probability per question in one prefill, no generation. Works with the model you already run.

What makes it different from reading logits with max_tokens=1:

• Training-free debiasing on by default (L0). Cyclic-shift marginalization removes option-order bias; a label-free prior estimate removes the model's label bias. Zero labels.
• Post-hoc calibration when you have labels (L1). Temperature scaling per (model, question), stored as a small artifact.
• Every result says which one you got. decision.level is raw, L0, or L1. Downstream code can refuse to act on the wrong one.
• A benchmark that reports calibration. Accuracy, Brier, ECE, order-flip rate, coverage at 5% risk, from one command.

Not affiliated with, endorsed by, or derived from TypeSafe AI or Jev. All comparisons are measured and reproducible from bench/results/.

Install

pip install -e ".[hf]"          # library + transformers backend
pip install -e ".[hf,bench,dev]" # plus datasets and pytest

Ten lines

from anyjev import Decider, Question
from anyjev.backends.hf import HFBackend

d = Decider(HFBackend("Qwen/Qwen3-8B"))

route = Question.choice("Which handler should process this request?",
                        ["billing", "technical", "sales", "other"], name="route")
safe  = Question.noul("Is the proposed tool call destructive or irreversible?", name="safe")
done  = Question.score("How complete is the task on a 0 to 1 scale?", bins=5, name="done")

state = {"conversation": [...], "proposed_tool_call": {...}}
r = d.decide(state, [route, safe, done])

r["route"].argmax          # "billing"
r["route"].distribution    # {"billing": 0.81, "technical": 0.07, ...}
r["safe"].p_true           # 0.12
r["done"].value            # 0.35
r.level                    # "L0"  (debiased, not calibrated)

With labels:

art = d.calibrate(safe, calib_states, calib_labels)   # ~100 to 500 examples -> L1 artifact
r = d.decide(state, [safe], level="L1")

Why L0 is not optional

A noul question, "Is this email spam?". Raw readout on the email gives P(Yes) = 0.62. The same prompt with the email replaced by N/A gives P(Yes) = 0.70: the model leans Yes regardless of content. Divide by that prior and renormalize, and the answer is P(Yes) = 0.41. The judgment flips. Position bias does the same thing to choice questions when you reorder the options. Both are fixable without training, and AnyJev fixes them by default.

Levels

Full contract in docs/levels.md.

Benchmark

python -m bench.run --model Qwen/Qwen3-8B --tasks newsgroups,injection,banking20 --n 300 --calib 200

Results land in bench/results/<date>/ as Markdown and JSON with hardware and library versions. python -m bench.table bench/results/<date> regenerates the table below; nothing is typed in by hand.

Reorder the options and one in five answers changes

Three open models, three tasks, 300 test items each. flip is the fraction of items whose answer changes when the option list is reversed (choice) or the phrasing is swapped between "Yes or No" and "No or Yes" (noul). raw is what every logit-reading clone does. L0 is AnyJev's default, zero labels. L1 adds one temperature fit on 200 labels.

Full table with every ablation row (permutation only, each prior alone, Brier, coverage at 5% risk): docs/results_bench.md. One H100, bf16, transformers 4.55.4; regenerate with python -m bench.table bench/results_batchprior_v0/2026-09-20.

On Laya's own benchmark, zero-shot

Laya's headline is 0.766 on this set, above Jev's 0.727. All rows except Jev were measured here on the same 2,000 decisions; the Laya fine-tuned checkpoint reproduces its published number. Read it two ways. On argmax accuracy, Laya fine-tuned on the train split wins, and a 32B open model with zero training is 2.7 points behind Jev. On the probabilities, which is what a System One model is for, the order flips: the fine-tuned Laya's soft accuracy (0.471) is below every Qwen row, and its ECE (0.215) is six times AnyJev L1's (0.034). Laya's zero-shot checkpoints, the ones you would use on a question they were not trained for, score 0.34 to 0.36 against a 0.32 random baseline. Full table with per-workflow and per-type breakdown: docs/results_typed.md; regenerate with python -m bench.run_typed --model <model> and python -m bench.providers.laya.

Inside NanoJev's maze harness

NanoJev's README compares its trained 0.6B model against "Untuned Qwen3-0.6B", which reads A/B logits for four Boolean questions per maze cell. We ran NanoJev's own frozen exploration code on its 15 test and out-of-distribution mazes and only swapped the engine. Two things are true at once. The untuned model's result depends heavily on how you read it: the same Qwen3-0.6B goes from 13/15 mazes and 20,555 attempts under NanoJev's A/B readout to 15/15 and 5,825 attempts under AnyJev's raw Yes/No readout. And no LLM readout, not even Qwen3-8B, answers "is one step north clear?" better than always saying the majority label (edge accuracy 0.42 to 0.56 against a 0.54 to 0.62 majority). The maze differences come from how each readout's average probability interacts with the controller's p >= 0.5 probe rule, not from map reading. We report it because it is the comparison NanoJev invites; we do not headline it. Full table: docs/results_maze.md.

Status and roadmap

v0.0.1. The library, both backends, and all three benches are real and measured; the tables above are regenerated from committed JSON. Actively developed: the plan with dates is in ROADMAP.md (next up: PyPI release, span readout for more than 26 options, conformal abstention, latency column, a live demo, Llama and Gemma rows; then a Jev-compatible server, more backends, and multimodal state). Backends and bench providers are one file each; see CONTRIBUTING.md. What landed: CHANGELOG.md. Who we build on: CREDITS.md.

License

Apache-2.0. Datasets keep their own licenses, see THIRD_PARTY.md.