JevBench
Coherence tests for Jev-compatible typed decision models.
Website and leaderboard · PyPI · Reproducing the results · How the data were made
JevBench measures whether a model's answers are consistent with one another, not whether they are right and not how fast they come: it is a test of stability and coherence under meaning-preserving and logically related edits, with no gold labels.
Typed decision models ("System One" models) answer questions about a fixed
state with probabilities over declared answers: a Noul (yes/no), a Choice
over named options, or a Score over ordered levels. Many such models now
exist, trained in very different ways. What they share is the interface:
f(state, question key, instructions, criteria) -> probability distribution over criteria
JevBench tests that interface. It never needs gold labels. It never edits the state. It transforms only the question key, the instructions and the criteria, and checks whether the answers keep the relations that probability and choice theory say they must keep: the same answer after a meaning-preserving edit, complementary probabilities for a question and its negation, Frechet bounds for conjunctions, additivity when options are merged or cloned, independence of irrelevant alternatives, isolation between questions in one batch.
JevBench is not affiliated with or endorsed by TypeSafe AI. "Jev" names the
wire interface (POST /v1/systemone) that JevBench speaks.
Quick start
pip install jevbench # Python 3.10+, depends only on httpx
jevbench suites # the bundled suites: jevbench-mini (default), jevbench-240, examples
jevbench eval --backend fake:coherent --suite examples # seconds: check that everything works
Against any server that speaks the System One wire format:
import jevbench as jb
model = jb.systemone("http://127.0.0.1:8000/v1/systemone", model="open-jev")
jb.evaluate(model, suite="examples") # seconds: check that your model works with JevBench
report = jb.evaluate(model) # jevbench-mini, the suite results are reported on
full = jb.evaluate(model, suite="jevbench-240") # optional: every test, narrower intervals
print(report.table("dimension")) # also "group", "relation", "domain"
report.save("runs/open-jev.json")
jb.evaluate(model, "jevbench-240", domains=["finance_ops", "logistics"]) # some domains (cases)
jb.evaluate(model, dimensions=["MEA"], groups=["CHO.conditioning"]) # some dimensions and groups
jb.evaluate(model, relations=["option_clone", "de_morgan"]) # some relations
jb.estimate("jevbench-240", domains=["finance_ops"]) # requests a run would send
print(jb.compare([report, other_report]))
The same from the shell:
export JEVBENCH_API_KEY=... # if the server needs a bearer token
jevbench eval --url http://127.0.0.1:8000/v1/systemone --model open-jev \
--dimensions MEA --concurrency 16 --out runs/open-jev.json
jevbench report runs/open-jev.json runs/other.json # compare saved reports
-
Models.
jb.systemone(url, model)covers every Jev-compatible server; pass several comma-separated URLs to use replicas of one model in turn. A model in your Python process is tested withjb.from_callable(fn, name="my-model-v1"), wherefn(state, questions) -> answers; pass a list of functions, one per GPU, to use them as replicas. The name keys the request cache, so give each model and version its own (without one, answers are cached for the run only). Seeexamples/in_process_model.py. -
Selection.
domainsselects cases;dimensions,groupsandrelationsselect relations (their union). The overall score is reported only when all five dimensions were tested. Select domains on jevbench-240: jevbench-mini tests each relation on 2 cases per domain, fewer than the 5 tests a relation needs to be scored (min_n). -
Your own data.
jb.evaluate(model, cases="my-cases.jsonl"); generate cases for a new domain withpip install jevbench[datagen]andjevbench datagen(see below). -
Speed. Requests are cached (
.jevbench-cache/). Most Jev servers answer one request at a time, so a run is sped up by replicas: start the model several times (one per GPU) and pass every URL; JevBench keeps one request in flight per replica, which gives exactly the sequential answers. -
One fixed test set. A bundled suite ships one frozen test plan, generated once without any model, with every trial's exact requests. It tests each relation exactly once per case: every relation of jevbench-240 has 240 tests, spread over all cases and, within a domain, as evenly as possible over its questions, with all of each test's variants. A run sends these requests and judges the answers; it draws no random numbers, and settings that would run other tests (another seed, rewrite bank or number of trials) are refused, so everyone runs the same tests. The plan is built against a sealed context: a relation that read model outputs while building a trial could not be frozen. Your own
cases=are built from the seed, which is deterministic too, andjevbench plan <suite dir>freezes a suite of your own. -
Which suite to report. Results are reported on
jevbench-mini, the default: the 240 cases of jevbench-240, each relation tested on 2 cases per domain (24 tests per relation), a ninth of the requests. Every one of its tests is a jevbench-240 test. On the 17 models we ran on both, the overall scores agree (Pearson 0.993, Spearman 0.978, mini minus 240 at most 2.2 points, ranks moved by at most 2), and every pair of models that jevbench-240 separates (95% intervals apart) is ordered the same way by jevbench-mini. Its intervals are about 2.5 times as wide (4.4 against 1.7 points), so read differences of a few points between models as ties, or runsuite="jevbench-240"(optional) to separate them. Even on jevbench-240 a model's domain scores differ from its overall score by about as much as sampling noise, so we do not compare models by domain. -
The demo suite.
examplesis one hand-written customer-support case: a run takes seconds and shows that the tool works with your model. It scores each relation from its one test (min_n1), so its numbers are a demonstration, never a result.
jevbench run is the low-level engine run over a JSONL of cases, kept for
scripts: jevbench run --cases my-cases.jsonl --backend systemone:<model> --url ... --out x.json.
Results
The leaderboard of 21 openly released models, with every score, interval and accuracy, and the full reports behind them, is on the JevBench website. docs/reproducing.md explains how to recompute it from the published reports or run any of the models again.
Connecting your model
JevBench reads the standard Jev format and nothing else: a model that returns another format is asked to adapt, with a message that says how. Check a model before a full run:
jevbench check --url http://127.0.0.1:8000/v1/systemone --model my-model
jb.check(model) # one request; returns a summary, or raises with what to fix
What the model receives. A state (the input to decide about: text or
JSON, never changed by JevBench) and a batch of typed questions, keyed by
question id:
{
"model": "my-model",
"state": "Customer writes: Our nightly export has failed three nights in a row...",
"questions": {
"export_failing": {"type": "noul", "instructions": "Is a scheduled data export currently failing?"},
"primary_team": {"type": "choice", "instructions": "Which team should own the primary issue?",
"criteria": {"engineering": "Fixes broken product functionality",
"billing": "Handles invoices, charges and refunds"}},
"frustration": {"type": "score", "instructions": "How frustrated does the customer sound?",
"criteria": ["Calm", "Mildly annoyed", "Clearly frustrated", "Angry"]}
}
}
A Noul is a yes/no question and has no criteria. A Choice names its options, each with a description. A Score lists ordered levels, lowest first. Question ids, option ids and the number of questions per request vary from test to test: a model must answer whatever it is asked.
What the model returns. One answer per question, under the same id:
{
"answers": {
"export_failing": {"type": "noul", "noul": 0.93},
"primary_team": {"type": "choice", "probabilities": {"engineering": 0.88, "billing": 0.12}},
"frustration": {"type": "score", "probabilities": {"0": 0.05, "1": 0.25, "2": 0.6, "3": 0.1}}
}
}
| Type | Answer | Rules |
|---|---|---|
| Noul | {"type": "noul", "noul": p} |
p is the probability of yes, a number in [0, 1] |
| Choice | {"type": "choice", "probabilities": {...}} |
a probability for every option id of the question, under that id |
| Score | {"type": "score", "probabilities": {...}} |
a probability for every level, keyed by level index "0" ... "n-1" (or by every level text) |
Probabilities are finite and non-negative; those of a Choice or Score
should sum to 1 (JevBench renormalises them over the options). Other
fields of an answer ("choice", "score", "legend" and the like) are
ignored.
As a server. Answer POST /v1/systemone with the request above and
return {"answers": {...}} as JSON. An HTTP 4xx response counts as a
refused request; 429 and 5xx responses are retried. A bearer token, if
the server needs one, is read from JEVBENCH_API_KEY.
As a Python function. Return the answers object itself, not the whole
response:
import jevbench as jb
def my_model(state, questions):
answers = {}
for key, q in questions.items():
if q["type"] == "noul":
answers[key] = {"type": "noul", "noul": prob_yes(state, q)}
elif q["type"] == "choice":
answers[key] = {"type": "choice",
"probabilities": {o: prob_option(state, q, o) for o in q["criteria"]}}
else: # score: q["criteria"] is the list of levels, lowest first
answers[key] = {"type": "score",
"probabilities": {str(i): prob_level(state, q, i) for i in range(len(q["criteria"]))}}
return answers
model = jb.from_callable(my_model, name="my-model-v1")
jb.check(model)
report = jb.evaluate(model)
When something is wrong. jb.evaluate sends the first request before
the run and stops if the model cannot be used:
AnswerFormatError: an answer is not in the format above. The message names the question, shows what came back and states the format.ModelError: the model did not answer at all (the server is not running at the URL, it refused the request, it returned noanswersobject, or the function raised or takes other arguments).
Failures later in a run (a refused or invalid request) are recorded, shown under the report's table, and the relations they affect are left unscored rather than scored on fewer tests.
What JevBench does not measure
JevBench never uses gold answers: a relation holds or fails whatever the correct decision is. It measures whether a model's answers are consistent with one another (stable under rewording, additive over options, ordered by entailment, independent of the batch, conditioned correctly on the menu), not whether they are right. Coherence is necessary for a trustworthy decision model, not sufficient: a model that ignores its input and answers every question with a uniform distribution over its answers keeps every invariance and many additive laws by construction and scores high while deciding nothing. On jevbench-mini, a model that gives every answer equal probability scores 89.4, above every model we tested (56.5 to 83.9), at chance accuracy (39.8%, against 55.8% to 89.2% for those models). Read JevBench scores beside an accuracy or calibration evaluation of the same model, never instead of one; our leaderboard reports both.
Concepts: base data, engine, derived materials
base data (cases.jsonl) engine (jevbench) report
state + questions ---> relations transform the questions ---> coherence card,
a. computed at run time from code scores, radars
(permutations, merges, templates ...)
b. natural rewrites: generated once by
`jevbench rewrite`, judged, frozen as
a bank (derived material, versioned
next to the cases)
- The state never changes. It stands for the outside input, from a user or an upstream system, that a model is asked to decide about. Every relation edits only the question side: keys, instructions, criteria, answer types and the other questions of the batch. The engine refuses a trial that tries to send another state, and a test checks that every request carries the case's own state.
- Base data is what is tested: states and question batches.
- The engine turns each case into tests. Most transformations are computed from code at run time and are reproducible from the seed. Those that need new wording (natural paraphrases, negations, translations) are generated once, judged by a model, frozen as a rewrite bank and reused by every run, so all models see the same words.
Cases
A case is one line of JSONL with the exact request body you would send:
{"id": "invoice-001", "state": "Invoice INV-20931 ...",
"questions": {"po_missing": {"type": "noul", "instructions": "Is the purchase order number missing?"},
"next_step": {"type": "choice", "instructions": "What should accounts payable do next?",
"criteria": {"approve": "...", "hold": "...", "reject": "...", "escalate": "..."}}}}
Use your own production schemas: JevBench then tells you which of your questions are fragile before software acts on their answers.
Two optional fields describe facts of the case that JevBench cannot infer:
| Field | Used by | Example |
|---|---|---|
thresholds |
threshold_sweep |
[{"subject": "failed SSH login attempts", "value": 214}]; without it the count of digit groups in the state is used |
hierarchies |
partition_hierarchy |
{"next_step": [{"act_now": ["approve", "escalate"], "wait": ["hold", "reject"]}]}; without it options are split into two random blocks |
How a run works
flowchart LR
cases[("cases.jsonl<br/>state + questions")] --> base["Base request<br/>the source output"]
bank[("rewrite bank<br/>jevbench rewrite")] -.-> rel
base --> rel["Relations x targets x variants<br/>seeded trials"]
rel --> req["Trial requests<br/>state unchanged"]
req --> be{{"Backend<br/>/v1/systemone · Python function · fake"}}
be --> cache[("request cache")]
be --> judge["Evaluate<br/>deviation vs the fixed tolerance"]
judge -->|violated, optional| mini["Minimize<br/>rerun without the batch"]
judge --> card["Coherence card<br/>JSON · summarize · compare"]
mini --> card
- Source output. Each case's batch is asked once; its answers are the
source outputs the relations compare against. No model we have served
ourselves varied across three repeated requests (repeat noise 0), and scores judge every model
with one fixed tolerance, so one request suffices. With your own cases,
--repeats Nasks N times, takes the mean and reports the repeat noise (the largest total-variation distance of a repeat from the mean), a diagnostic for models that sample; bundled suites always ask once. - Frozen trials. The trials were generated once, with every choice
(which options to permute, merge, clone; which question to pair) drawn from a
generator seeded by
(seed, case, relation, question), then reduced to one test per relation and case and one variant per test, and frozen in the suite's plan. A run reads them; it draws no random numbers. Your own cases are generated the same way at run time, reproducibly. - Comparisons with the source output. Probes such as negations and conjunctions are appended to the case's full batch, and their answers are compared with the source output P_q, the answer to the case's own request, as the definition states. A model whose answers depend on the batch can therefore also lose points in MEA and LOG; how much its answers depend on the batch is measured on its own by BAT.
- Minimization (optional). With
minimize=True(--minimize), a violated trial is rerun with the batch reduced to its target questions;no-batch%in the coherence card is the share of violations that persist without the rest of the batch. It is a diagnostic, in no score, and off by default, so a run of a suite sends exactly the requests its manifest lists. - Report. Per relation and check: violations with a Wilson 95% interval, mean deviation, the share of trials whose argmax decision flips, and the worst examples with their full requests in the JSON.
Requests are cached on disk (.jevbench-cache/), keyed by backend, request
(order-preserving) and repeat index. --max-requests caps the uncached requests sent.
A run has three rounds (the base, the trials, the minimization reruns), and the
requests inside a round are independent, so they can be sent in parallel.
Most Jev servers answer one request at a time: in our measurements, 16 requests
in flight gave no gain on jevany-qwen-4b (0.97x) and slowed openthai-systemone
down (0.39x). The way to go faster is replicas of the same model, passed together,
--url http://host:8001/v1/systemone,http://host:8002/v1/systemone; they are used
in turn, share one cache, and by default get one request each (--concurrency
defaults to the number of replicas), which gives exactly the sequential answers.
More requests in flight help only servers that batch on the GPU (decider-4b on
vLLM: 1.35x), and there they change answers: 19 of 1181 verdicts flipped.
Scores
jevbench report runs/*.json turns saved reports (or raw jevbench run
results) into coherence scores (100 = no violation), compares models side by
side, and draws radar charts (--radar, --grid, --family-dir); --card
prints each check's violation rate and worst cases. In Python the same is
report.scores, jb.compare and jb.radar:
- Test: one relation on one question of one case. A relation states one law; when the law is stated for every template, option, pair or batch size, those variants are judged together and the test fails if any variant does.
- Relation:
score= 1 − violation rate over its tests;graded= 1 − mean excess beyond the tolerance, scaled so a deviation at the maximum counts fully.scoresays how often a model breaks a law;gradedsays how far it breaks it on average. Read them together. Each relation measures one quantity: a total variation distance between two distributions,tv= ½ Σₒ |P(o) − P′(o)| (0: identical, 1: disjoint), the absolute gap in an identity (*_gap,abs_diff), or the excess over a bound (excess). - Dimension = mean of its relations; groups report the mean of theirs but add no weighting level, so regrouping relations changes no score. Overall = mean of the 5 dimensions; it is not reported when a dimension has no scored relation.
- Same tolerance for every model (default 0.05): every check is judged against it, so a noisy model gains no slack. Checks with a tolerance of their own keep it. When the base was asked more than once, the mean repeat noise is reported beside the scores.
- Left out of the means, but listed: relations with fewer than
--min-ntests (default 5), and diagnostics (response-field checks, which run only when named). A group with no scored relation is a gap, not a zero. - Intervals: 95% bootstrap over cases (
--bootstrap, default 1,000 draws, fixed seed). With few cases they are wide in truth and narrow only by luck; score claims need many cases.
Charts: --radar overlays all models on one radar; --grid draws one small
radar per model; --families bench/families.json --family-dir DIR draws one
overlaid radar per architecture family. --level dimension uses the five
dimension axes instead of the 11 groups; --metric graded plots graded scores.
Relation taxonomy
Every relation is classified on two axes, defined once in
jevbench.taxonomy:
- The operation its law uses (dimension, then group). Every relation is one transformation with one law, of form (E) two distributions are equal, (I) an identity holds, or (L) an inequality holds. The dimension is fixed by the operation that links the two distributions: Representation (REP), a bijection of answers, for two descriptions of the same events; Batch independence (BAT), the identity, when only the other questions change; Probability measure (MEA), a many-to-one map of outcomes or the additive identity it induces, for different events of one measure; Logical coherence (LOG), the order induced by entailment and Boolean combination; and Choice-set coherence (CHO), restriction to a sub-menu, because a Choice is the model's distribution given that the answer is one of its options.
- What it edits: the question
key, itsinstructions, itscriteria(options or levels), its answertype, or the rest of thebatch. Never thestate: it is the outside input and is fixed for every test.
Checks on auxiliary response fields (confidence_consistency,
routing_stability) test the output format rather than the distribution. They
are diagnostics: outside the dimensions, never scored, and run only with
--relations diagnostics (or by name).
Membership follows the operation, not the data. The mathematical
definitions, the full list of laws and the reasons further candidate laws are
not relations are in the technical report (paper/).
The diagram, the coverage matrix and the tables below are generated by
jevbench taxonomy --write README.md; a test fails if they drift from the code.
jevbench relations -v prints the same tree with each relation's
expectation and source.
50 relations in 5 dimensions and 11 groups, plus 2 unscored diagnostics. Each relation is one transformation with one law; each dimension is defined by the operation that links the two distributions its laws compare.
| Dimension | Title | Compares | Operation | Relations |
|---|---|---|---|---|
| REP | Representation consistency | two questions describe the same events | bijection of answers: P′ = σ#P |
15 |
| BAT | Batch independence | the question is fixed and only its companions change | `identity: P_q | B′ = P_q |
| MEA | Probability measure coherence | the questions ask about different events of one measure | many-to-one map of outcomes, or the additive identity it induces |
17 |
| LOG | Logical coherence | the questions are propositions ordered by entailment | order: A ⊨ B ⇒ P(A) ≤ P(B), and the bounds of Boolean combinations |
10 |
| CHO | Choice-set coherence | the menu a Choice is conditioned on changes | `restriction to a sub-menu: P′ | K = P |
flowchart LR
root(["JevBench relations"])
REP["<b>REP · Representation consistency</b><br/><i>bijection of answers: P′ = σ#P</i>"]
root --> REP
REP_invariance["<b>REP.invariance · Invariance</b><br/>● instruction_paraphrase<br/>● cross_lingual<br/>● typo_noise<br/>● verbosity<br/>● key_rename<br/>● description_paraphrase<br/>● de_morgan"]
REP --> REP_invariance
REP_equivariance["<b>REP.equivariance · Equivariance</b><br/>● option_permutation<br/>● option_rename<br/>● clone_symmetry"]
REP --> REP_equivariance
REP_binding["<b>REP.binding · Binding</b><br/>● key_instruction_conflict<br/>● name_description_conflict<br/>● description_swap"]
REP --> REP_binding
REP_type["<b>REP.type · Answer type</b><br/>● noul_as_choice<br/>● noul_as_score"]
REP --> REP_type
BAT["<b>BAT · Batch independence</b><br/><i>identity: P_q|B′ = P_q|B</i>"]
root --> BAT
BAT_isolation["<b>BAT.isolation · Isolation</b><br/>● batch_solo<br/>● batch_order<br/>● duplicate_question<br/>● batch_content<br/>● batch_size_sweep"]
BAT --> BAT_isolation
MEA["<b>MEA · Probability measure coherence</b><br/><i>many-to-one map of outcomes, or the additive identity it induces</i>"]
root --> MEA
MEA_complement["<b>MEA.complement · Complement</b><br/>● negation_wrapper<br/>● natural_negation<br/>● choice_complement<br/>● score_complement<br/>● compound_negation"]
MEA --> MEA_complement
MEA_partition["<b>MEA.partition · Partition</b><br/>● option_merge<br/>● option_split<br/>● option_clone<br/>● similarity_effect<br/>● partition_hierarchy<br/>● score_granularity<br/>● extreme_level<br/>● option_closure"]
MEA --> MEA_partition
MEA_marginalisation["<b>MEA.marginalisation · Marginalisation</b><br/>● score_cumulative<br/>● choice_indicator<br/>● pair_choice_marginals<br/>● inclusion_exclusion"]
MEA --> MEA_marginalisation
LOG["<b>LOG · Logical coherence</b><br/><i>order: A ⊨ B ⇒ P(A) ≤ P(B), and the bounds of Boolean combinations</i>"]
root --> LOG
LOG_bounds["<b>LOG.bounds · Bounds</b><br/>● frechet_and_upper<br/>● nary_and_upper<br/>● nary_and_lower<br/>● nary_or_upper<br/>● nary_or_lower<br/>● count_upper<br/>● count_lower"]
LOG --> LOG_bounds
LOG_monotonicity["<b>LOG.monotonicity · Monotonicity</b><br/>● threshold_sweep<br/>● entailment_strength<br/>● quantifier_monotonicity"]
LOG --> LOG_monotonicity
CHO["<b>CHO · Choice-set coherence</b><br/><i>restriction to a sub-menu: P′|K = P|K</i>"]
root --> CHO
CHO_conditioning["<b>CHO.conditioning · Conditioning</b><br/>● irrelevant_option<br/>● remove_option_iia<br/>● pairwise_iia"]
CHO --> CHO_conditioning
classDef planned stroke-dasharray: 5 4,color:#777
Coverage by edited component. Each mark is one relation that edits that part of the request. Empty cells (·) are combinations not yet tested. The state is not a column: no relation edits it.
| Group | key | instructions | criteria | type | batch |
|---|---|---|---|---|---|
| REP.invariance Invariance | ● | ●●●●● | ●● | · | ● |
| REP.equivariance Equivariance | · | · | ●●● | · | · |
| REP.binding Binding | ● | · | ●● | · | · |
| REP.type Answer type | · | · | ●● | ●● | · |
| BAT.isolation Isolation | · | · | · | · | ●●●●● |
| MEA.complement Complement | · | ●●●●● | · | ●● | ●●●●● |
| MEA.partition Partition | · | · | ●●●●●●●● | · | · |
| MEA.marginalisation Marginalisation | · | ●●●● | · | ●●● | ●●●● |
| LOG.bounds Bounds | · | ●●●●●●● | · | · | ●●●●●●● |
| LOG.monotonicity Monotonicity | · | ●●● | · | · | ●●● |
| CHO.conditioning Conditioning | · | · | ●●● | · | · |
Forms of law: E distributional equality; I identity; L inequality; field field agrees with probabilities.
REP · Representation consistency: bijection of answers: P′ = σ#P
Two questions describe the same events.
| Group | Relation | Edits | Form | Law | Check | Variants |
|---|---|---|---|---|---|---|
| REP.invariance Invariance | ● instruction_paraphrase |
instructions | E | judged paraphrase of the instructions | tv |
1 rewrite (bank) |
| REP.invariance Invariance | ● cross_lingual |
instructions, criteria | E | question translated, state untouched | tv |
1 rewrite (bank) |
| REP.invariance Invariance | ● typo_noise |
instructions | E | typos, casing, punctuation | tv |
3 templates |
| REP.invariance Invariance | ● verbosity |
instructions | E | redundant clarification added | tv |
3 templates |
| REP.invariance Invariance | ● key_rename |
key | E | neutral question key | tv |
random |
| REP.invariance Invariance | ● description_paraphrase |
criteria | E | judged paraphrase of one option | tv |
1 rewrite (bank) |
| REP.invariance Invariance | ● de_morgan |
instructions, batch | E | not (A and B) equals not A or not B | nand_vs_or_not |
random |
| REP.equivariance Equivariance | ● option_permutation |
criteria | E | options reordered, Score levels reversed | tv |
random |
| REP.equivariance Equivariance | ● option_rename |
criteria | E | neutral option ids, same descriptions | tv |
fixed |
| REP.equivariance Equivariance | ● clone_symmetry |
criteria | E | two identical options get equal probability | clone_symmetry |
random |
| REP.binding Binding | ● key_instruction_conflict |
key | E | negated or swapped question keys | tv |
2 templates |
| REP.binding Binding | ● name_description_conflict |
criteria | E | option ids exchanged, descriptions fixed | tv |
random |
| REP.binding Binding | ● description_swap |
criteria | E | descriptions exchanged, ids fixed | tv |
random |
| REP.type Answer type | ● noul_as_choice |
type, criteria | E | Noul versus yes/no Choice | abs_diff |
random |
| REP.type Answer type | ● noul_as_score |
type, criteria | E | Noul versus two-level Score | abs_diff |
2 templates |
BAT · Batch independence: identity: P_q|B′ = P_q|B
The question is fixed and only its companions change.
| Group | Relation | Edits | Form | Law | Check | Variants |
|---|---|---|---|---|---|---|
| BAT.isolation Isolation | ● batch_solo |
batch | E | asked alone versus in its batch | tv |
fixed |
| BAT.isolation Isolation | ● batch_order |
batch | E | question order reversed | tv |
fixed |
| BAT.isolation Isolation | ● duplicate_question |
batch | E | same question twice in one request | within_request_tv |
fixed |
| BAT.isolation Isolation | ● batch_content |
batch | E | other questions replaced, size fixed | tv |
fixed |
| BAT.isolation Isolation | ● batch_size_sweep |
batch | E | batches of 2, 5 and 10 questions | tv |
every size |
MEA · Probability measure coherence: many-to-one map of outcomes, or the additive identity it induces
The questions ask about different events of one measure.
| Group | Relation | Edits | Form | Law | Check | Variants |
|---|---|---|---|---|---|---|
| MEA.complement Complement | ● negation_wrapper |
instructions, batch | I | P(Q) + P(not Q) = 1, templates | complement_gap |
3 templates |
| MEA.complement Complement | ● natural_negation |
instructions, batch | I | P(Q) + P(Q') = 1, natural opposite | complement_gap |
1 rewrite (bank) |
| MEA.complement Complement | ● choice_complement |
type, instructions, batch | I | Noul 'not o' = 1 - P(o) | complement_gap |
3 templates |
| MEA.complement Complement | ● score_complement |
type, instructions, batch | I | Noul 'not level k' = 1 - P(k) | complement_gap |
2 templates |
| MEA.complement Complement | ● compound_negation |
instructions, batch | I | P(not (A and B)) + P(A and B) = 1 | complement_gap |
random |
| MEA.partition Partition | ● option_merge |
criteria | E | 'a or b' gets P(a) + P(b) | tv_to_coarsened |
random |
| MEA.partition Partition | ● option_split |
criteria | E | 'o and X' + 'o and not X' = P(o) | tv_after_collapse |
random |
| MEA.partition Partition | ● option_clone |
criteria | E | a clone and its original keep P(o) | tv_after_collapse |
random |
| MEA.partition Partition | ● similarity_effect |
criteria | E | a near-duplicate and its original keep P(b) | tv_after_collapse |
random |
| MEA.partition Partition | ● partition_hierarchy |
criteria | E | coarse blocks = sums of their options | tv_to_coarsened |
random |
| MEA.partition Partition | ● score_granularity |
criteria | E | adjacent levels merged keep their sum | tv_to_coarsened |
fixed |
| MEA.partition Partition | ● extreme_level |
criteria | E | a level beyond the top refines the top level | tv_after_collapse |
fixed |
| MEA.partition Partition | ● option_closure |
criteria | E | each option in turn replaced by a catch-all | catch_all_tv |
every option |
| MEA.marginalisation Marginalisation | ● score_cumulative |
type, instructions, batch | I | 'level k or higher' = upper tail, every k | tail_gap_max* |
2 templates |
| MEA.marginalisation Marginalisation | ● choice_indicator |
type, instructions, batch | I | Noul 'is it o?' = P(o), every o | indicator_gap |
2 templates |
| MEA.marginalisation Marginalisation | ● pair_choice_marginals |
type, instructions, batch | I | four-way pair Choice has the Nouls as marginals | marginal_gap |
random |
| MEA.marginalisation Marginalisation | ● inclusion_exclusion |
instructions, batch | I | P(A and B) + P(A or B) = P(A) + P(B) | inclusion_exclusion |
3 templates |
LOG · Logical coherence: order: A ⊨ B ⇒ P(A) ≤ P(B), and the bounds of Boolean combinations
The questions are propositions ordered by entailment.
| Group | Relation | Edits | Form | Law | Check | Variants |
|---|---|---|---|---|---|---|
| LOG.bounds Bounds | ● frechet_and_upper |
instructions, batch | L | P(A and B) <= min(P(A), P(B)) | excess |
3 templates |
| LOG.bounds Bounds | ● nary_and_upper |
instructions, batch | L | P(all) <= min P(Ai) | excess |
random |
| LOG.bounds Bounds | ● nary_and_lower |
instructions, batch | L | P(all) >= sum P(Ai) - (n - 1) | excess |
random |
| LOG.bounds Bounds | ● nary_or_upper |
instructions, batch | L | P(any) <= min(1, sum P(Ai)) | excess |
random |
| LOG.bounds Bounds | ● nary_or_lower |
instructions, batch | L | P(any) >= max P(Ai) | excess |
random |
| LOG.bounds Bounds | ● count_upper |
instructions, batch | L | P(at least two) <= sum P(Ai) / 2 | excess |
random |
| LOG.bounds Bounds | ● count_lower |
instructions, batch | L | P(at least two) >= (sum P(Ai) - 1) / (n - 1) | excess |
random |
| LOG.monotonicity Monotonicity | ● threshold_sweep |
instructions, batch | L | 'more than k' falls with k | monotonicity |
declared or counted |
| LOG.monotonicity Monotonicity | ● entailment_strength |
instructions, batch | L | Q1 entails Q2 => P(Q1) <= P(Q2) | chain |
fixed |
| LOG.monotonicity Monotonicity | ● quantifier_monotonicity |
instructions, batch | L | all <= at least two <= at least one | chain |
random |
CHO · Choice-set coherence: restriction to a sub-menu: P′|K = P|K
The menu a Choice is conditioned on changes.
| Group | Relation | Edits | Form | Law | Check | Variants |
|---|---|---|---|---|---|---|
| CHO.conditioning Conditioning | ● irrelevant_option |
criteria | E | off-topic option added | iia_tv |
random |
| CHO.conditioning Conditioning | ● remove_option_iia |
criteria | E | each option in turn removed | iia_tv |
every option |
| CHO.conditioning Conditioning | ● pairwise_iia |
criteria | E | each pair of options asked alone | iia_tv |
every pair |
Diagnostics (not scored)
Checks on auxiliary response fields. They test the output format, not the distribution, so they belong to no dimension, never enter a score and run only when named (--relations diagnostics).
| Group | Relation | Edits | Form | Law | Check | Variants |
|---|---|---|---|---|---|---|
| DIAG.fields Response fields | ● confidence_consistency |
- | field | confidence is a function of the probabilities | fits_definition |
fixed |
| DIAG.fields Response fields | ● routing_stability |
- | field | 0.8 routing survives key, order, batch edits | route_flip |
fixed |
Form: (E) distributional equality, (I) identity, (L) inequality. Check: the one quantity the relation measures. Variants: a law stated for every template, option, pair or size holds for a question only if it holds for every variant; its measure is the largest.
* score_cumulative/tail_gap_max: largest gap over the levels k; strict, so read it with the graded score.
Tensions between expectations
Some expectations pull against each other for a given model class. A Luce-type
scorer (independent option scores, softmax) satisfies conditioning on a
changed menu (irrelevant_option, remove_option_iia, pairwise_iia)
exactly, and fails every refinement of an option by construction:
option_clone, option_split, similarity_effect, extreme_level and
option_closure, the red-bus/blue-bus problem. The bundled fake:coherent
backend is such a scorer, and the test suite checks exactly this.
Templates and rewrites
Probe relations run every template as a separate variant. A test passes only if every template does; the report still breaks violation rates down by template, so a failure can be attributed to a model rather than to one wording.
Natural rewrites are generated once and frozen:
jevbench rewrite --cases my-cases.jsonl --kinds paraphrase,negation,description,translate:zh \
--chat-url https://<host>/v1/chat/completions --model <chat model> --out rewrites/my-bank.jsonl
jevbench eval --cases my-cases.jsonl --url ... --rewrites rewrites/my-bank.jsonl --out runs/x.json
The chat model proposes candidates and then judges each one (both directions
for paraphrases and negations; faithfulness plus unchanged structure for
translations). Every candidate is kept in the bank with its verdict; runs use
only accepted ones. Set "audited": true or false after human review;
--audited-only restricts a run to reviewed rewrites. The banks of the bundled
suites (src/jevbench/suites/*/rewrites.jsonl) were generated and judged by
gpt-oss-20b and are not yet human-audited.
Numeric thresholds default to counting digit groups in the state. Declare
real quantities in a case to probe them instead:
"thresholds": [{"subject": "failed SSH login attempts", "value": 214}].
Writing a relation: subclass jevbench.relations.Relation, decorate it with
@register, and implement applicable, build (return a Trial of requests)
and evaluate (return one Outcome per tested question, with a deviation and
a tolerance). Set templates to run several wordings, or override variants.
A relation that states another law about the same questions subclasses the
first and sets rng_name to its name, so both read the same requests.
Scope and limits
- A violation is a property of a model and a wording. Report per-template rates, and prefer claims that hold across templates and natural rewrites.
- The rewrite judge is an LLM. Its verdicts are recorded, not trusted: audit a sample before publishing, and report results on audited rewrites.
- Coherence is necessary, not sufficient: a model that ignores its input can be
perfectly invariant. The sensitivity relations (
description_swap,name_description_conflict,key_instruction_conflict) andthreshold_sweepcheck, from the question side, that answers move when they should and only then. - Hosted services have terms of use. Check them before publishing results.
Authors
JevBench is developed by Chen Feng at the ML Lab, Queen's University Belfast (c.feng@qub.ac.uk). Issues and contributions: https://github.com/JevBench/jevbench.
License
Apache-2.0. See the LICENSE and NOTICE files shipped with the package.
Metadata
Release files for jevbench 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| jevbench-0.1.0.tar.gz | 1.4 MB | Without provenance | Details for jevbench-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| jevbench-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for jevbench-0.1.0-py3-none-any.whl |
Total release size: 2.8 MB
Release files / jevbench-0.1.0.tar.gz
| Download URL | jevbench-0.1.0.tar.gz |
|---|---|
| Size | 1.4 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
45460eae2c3e94a8ed980f04eba9cd2eedc13a3942b3ff8537b6665924e4d655
|
|
BLAKE2b-256 checksum How to use checksums |
d55a48842bc918d5a5ed511c1f9c3d10c588900ec48c66c1f48db848c2f8a55d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.9
|
Release files / jevbench-0.1.0-py3-none-any.whl
| Download URL | jevbench-0.1.0-py3-none-any.whl |
|---|---|
| Size | 1.4 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4bf93419a987d09238b1427b70ac46ba16fb5f36206e67a27f4b370d459568e5
|
|
BLAKE2b-256 checksum How to use checksums |
81127664f89675907f3ca6c586cbf4a2777905732d941d49f1f358eb27387db5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.9
|
Security for jevbench 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.