Skip to main content

JevBench

Coherence tests for Jev-compatible typed decision models.

Website and leaderboard · PyPI · Reproducing the results · How the data were made

JevBench measures whether a model's answers are consistent with one another, not whether they are right and not how fast they come: it is a test of stability and coherence under meaning-preserving and logically related edits, with no gold labels.

Typed decision models ("System One" models) answer questions about a fixed state with probabilities over declared answers: a Noul (yes/no), a Choice over named options, or a Score over ordered levels. Many such models now exist, trained in very different ways. What they share is the interface:

f(state, question key, instructions, criteria) -> probability distribution over criteria

JevBench tests that interface. It never needs gold labels. It never edits the state. It transforms only the question key, the instructions and the criteria, and checks whether the answers keep the relations that probability and choice theory say they must keep: the same answer after a meaning-preserving edit, complementary probabilities for a question and its negation, Frechet bounds for conjunctions, additivity when options are merged or cloned, independence of irrelevant alternatives, isolation between questions in one batch.

JevBench is not affiliated with or endorsed by TypeSafe AI. "Jev" names the wire interface (POST /v1/systemone) that JevBench speaks.

Quick start

pip install jevbench                 # Python 3.10+, depends only on httpx
jevbench suites                      # the bundled suites: jevbench-mini (default), jevbench-240, examples
jevbench eval --backend fake:coherent --suite examples     # seconds: check that everything works

Against any server that speaks the System One wire format:

import jevbench as jb

model = jb.systemone("http://127.0.0.1:8000/v1/systemone", model="open-jev")
jb.evaluate(model, suite="examples")                     # seconds: check that your model works with JevBench
report = jb.evaluate(model)                              # jevbench-mini, the suite results are reported on
full = jb.evaluate(model, suite="jevbench-240")          # optional: every test, narrower intervals
print(report.table("dimension"))                         # also "group", "relation", "domain"
report.save("runs/open-jev.json")

jb.evaluate(model, "jevbench-240", domains=["finance_ops", "logistics"]) # some domains (cases)
jb.evaluate(model, dimensions=["MEA"], groups=["CHO.conditioning"])      # some dimensions and groups
jb.evaluate(model, relations=["option_clone", "de_morgan"])              # some relations
jb.estimate("jevbench-240", domains=["finance_ops"])     # requests a run would send
print(jb.compare([report, other_report]))

The same from the shell:

export JEVBENCH_API_KEY=...          # if the server needs a bearer token
jevbench eval --url http://127.0.0.1:8000/v1/systemone --model open-jev \
  --dimensions MEA --concurrency 16 --out runs/open-jev.json
jevbench report runs/open-jev.json runs/other.json      # compare saved reports
  • Models. jb.systemone(url, model) covers every Jev-compatible server; pass several comma-separated URLs to use replicas of one model in turn. A model in your Python process is tested with jb.from_callable(fn, name="my-model-v1"), where fn(state, questions) -> answers; pass a list of functions, one per GPU, to use them as replicas. The name keys the request cache, so give each model and version its own (without one, answers are cached for the run only). See examples/in_process_model.py.

  • Selection. domains selects cases; dimensions, groups and relations select relations (their union). The overall score is reported only when all five dimensions were tested. Select domains on jevbench-240: jevbench-mini tests each relation on 2 cases per domain, fewer than the 5 tests a relation needs to be scored (min_n).

  • Your own data. jb.evaluate(model, cases="my-cases.jsonl"); generate cases for a new domain with pip install jevbench[datagen] and jevbench datagen (see below).

  • Speed. Requests are cached (.jevbench-cache/). Most Jev servers answer one request at a time, so a run is sped up by replicas: start the model several times (one per GPU) and pass every URL; JevBench keeps one request in flight per replica, which gives exactly the sequential answers.

  • One fixed test set. A bundled suite ships one frozen test plan, generated once without any model, with every trial's exact requests. It tests each relation exactly once per case: every relation of jevbench-240 has 240 tests, spread over all cases and, within a domain, as evenly as possible over its questions, with all of each test's variants. A run sends these requests and judges the answers; it draws no random numbers, and settings that would run other tests (another seed, rewrite bank or number of trials) are refused, so everyone runs the same tests. The plan is built against a sealed context: a relation that read model outputs while building a trial could not be frozen. Your own cases= are built from the seed, which is deterministic too, and jevbench plan <suite dir> freezes a suite of your own.

  • Which suite to report. Results are reported on jevbench-mini, the default: the 240 cases of jevbench-240, each relation tested on 2 cases per domain (24 tests per relation), a ninth of the requests. Every one of its tests is a jevbench-240 test. On the 17 models we ran on both, the overall scores agree (Pearson 0.993, Spearman 0.978, mini minus 240 at most 2.2 points, ranks moved by at most 2), and every pair of models that jevbench-240 separates (95% intervals apart) is ordered the same way by jevbench-mini. Its intervals are about 2.5 times as wide (4.4 against 1.7 points), so read differences of a few points between models as ties, or run suite="jevbench-240" (optional) to separate them. Even on jevbench-240 a model's domain scores differ from its overall score by about as much as sampling noise, so we do not compare models by domain.

  • The demo suite. examples is one hand-written customer-support case: a run takes seconds and shows that the tool works with your model. It scores each relation from its one test (min_n 1), so its numbers are a demonstration, never a result.

jevbench run is the low-level engine run over a JSONL of cases, kept for scripts: jevbench run --cases my-cases.jsonl --backend systemone:<model> --url ... --out x.json.

Results

The leaderboard of 21 openly released models, with every score, interval and accuracy, and the full reports behind them, is on the JevBench website. docs/reproducing.md explains how to recompute it from the published reports or run any of the models again.

Connecting your model

JevBench reads the standard Jev format and nothing else: a model that returns another format is asked to adapt, with a message that says how. Check a model before a full run:

jevbench check --url http://127.0.0.1:8000/v1/systemone --model my-model
jb.check(model)        # one request; returns a summary, or raises with what to fix

What the model receives. A state (the input to decide about: text or JSON, never changed by JevBench) and a batch of typed questions, keyed by question id:

{
  "model": "my-model",
  "state": "Customer writes: Our nightly export has failed three nights in a row...",
  "questions": {
    "export_failing": {"type": "noul", "instructions": "Is a scheduled data export currently failing?"},
    "primary_team":   {"type": "choice", "instructions": "Which team should own the primary issue?",
                       "criteria": {"engineering": "Fixes broken product functionality",
                                    "billing": "Handles invoices, charges and refunds"}},
    "frustration":    {"type": "score", "instructions": "How frustrated does the customer sound?",
                       "criteria": ["Calm", "Mildly annoyed", "Clearly frustrated", "Angry"]}
  }
}

A Noul is a yes/no question and has no criteria. A Choice names its options, each with a description. A Score lists ordered levels, lowest first. Question ids, option ids and the number of questions per request vary from test to test: a model must answer whatever it is asked.

What the model returns. One answer per question, under the same id:

{
  "answers": {
    "export_failing": {"type": "noul", "noul": 0.93},
    "primary_team":   {"type": "choice", "probabilities": {"engineering": 0.88, "billing": 0.12}},
    "frustration":    {"type": "score", "probabilities": {"0": 0.05, "1": 0.25, "2": 0.6, "3": 0.1}}
  }
}
Type Answer Rules
Noul {"type": "noul", "noul": p} p is the probability of yes, a number in [0, 1]
Choice {"type": "choice", "probabilities": {...}} a probability for every option id of the question, under that id
Score {"type": "score", "probabilities": {...}} a probability for every level, keyed by level index "0" ... "n-1" (or by every level text)

Probabilities are finite and non-negative; those of a Choice or Score should sum to 1 (JevBench renormalises them over the options). Other fields of an answer ("choice", "score", "legend" and the like) are ignored.

As a server. Answer POST /v1/systemone with the request above and return {"answers": {...}} as JSON. An HTTP 4xx response counts as a refused request; 429 and 5xx responses are retried. A bearer token, if the server needs one, is read from JEVBENCH_API_KEY.

As a Python function. Return the answers object itself, not the whole response:

import jevbench as jb

def my_model(state, questions):
    answers = {}
    for key, q in questions.items():
        if q["type"] == "noul":
            answers[key] = {"type": "noul", "noul": prob_yes(state, q)}
        elif q["type"] == "choice":
            answers[key] = {"type": "choice",
                            "probabilities": {o: prob_option(state, q, o) for o in q["criteria"]}}
        else:   # score: q["criteria"] is the list of levels, lowest first
            answers[key] = {"type": "score",
                            "probabilities": {str(i): prob_level(state, q, i) for i in range(len(q["criteria"]))}}
    return answers

model = jb.from_callable(my_model, name="my-model-v1")
jb.check(model)
report = jb.evaluate(model)

When something is wrong. jb.evaluate sends the first request before the run and stops if the model cannot be used:

  • AnswerFormatError: an answer is not in the format above. The message names the question, shows what came back and states the format.
  • ModelError: the model did not answer at all (the server is not running at the URL, it refused the request, it returned no answers object, or the function raised or takes other arguments).

Failures later in a run (a refused or invalid request) are recorded, shown under the report's table, and the relations they affect are left unscored rather than scored on fewer tests.

What JevBench does not measure

JevBench never uses gold answers: a relation holds or fails whatever the correct decision is. It measures whether a model's answers are consistent with one another (stable under rewording, additive over options, ordered by entailment, independent of the batch, conditioned correctly on the menu), not whether they are right. Coherence is necessary for a trustworthy decision model, not sufficient: a model that ignores its input and answers every question with a uniform distribution over its answers keeps every invariance and many additive laws by construction and scores high while deciding nothing. On jevbench-mini, a model that gives every answer equal probability scores 89.4, above every model we tested (56.5 to 83.9), at chance accuracy (39.8%, against 55.8% to 89.2% for those models). Read JevBench scores beside an accuracy or calibration evaluation of the same model, never instead of one; our leaderboard reports both.

Concepts: base data, engine, derived materials

base data (cases.jsonl)        engine (jevbench)                          report
state + questions      --->    relations transform the questions   --->  coherence card,
                               a. computed at run time from code          scores, radars
                                  (permutations, merges, templates ...)
                               b. natural rewrites: generated once by
                                  `jevbench rewrite`, judged, frozen as
                                  a bank (derived material, versioned
                                  next to the cases)
  • The state never changes. It stands for the outside input, from a user or an upstream system, that a model is asked to decide about. Every relation edits only the question side: keys, instructions, criteria, answer types and the other questions of the batch. The engine refuses a trial that tries to send another state, and a test checks that every request carries the case's own state.
  • Base data is what is tested: states and question batches.
  • The engine turns each case into tests. Most transformations are computed from code at run time and are reproducible from the seed. Those that need new wording (natural paraphrases, negations, translations) are generated once, judged by a model, frozen as a rewrite bank and reused by every run, so all models see the same words.

Cases

A case is one line of JSONL with the exact request body you would send:

{"id": "invoice-001", "state": "Invoice INV-20931 ...",
 "questions": {"po_missing": {"type": "noul", "instructions": "Is the purchase order number missing?"},
               "next_step": {"type": "choice", "instructions": "What should accounts payable do next?",
                             "criteria": {"approve": "...", "hold": "...", "reject": "...", "escalate": "..."}}}}

Use your own production schemas: JevBench then tells you which of your questions are fragile before software acts on their answers.

Two optional fields describe facts of the case that JevBench cannot infer:

Field Used by Example
thresholds threshold_sweep [{"subject": "failed SSH login attempts", "value": 214}]; without it the count of digit groups in the state is used
hierarchies partition_hierarchy {"next_step": [{"act_now": ["approve", "escalate"], "wait": ["hold", "reject"]}]}; without it options are split into two random blocks

How a run works

flowchart LR
    cases[("cases.jsonl<br/>state + questions")] --> base["Base request<br/>the source output"]
    bank[("rewrite bank<br/>jevbench rewrite")] -.-> rel
    base --> rel["Relations x targets x variants<br/>seeded trials"]
    rel --> req["Trial requests<br/>state unchanged"]
    req --> be{{"Backend<br/>/v1/systemone · Python function · fake"}}
    be --> cache[("request cache")]
    be --> judge["Evaluate<br/>deviation vs the fixed tolerance"]
    judge -->|violated, optional| mini["Minimize<br/>rerun without the batch"]
    judge --> card["Coherence card<br/>JSON · summarize · compare"]
    mini --> card
  1. Source output. Each case's batch is asked once; its answers are the source outputs the relations compare against. No model we have served ourselves varied across three repeated requests (repeat noise 0), and scores judge every model with one fixed tolerance, so one request suffices. With your own cases, --repeats N asks N times, takes the mean and reports the repeat noise (the largest total-variation distance of a repeat from the mean), a diagnostic for models that sample; bundled suites always ask once.
  2. Frozen trials. The trials were generated once, with every choice (which options to permute, merge, clone; which question to pair) drawn from a generator seeded by (seed, case, relation, question), then reduced to one test per relation and case and one variant per test, and frozen in the suite's plan. A run reads them; it draws no random numbers. Your own cases are generated the same way at run time, reproducibly.
  3. Comparisons with the source output. Probes such as negations and conjunctions are appended to the case's full batch, and their answers are compared with the source output P_q, the answer to the case's own request, as the definition states. A model whose answers depend on the batch can therefore also lose points in MEA and LOG; how much its answers depend on the batch is measured on its own by BAT.
  4. Minimization (optional). With minimize=True (--minimize), a violated trial is rerun with the batch reduced to its target questions; no-batch% in the coherence card is the share of violations that persist without the rest of the batch. It is a diagnostic, in no score, and off by default, so a run of a suite sends exactly the requests its manifest lists.
  5. Report. Per relation and check: violations with a Wilson 95% interval, mean deviation, the share of trials whose argmax decision flips, and the worst examples with their full requests in the JSON.

Requests are cached on disk (.jevbench-cache/), keyed by backend, request (order-preserving) and repeat index. --max-requests caps the uncached requests sent.

A run has three rounds (the base, the trials, the minimization reruns), and the requests inside a round are independent, so they can be sent in parallel. Most Jev servers answer one request at a time: in our measurements, 16 requests in flight gave no gain on jevany-qwen-4b (0.97x) and slowed openthai-systemone down (0.39x). The way to go faster is replicas of the same model, passed together, --url http://host:8001/v1/systemone,http://host:8002/v1/systemone; they are used in turn, share one cache, and by default get one request each (--concurrency defaults to the number of replicas), which gives exactly the sequential answers. More requests in flight help only servers that batch on the GPU (decider-4b on vLLM: 1.35x), and there they change answers: 19 of 1181 verdicts flipped.

Scores

jevbench report runs/*.json turns saved reports (or raw jevbench run results) into coherence scores (100 = no violation), compares models side by side, and draws radar charts (--radar, --grid, --family-dir); --card prints each check's violation rate and worst cases. In Python the same is report.scores, jb.compare and jb.radar:

  • Test: one relation on one question of one case. A relation states one law; when the law is stated for every template, option, pair or batch size, those variants are judged together and the test fails if any variant does.
  • Relation: score = 1 − violation rate over its tests; graded = 1 − mean excess beyond the tolerance, scaled so a deviation at the maximum counts fully. score says how often a model breaks a law; graded says how far it breaks it on average. Read them together. Each relation measures one quantity: a total variation distance between two distributions, tv = ½ Σₒ |P(o) − P′(o)| (0: identical, 1: disjoint), the absolute gap in an identity (*_gap, abs_diff), or the excess over a bound (excess).
  • Dimension = mean of its relations; groups report the mean of theirs but add no weighting level, so regrouping relations changes no score. Overall = mean of the 5 dimensions; it is not reported when a dimension has no scored relation.
  • Same tolerance for every model (default 0.05): every check is judged against it, so a noisy model gains no slack. Checks with a tolerance of their own keep it. When the base was asked more than once, the mean repeat noise is reported beside the scores.
  • Left out of the means, but listed: relations with fewer than --min-n tests (default 5), and diagnostics (response-field checks, which run only when named). A group with no scored relation is a gap, not a zero.
  • Intervals: 95% bootstrap over cases (--bootstrap, default 1,000 draws, fixed seed). With few cases they are wide in truth and narrow only by luck; score claims need many cases.

Charts: --radar overlays all models on one radar; --grid draws one small radar per model; --families bench/families.json --family-dir DIR draws one overlaid radar per architecture family. --level dimension uses the five dimension axes instead of the 11 groups; --metric graded plots graded scores.

Relation taxonomy

Every relation is classified on two axes, defined once in jevbench.taxonomy:

  • The operation its law uses (dimension, then group). Every relation is one transformation with one law, of form (E) two distributions are equal, (I) an identity holds, or (L) an inequality holds. The dimension is fixed by the operation that links the two distributions: Representation (REP), a bijection of answers, for two descriptions of the same events; Batch independence (BAT), the identity, when only the other questions change; Probability measure (MEA), a many-to-one map of outcomes or the additive identity it induces, for different events of one measure; Logical coherence (LOG), the order induced by entailment and Boolean combination; and Choice-set coherence (CHO), restriction to a sub-menu, because a Choice is the model's distribution given that the answer is one of its options.
  • What it edits: the question key, its instructions, its criteria (options or levels), its answer type, or the rest of the batch. Never the state: it is the outside input and is fixed for every test.

Checks on auxiliary response fields (confidence_consistency, routing_stability) test the output format rather than the distribution. They are diagnostics: outside the dimensions, never scored, and run only with --relations diagnostics (or by name).

Membership follows the operation, not the data. The mathematical definitions, the full list of laws and the reasons further candidate laws are not relations are in the technical report (paper/).

The diagram, the coverage matrix and the tables below are generated by jevbench taxonomy --write README.md; a test fails if they drift from the code. jevbench relations -v prints the same tree with each relation's expectation and source.

50 relations in 5 dimensions and 11 groups, plus 2 unscored diagnostics. Each relation is one transformation with one law; each dimension is defined by the operation that links the two distributions its laws compare.

Dimension Title Compares Operation Relations
REP Representation consistency two questions describe the same events bijection of answers: P′ = σ#P 15
BAT Batch independence the question is fixed and only its companions change `identity: P_q B′ = P_q
MEA Probability measure coherence the questions ask about different events of one measure many-to-one map of outcomes, or the additive identity it induces 17
LOG Logical coherence the questions are propositions ordered by entailment order: A ⊨ B ⇒ P(A) ≤ P(B), and the bounds of Boolean combinations 10
CHO Choice-set coherence the menu a Choice is conditioned on changes `restriction to a sub-menu: P′ K = P
flowchart LR
    root(["JevBench relations"])
    REP["<b>REP · Representation consistency</b><br/><i>bijection of answers: P′ = σ#P</i>"]
    root --> REP
    REP_invariance["<b>REP.invariance · Invariance</b><br/>● instruction_paraphrase<br/>● cross_lingual<br/>● typo_noise<br/>● verbosity<br/>● key_rename<br/>● description_paraphrase<br/>● de_morgan"]
    REP --> REP_invariance
    REP_equivariance["<b>REP.equivariance · Equivariance</b><br/>● option_permutation<br/>● option_rename<br/>● clone_symmetry"]
    REP --> REP_equivariance
    REP_binding["<b>REP.binding · Binding</b><br/>● key_instruction_conflict<br/>● name_description_conflict<br/>● description_swap"]
    REP --> REP_binding
    REP_type["<b>REP.type · Answer type</b><br/>● noul_as_choice<br/>● noul_as_score"]
    REP --> REP_type
    BAT["<b>BAT · Batch independence</b><br/><i>identity: P_q|B′ = P_q|B</i>"]
    root --> BAT
    BAT_isolation["<b>BAT.isolation · Isolation</b><br/>● batch_solo<br/>● batch_order<br/>● duplicate_question<br/>● batch_content<br/>● batch_size_sweep"]
    BAT --> BAT_isolation
    MEA["<b>MEA · Probability measure coherence</b><br/><i>many-to-one map of outcomes, or the additive identity it induces</i>"]
    root --> MEA
    MEA_complement["<b>MEA.complement · Complement</b><br/>● negation_wrapper<br/>● natural_negation<br/>● choice_complement<br/>● score_complement<br/>● compound_negation"]
    MEA --> MEA_complement
    MEA_partition["<b>MEA.partition · Partition</b><br/>● option_merge<br/>● option_split<br/>● option_clone<br/>● similarity_effect<br/>● partition_hierarchy<br/>● score_granularity<br/>● extreme_level<br/>● option_closure"]
    MEA --> MEA_partition
    MEA_marginalisation["<b>MEA.marginalisation · Marginalisation</b><br/>● score_cumulative<br/>● choice_indicator<br/>● pair_choice_marginals<br/>● inclusion_exclusion"]
    MEA --> MEA_marginalisation
    LOG["<b>LOG · Logical coherence</b><br/><i>order: A ⊨ B ⇒ P(A) ≤ P(B), and the bounds of Boolean combinations</i>"]
    root --> LOG
    LOG_bounds["<b>LOG.bounds · Bounds</b><br/>● frechet_and_upper<br/>● nary_and_upper<br/>● nary_and_lower<br/>● nary_or_upper<br/>● nary_or_lower<br/>● count_upper<br/>● count_lower"]
    LOG --> LOG_bounds
    LOG_monotonicity["<b>LOG.monotonicity · Monotonicity</b><br/>● threshold_sweep<br/>● entailment_strength<br/>● quantifier_monotonicity"]
    LOG --> LOG_monotonicity
    CHO["<b>CHO · Choice-set coherence</b><br/><i>restriction to a sub-menu: P′|K = P|K</i>"]
    root --> CHO
    CHO_conditioning["<b>CHO.conditioning · Conditioning</b><br/>● irrelevant_option<br/>● remove_option_iia<br/>● pairwise_iia"]
    CHO --> CHO_conditioning
    classDef planned stroke-dasharray: 5 4,color:#777

Coverage by edited component. Each mark is one relation that edits that part of the request. Empty cells (·) are combinations not yet tested. The state is not a column: no relation edits it.

Group key instructions criteria type batch
REP.invariance Invariance ● ●●●●● ●● · ●
REP.equivariance Equivariance · · ●●● · ·
REP.binding Binding ● · ●● · ·
REP.type Answer type · · ●● ●● ·
BAT.isolation Isolation · · · · ●●●●●
MEA.complement Complement · ●●●●● · ●● ●●●●●
MEA.partition Partition · · ●●●●●●●● · ·
MEA.marginalisation Marginalisation · ●●●● · ●●● ●●●●
LOG.bounds Bounds · ●●●●●●● · · ●●●●●●●
LOG.monotonicity Monotonicity · ●●● · · ●●●
CHO.conditioning Conditioning · · ●●● · ·

Forms of law: E distributional equality; I identity; L inequality; field field agrees with probabilities.

REP · Representation consistency: bijection of answers: P′ = σ#P

Two questions describe the same events.

Group Relation Edits Form Law Check Variants
REP.invariance Invariance ● instruction_paraphrase instructions E judged paraphrase of the instructions tv 1 rewrite (bank)
REP.invariance Invariance ● cross_lingual instructions, criteria E question translated, state untouched tv 1 rewrite (bank)
REP.invariance Invariance ● typo_noise instructions E typos, casing, punctuation tv 3 templates
REP.invariance Invariance ● verbosity instructions E redundant clarification added tv 3 templates
REP.invariance Invariance ● key_rename key E neutral question key tv random
REP.invariance Invariance ● description_paraphrase criteria E judged paraphrase of one option tv 1 rewrite (bank)
REP.invariance Invariance ● de_morgan instructions, batch E not (A and B) equals not A or not B nand_vs_or_not random
REP.equivariance Equivariance ● option_permutation criteria E options reordered, Score levels reversed tv random
REP.equivariance Equivariance ● option_rename criteria E neutral option ids, same descriptions tv fixed
REP.equivariance Equivariance ● clone_symmetry criteria E two identical options get equal probability clone_symmetry random
REP.binding Binding ● key_instruction_conflict key E negated or swapped question keys tv 2 templates
REP.binding Binding ● name_description_conflict criteria E option ids exchanged, descriptions fixed tv random
REP.binding Binding ● description_swap criteria E descriptions exchanged, ids fixed tv random
REP.type Answer type ● noul_as_choice type, criteria E Noul versus yes/no Choice abs_diff random
REP.type Answer type ● noul_as_score type, criteria E Noul versus two-level Score abs_diff 2 templates

BAT · Batch independence: identity: P_q|B′ = P_q|B

The question is fixed and only its companions change.

Group Relation Edits Form Law Check Variants
BAT.isolation Isolation ● batch_solo batch E asked alone versus in its batch tv fixed
BAT.isolation Isolation ● batch_order batch E question order reversed tv fixed
BAT.isolation Isolation ● duplicate_question batch E same question twice in one request within_request_tv fixed
BAT.isolation Isolation ● batch_content batch E other questions replaced, size fixed tv fixed
BAT.isolation Isolation ● batch_size_sweep batch E batches of 2, 5 and 10 questions tv every size

MEA · Probability measure coherence: many-to-one map of outcomes, or the additive identity it induces

The questions ask about different events of one measure.

Group Relation Edits Form Law Check Variants
MEA.complement Complement ● negation_wrapper instructions, batch I P(Q) + P(not Q) = 1, templates complement_gap 3 templates
MEA.complement Complement ● natural_negation instructions, batch I P(Q) + P(Q') = 1, natural opposite complement_gap 1 rewrite (bank)
MEA.complement Complement ● choice_complement type, instructions, batch I Noul 'not o' = 1 - P(o) complement_gap 3 templates
MEA.complement Complement ● score_complement type, instructions, batch I Noul 'not level k' = 1 - P(k) complement_gap 2 templates
MEA.complement Complement ● compound_negation instructions, batch I P(not (A and B)) + P(A and B) = 1 complement_gap random
MEA.partition Partition ● option_merge criteria E 'a or b' gets P(a) + P(b) tv_to_coarsened random
MEA.partition Partition ● option_split criteria E 'o and X' + 'o and not X' = P(o) tv_after_collapse random
MEA.partition Partition ● option_clone criteria E a clone and its original keep P(o) tv_after_collapse random
MEA.partition Partition ● similarity_effect criteria E a near-duplicate and its original keep P(b) tv_after_collapse random
MEA.partition Partition ● partition_hierarchy criteria E coarse blocks = sums of their options tv_to_coarsened random
MEA.partition Partition ● score_granularity criteria E adjacent levels merged keep their sum tv_to_coarsened fixed
MEA.partition Partition ● extreme_level criteria E a level beyond the top refines the top level tv_after_collapse fixed
MEA.partition Partition ● option_closure criteria E each option in turn replaced by a catch-all catch_all_tv every option
MEA.marginalisation Marginalisation ● score_cumulative type, instructions, batch I 'level k or higher' = upper tail, every k tail_gap_max* 2 templates
MEA.marginalisation Marginalisation ● choice_indicator type, instructions, batch I Noul 'is it o?' = P(o), every o indicator_gap 2 templates
MEA.marginalisation Marginalisation ● pair_choice_marginals type, instructions, batch I four-way pair Choice has the Nouls as marginals marginal_gap random
MEA.marginalisation Marginalisation ● inclusion_exclusion instructions, batch I P(A and B) + P(A or B) = P(A) + P(B) inclusion_exclusion 3 templates

LOG · Logical coherence: order: A ⊨ B ⇒ P(A) ≤ P(B), and the bounds of Boolean combinations

The questions are propositions ordered by entailment.

Group Relation Edits Form Law Check Variants
LOG.bounds Bounds ● frechet_and_upper instructions, batch L P(A and B) <= min(P(A), P(B)) excess 3 templates
LOG.bounds Bounds ● nary_and_upper instructions, batch L P(all) <= min P(Ai) excess random
LOG.bounds Bounds ● nary_and_lower instructions, batch L P(all) >= sum P(Ai) - (n - 1) excess random
LOG.bounds Bounds ● nary_or_upper instructions, batch L P(any) <= min(1, sum P(Ai)) excess random
LOG.bounds Bounds ● nary_or_lower instructions, batch L P(any) >= max P(Ai) excess random
LOG.bounds Bounds ● count_upper instructions, batch L P(at least two) <= sum P(Ai) / 2 excess random
LOG.bounds Bounds ● count_lower instructions, batch L P(at least two) >= (sum P(Ai) - 1) / (n - 1) excess random
LOG.monotonicity Monotonicity ● threshold_sweep instructions, batch L 'more than k' falls with k monotonicity declared or counted
LOG.monotonicity Monotonicity ● entailment_strength instructions, batch L Q1 entails Q2 => P(Q1) <= P(Q2) chain fixed
LOG.monotonicity Monotonicity ● quantifier_monotonicity instructions, batch L all <= at least two <= at least one chain random

CHO · Choice-set coherence: restriction to a sub-menu: P′|K = P|K

The menu a Choice is conditioned on changes.

Group Relation Edits Form Law Check Variants
CHO.conditioning Conditioning ● irrelevant_option criteria E off-topic option added iia_tv random
CHO.conditioning Conditioning ● remove_option_iia criteria E each option in turn removed iia_tv every option
CHO.conditioning Conditioning ● pairwise_iia criteria E each pair of options asked alone iia_tv every pair

Diagnostics (not scored)

Checks on auxiliary response fields. They test the output format, not the distribution, so they belong to no dimension, never enter a score and run only when named (--relations diagnostics).

Group Relation Edits Form Law Check Variants
DIAG.fields Response fields ● confidence_consistency - field confidence is a function of the probabilities fits_definition fixed
DIAG.fields Response fields ● routing_stability - field 0.8 routing survives key, order, batch edits route_flip fixed

Form: (E) distributional equality, (I) identity, (L) inequality. Check: the one quantity the relation measures. Variants: a law stated for every template, option, pair or size holds for a question only if it holds for every variant; its measure is the largest.

* score_cumulative/tail_gap_max: largest gap over the levels k; strict, so read it with the graded score.

Tensions between expectations

Some expectations pull against each other for a given model class. A Luce-type scorer (independent option scores, softmax) satisfies conditioning on a changed menu (irrelevant_option, remove_option_iia, pairwise_iia) exactly, and fails every refinement of an option by construction: option_clone, option_split, similarity_effect, extreme_level and option_closure, the red-bus/blue-bus problem. The bundled fake:coherent backend is such a scorer, and the test suite checks exactly this.

Templates and rewrites

Probe relations run every template as a separate variant. A test passes only if every template does; the report still breaks violation rates down by template, so a failure can be attributed to a model rather than to one wording.

Natural rewrites are generated once and frozen:

jevbench rewrite --cases my-cases.jsonl --kinds paraphrase,negation,description,translate:zh \
  --chat-url https://<host>/v1/chat/completions --model <chat model> --out rewrites/my-bank.jsonl
jevbench eval --cases my-cases.jsonl --url ... --rewrites rewrites/my-bank.jsonl --out runs/x.json

The chat model proposes candidates and then judges each one (both directions for paraphrases and negations; faithfulness plus unchanged structure for translations). Every candidate is kept in the bank with its verdict; runs use only accepted ones. Set "audited": true or false after human review; --audited-only restricts a run to reviewed rewrites. The banks of the bundled suites (src/jevbench/suites/*/rewrites.jsonl) were generated and judged by gpt-oss-20b and are not yet human-audited.

Numeric thresholds default to counting digit groups in the state. Declare real quantities in a case to probe them instead: "thresholds": [{"subject": "failed SSH login attempts", "value": 214}].

Writing a relation: subclass jevbench.relations.Relation, decorate it with @register, and implement applicable, build (return a Trial of requests) and evaluate (return one Outcome per tested question, with a deviation and a tolerance). Set templates to run several wordings, or override variants. A relation that states another law about the same questions subclasses the first and sets rng_name to its name, so both read the same requests.

Scope and limits

  • A violation is a property of a model and a wording. Report per-template rates, and prefer claims that hold across templates and natural rewrites.
  • The rewrite judge is an LLM. Its verdicts are recorded, not trusted: audit a sample before publishing, and report results on audited rewrites.
  • Coherence is necessary, not sufficient: a model that ignores its input can be perfectly invariant. The sensitivity relations (description_swap, name_description_conflict, key_instruction_conflict) and threshold_sweep check, from the question side, that answers move when they should and only then.
  • Hosted services have terms of use. Check them before publishing results.

Authors

JevBench is developed by Chen Feng at the ML Lab, Queen's University Belfast (c.feng@qub.ac.uk). Issues and contributions: https://github.com/JevBench/jevbench.

License

Apache-2.0. See the LICENSE and NOTICE files shipped with the package.

Metadata

Release files for jevbench 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for jevbench 0.1.0
File Size Uploaded Provenance Actions
jevbench-0.1.0.tar.gz 1.4 MB Without provenance Details for jevbench-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for jevbench 0.1.0
File Interpreter ABI Platform Actions
jevbench-0.1.0-py3-none-any.whl Python 3 none any Details for jevbench-0.1.0-py3-none-any.whl

Total release size: 2.8 MB

Release files / jevbench-0.1.0.tar.gz

Download URL jevbench-0.1.0.tar.gz
Size 1.4 MB
Tags Source
SHA-256 checksum
How to use checksums
45460eae2c3e94a8ed980f04eba9cd2eedc13a3942b3ff8537b6665924e4d655
BLAKE2b-256 checksum
How to use checksums
d55a48842bc918d5a5ed511c1f9c3d10c588900ec48c66c1f48db848c2f8a55d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.9

Release files / jevbench-0.1.0-py3-none-any.whl

Download URL jevbench-0.1.0-py3-none-any.whl
Size 1.4 MB
Tags Python 3
SHA-256 checksum
How to use checksums
4bf93419a987d09238b1427b70ac46ba16fb5f36206e67a27f4b370d459568e5
BLAKE2b-256 checksum
How to use checksums
81127664f89675907f3ca6c586cbf4a2777905732d941d49f1f358eb27387db5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.9

Security for jevbench 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page