pytest-jev

Semantic assertions for pytest. Test what your LLM app's output means ("apologizes", "offers a refund", "doesn't leak the system prompt") instead of the exact words. Each claim is judged by Jev, TypeSafe's model that returns calibrated probabilities instead of text.

def test_refund_reply(jev):
    reply = support_bot("I was charged twice for order #1042.")

    jev.expect(
        reply,
        holds=["apologizes to the customer", "says the duplicate payment was refunded"],
        lacks=["blames the customer", "asks for a password or a full card number"],
    )

When a prompt change breaks the reply, the failure says which claim broke and how sure Jev was:

>       jev.expect(
E       AssertionError: jev: 3 of 4 claims failed
E       text: "Double charges happen when you click twice. You'll get store credit within 24 hours."
E         ✗ holds  p=0.04  apologizes to the customer  (needs >= 0.80)
E         ✗ holds  p=0.21  says the duplicate payment was refunded  (needs >= 0.80)
E         ✗ lacks  p=0.79  blames the customer  (needs <= 0.20)
E         ✓ lacks  p=0.01  asks for a password or a full card number
------------------------------------- jev --------------------------------------
jev: 4 questions · 1 request · 365 input tokens · $0.000015 · 1.40 s in Jev · typesafe/jev-1.13-20260917 via openrouter

Every output in this README is from a real run of examples/ against jev-1.13.

Why

String assertions break every time the model rewords a reply. Using an LLM as the judge works, but it's slow, costs real money per test, and returns text you then have to parse. pytest-jev sends each check to Jev instead:

• One request per text. Every claim in jev.expect goes into a single request, answered in parallel.
• Typed answers. Jev answers each claim with a probability, picks from the options you list, or rates on the levels you define. There is no output to parse and nothing outside the answer space.
• Cheap enough for every commit. Jev costs $0.042 per million input tokens and output is free. The 7 tests in examples/test_support_bot.py ran in 3.9 s for $0.0001. The summary line prints the tokens and cost of every run.
• No passing on a coin flip. A claim holds at p ≥ 0.8 and lacks at p ≤ 0.2. When Jev is unsure, both fail.
• Free, stable reruns. Answers are cached in .pytest_cache, so rerunning unchanged tests makes no requests and gives the same verdicts.

Install

pip install pytest-jev

It needs Python 3.10+ and pytest 7.4+.

Set one API key:

export TYPESAFE_API_KEY=...     # https://console.typesafe.ai (early access)
export OPENROUTER_API_KEY=...   # or https://openrouter.ai/settings/keys (no waitlist)

Without a key, tests that use jev are skipped (see CI).

Usage

The jev fixture has five methods. Each sends one request and returns a result that works in a plain assert.

holds and lacks: one claim

def test_reply_confirms_the_refund(jev):
    reply = support_bot("I was charged twice for order #1042.")
    assert jev.holds(reply, "says the duplicate payment was refunded")
    assert jev.lacks(reply, "asks for a password or a full card number")

With the broken reply from above:

E       assert <jev holds 'says the duplicate payment was refunded': p=0.16, needs >= 0.80>

The result also carries the probability: jev.holds(reply, "...").p.

expect: many claims, one request

jev.expect(reply, holds=["apologizes", "offers a refund"], lacks=["blames the customer"])

It fails with a report of every claim, as shown at the top. It returns the claims when they pass.

context: check the text against something else

Extra state goes in context, such as a policy or the documents a RAG app retrieved. A claim can name it in backticks:

def test_reply_matches_the_policy(jev):
    reply = support_bot("I was charged twice for order #1042.")
    assert jev.lacks(reply, "contradicts the policy in `policy`", context={"policy": REFUND_POLICY})

The broken reply promises store credit in 24 hours; the policy says refunds to the card in 5 business days:

E       assert <jev lacks 'contradicts the policy in `policy`': p=0.95, needs <= 0.20>

The text under test is always text, so context can't use that key.

choice: which option fits

TEAMS = {
    "billing": "Payments, charges, invoices and refunds",
    "technical": "Bugs, errors, crashes and integrations",
    "account": "Logins, passwords and account settings",
    "other": "Anything that fits none of the teams above",
}


def test_checkout_errors_go_to_billing(jev):
    ticket = "Your checkout page throws a 500 error when I enter my card."
    assert jev.choice(ticket, "Which team should handle this ticket?", TEAMS) == "billing"

E       AssertionError: assert jev chose 'technical', not 'billing'
E         question: Which team should handle this ticket?
E           technical  0.90  ██████████████████░░
E           billing    0.10  ██░░░░░░░░░░░░░░░░░░
E           account    0.00  ░░░░░░░░░░░░░░░░░░░░
E           other      0.00  ░░░░░░░░░░░░░░░░░░░░
E         confidence 0.87

Sometimes the failure means the test's expectation needs another look: a 500 error at checkout is arguably a bug first.

Options can be a dict of label to description, or a plain list of labels. Comparing with a label that isn't an option (team == "biling") raises an error instead of quietly failing.

score: rate on ordered levels

POLITENESS = {
    "rude": "Rude, dismissive or blaming the customer",
    "neutral": "Neutral and matter-of-fact, no warmth",
    "warm": "Warm and polite, acknowledges the customer's frustration",
}


def test_reply_is_warm(jev):
    tone = jev.score(reply, "How polite is this support reply?", POLITENESS)
    assert tone >= "warm"

Levels go lowest first. The comparison is probabilistic: tone >= "warm" passes when Jev puts at least 80% of its probability on "warm" or higher. >, <=, <, == and != work the same way, with labels or level indices. The broken reply passes tone >= "neutral" (0.83) but not this:

E       AssertionError: assert jev gave P(level >= 'warm') = 0.00, needs >= 0.80
E         question: How polite is this support reply?
E           0 rude     0.17  ███░░░░░░░░░░░░░░░░░
E           1 neutral  0.83  █████████████████░░░
E           2 warm     0.00  ░░░░░░░░░░░░░░░░░░░░
E         expected level 0.84, confidence 0.75

Thresholds and models

The threshold defaults to 0.8: holds needs p ≥ 0.8, lacks needs p ≤ 0.2. It must be between 0.5 and 1. Set it per call, per test, or for the whole run:

assert jev.holds(reply, "offers a refund", threshold=0.9)


@pytest.mark.jev(threshold=0.9, model="jev-1.13")
def test_strict(jev): ...

# pytest.ini (or [tool.pytest.ini_options] in pyproject.toml)
[pytest]
jev_model = jev-1.13
jev_threshold = 0.85

jev-latest changes when TypeSafe ships a new version, so pin jev-1.13 when runs must be reproducible.

CI and running without a key

• Tests that use jev get the jev marker automatically. pytest -m "not jev" runs everything else offline.
• Without a key, jev tests are skipped and say why. In CI, pass --jev-require (or set jev_require = true) so a missing secret fails the build instead.
• Answers are cached by model, text, context and question. Pass --jev-no-cache to ask again.

Use another backend

Requests go through the session-scoped jev_client fixture. Override it in conftest.py with anything that has the TypeSafe SDK's system_one(state=, questions=, model=) method, such as a fake for offline unit tests, or system-one-adapter to run the same assertions through an LLM and compare:

@pytest.fixture(scope="session")
def jev_client():
    return MyFakeJev()

Writing claims that work

Jev reads claims literally (Jev 1.13 known limits):

• One condition per claim. Write "apologizes" and "offers a refund" as two claims, not one joined with "and".
• Say exactly what you mean. "Says the duplicate payment was refunded" works better than "handles the refund correctly".
• Keep numbers, counts and dates in code. assert "5 business days" in reply is exact; Jev is not a calculator.
• Name the context. "contradicts docs" points Jev at the right part of the state.

How it works

Each call sends one request to Jev's /v1/systemone endpoint with state = {"text": text, **context}. Every claim becomes a Noul question, Does `text` satisfy: <claim>?, which returns the probability it is true. choice sends a Choice question and score sends a Score question. The thresholds and comparisons are ordinary Python in this plugin.

Limitations

• Jev can be wrong. Treat a threshold as a policy you tune on your own cases, and read the failure report before trusting a pass or fail.
• Jev's probabilities move a little between calls. In five calls while this README was written, "says the duplicate payment was refunded" scored between 0.16 and 0.23 on the same reply. The unsure band between 0.2 and 0.8 absorbs this, and the cache keeps reruns identical.
• Text only: no images or audio.
• The text and context you assert on are sent to TypeSafe or OpenRouter. Keep secrets and personal data out of test fixtures.
• Not affiliated with or endorsed by TypeSafe AI.

Development

uv sync
uv run pytest          # offline: a fake Jev answers every question
uv run ruff check .

License

MIT