jevper

The Jev interface — state in, typed questions (noul, choice, score) out, answers carrying probabilities and confidence — on top of any OpenAI-compatible model.

Same call as typesafe-sdk, different backend: point jevper at a hosted LLM or a self-hosted llama.cpp server and code written for Jev keeps working, unchanged.

It does not call the hosted TypeSafe API and does not depend on typesafe-sdk or openai at runtime — the client object is duck-typed. Any object exposing responses.create or chat.completions.create works, including a self-hosted llama.cpp server.

jevper is an independent implementation of the documented System One wire format. It is not affiliated with, endorsed by, or supported by TypeSafe AI — questions about the API itself belong in their docs.

from openai import OpenAI
from jevper import Choice, SystemOneClient

client = SystemOneClient(OpenAI(), model="gpt-5.6-terra", method="logprobs")

response = client.system_one(
    state="I was charged twice for the same subscription this month.",
    questions={
        "intent": Choice(
            instructions="Pick the intent of the message.",
            criteria={
                "billing": "money, invoices, refunds, charges",
                "technical": "errors, crashes, login or performance problems",
                "sales": "pricing, plans, purchasing, upgrades",
            },
        )
    },
)

answer = response.answers["intent"]
answer.choice        # "billing"
answer.probabilities # {"billing": 0.88, "technical": 0.08, "sales": 0.03}
answer.confidence    # 0.83

Install

pip install jevper

Python 3.10+. The only runtime dependency is pydantic>=2.7.

For development:

git clone https://github.com/zhulinchng/jevper && cd jevper
uv venv && uv pip install -e '.[test]'
pytest -q

What one call does

flowchart LR
    A["state + questions"] --> B["build_parts + assemble: system prompt, few-shot turns, question block, state turns"]
    B --> C{"method (auto resolves first)"}
    C -->|logprobs| D["logprobs=true, top_logprobs=20"]
    C -->|grammar| E["+ GBNF grammar in extra_body"]
    C -->|structured| F["strict JSON schema: probabilities"]
    C -->|discrete| G["strict JSON schema: one label"]
    D --> H["first label token -> softmax over the labels"]
    E --> H
    F --> I["probability dict from JSON"]
    G --> J["one-hot from the chosen label"]
    H --> K["Answer: choice / noul / score"]
    I --> K
    J --> K

Each question becomes its own provider call, so questions are independent and run concurrently (max_concurrency, default 8). Answers come back keyed by your question ids, in insertion order.

Questions

Three types, mirroring the Jev API — Noul answers yes/no with one probability, Choice picks one of your labelled options, Score rates on an ordered scale:

Score.score is the probability-weighted level index (Σ i·pᵢ, levels zero-based), as in the Jev API. Choice takes up to 255 options, the Jev API limit. The two methods that read a label token — logprobs and grammar — stop at 26, because the first token of "AA" is "A"; past 26 options they raise InvalidQuestionError pointing at structured and discrete, which answer in JSON and use two-letter labels. The default method="auto" never hits that error: it answers a wide Choice in JSON.

Questions can also be passed as raw mappings ({"type": "choice", "criteria": {...}}) and are validated the same way.

Methods

method= decides how the decision is elicited. All four share the same label→option mapping, so switching methods does not change your types; only the label alphabet differs (logprobs and grammar need single-letter labels, so they cap at 26 options).

auto is the default because logprobs are not universal: OpenAI's reasoning models reject them (logprobs are not supported with reasoning models.), Anthropic and Gemini's OpenAI-compatibility endpoint never had them, and a model that returns a logprob with no alternatives gives you no distribution at all. auto reads the logprobs where they exist — they are one short call and the model's real distribution rather than a self-report — and answers in JSON where they do not, remembering the verdict per model and surface. See docs/methods.md for the provider table, the exact request bodies, the readout rules and the failure modes.

The same holds for the request fields jevper adds: a server that refuses structured output, the reasoning parameters, the Responses include list or the cache key gets that field dropped and the call re-asked, so a partially implemented server answers instead of failing. debug["server_limits"] reports what it refused.

Reasoning

Pass reasoning=ReasoningConfig(...) to make the model think before it classifies:

from jevper import ReasoningConfig, reasoning_text

client = SystemOneClient(OpenAI(), model="gpt-5.6-terra", reasoning=ReasoningConfig(effort="medium"))

response = client.system_one(state=..., questions=...)
reasoning_text(response.reasoning)   # the trace, as text

mode="auto" (the default) uses native provider reasoning on the Responses surface and a two-step think-then-classify path on Chat Completions, where the analysis text is replayed as an assistant turn before the answer. The trace always lands on response.reasoning, and the two-step analysis call's usage is counted in response.usage. See docs/reasoning.md.

Few-shot examples

Examples are chat turns (question block + example state, then the expected answer), so the demonstration is always in the format the active method expects. They can be attached at three levels:

from jevper import Choice, Example, SystemOneClient

question = Choice(
    criteria={"billing": "...", "technical": "..."},
    examples=[Example(state="Charged twice for one order", answer="billing")],
)

client = SystemOneClient(OpenAI(), model="gpt-5.6-terra",
                         examples=[Example(state="Login fails", answer="technical")])  # fallback for every question

client.system_one(state=..., questions={"intent": question},
                  examples={"intent": [...]})  # or a bare sequence for all questions

Precedence is question → per call → constructor, and the first non-empty level wins. examples is excluded from model_dump(), so question dumps keep exactly the Jev wire keys. See docs/few-shot.md.

Response

response.model                 # the model id jevper asked for
response.answers               # {"intent": ChoiceAnswer(...)}
response.nouls / .choices / .scores   # filtered views
response.usage                 # input_tokens, output_tokens, reasoning_tokens, cached_tokens, n_calls, n_retries, latency
response.reasoning             # tuple[ReasoningContentPart, ...]
response.debug                 # per-attempt requests/responses, retry reasons, normalization notes

response.model_dump_json() serializes to the Jev answer shape — the answer field names and JSON keys match POST /v1/systemone. Token counts are None when any constituent call omitted them; n_calls counts the provider calls that returned a result, including analysis passes and corrective retries, while n_retries counts transient-failure retries only. A failed attempt appears in debug["llm_attempts"] but not in usage. See docs/api.md for the full reference.

Prompt caching

Every provider that serves these calls caches the prefix of a prompt and reuses it for the next request that starts the same way, and jevper is shaped for it: the state comes last, so the system prompt, the few-shot examples and the question block are identical across every state classified with one rubric.

client = SystemOneClient(OpenAI(), model="gpt-5.6")

client.system_one(state=record_a, questions=rubric)
client.system_one(state=record_b, questions=rubric)   # the shared prefix is reused

Two things make it steerable and observable:

• prompt_cache_key is sent with every request, derived per question from the parts of the prompt that do not change between calls — model, method, examples, question block — so a rubric's requests are routed together, and a logprobs request is not routed with a structured one whose prefix differs. Pass your own to group or account for them your way, on the client (prompt_cache_key="tenant-42") or per call. A server that refuses the field gets it dropped and the call re-asked, like the other optional fields.
• usage.cached_tokens is the prompt tokens the provider read from its cache, summed over the call. None means the provider said nothing — vLLM needs --enable-prompt-tokens-details, and SGLang's Chat Completions route needs --enable-cache-report — while a reported 0 means a cold or disabled cache.

Measured on one 2388-token prompt carrying two examples, second call differing only in the state: reused tokens went from 40 — the system prompt alone — to 1010 on llama.cpp, 528 on vLLM and 896 on SGLang once the state moved to the end. Per-server flags, what each server accepts or ignores, and how to isolate a cache with cache_salt: docs/local-servers.md.

Anthropic-compatible servers

Every server in the local fleet serves the Anthropic Messages API at /v1/messages as well as the OpenAI ones. jevper speaks it with api="messages": point the anthropic client at the server and pass it in place of the OpenAI one.

from anthropic import Anthropic

from jevper import SystemOneClient

client = SystemOneClient(Anthropic(base_url="http://127.0.0.1:1234"), model="qwen3-4b-instruct")

client.system_one(state=record, questions=rubric, api="messages", method="structured")

Four things differ from the OpenAI surfaces, and all four come from the protocol rather than from any server:

• No logprobs exist in it. Not withheld by some servers — absent from the API. method="logprobs" and method="grammar" raise UnsupportedMethodError before a request is sent, and method="auto" answers in JSON without spending a call to find out. structured and discrete work exactly as they do elsewhere: the prompt already asks for one JSON object.
• There is no schema field either, so structured/discrete put the JSON Schema in the system prompt. The answer's shape is then only as good as the model's instruction-following, where the other surfaces constrain it in the request itself.
• max_tokens has no server-side default. jevper sends 1024 — or 1024 plus the caller's thinking budget, because Anthropic requires the budget to be strictly below max_tokens and would otherwise refuse the 1024 its own docs call the floor. extra_body={"max_tokens": n} overrides both.
• Thinking is asked for with a budget, not an effort name. ReasoningConfig(budget_tokens=2048) sends thinking={"type": "enabled", "budget_tokens": 2048}: a budget is the only reason to ask for this surface's own thinking, so it selects it even under mode="auto". A server that refuses the field gets it dropped and the call re-asked, reported in debug["server_limits"]["thinking"]; a server that refuses the value — SGLang answers budget_tokens: must be at least 1024 — gets its own error back instead, because a bad number is not a missing field.

Thinking blocks come back as ordinary response.reasoning parts with the block's signature kept, and usage.cached_tokens is read from cache_read_input_tokens. Which servers implement the route, and since which version: docs/local-servers.md.

Failures

Local problems fail before any request is sent: an invalid question, an empty questions mapping, an unusable state, or grammar on a surface that cannot carry a grammar.

Transient failures (HTTP 408/429/500/502/503/504/529, connection and timeout errors — including the httpx transport errors whose class names carry neither word) are retried per call with RetryPolicy(n_retries=2, base_delay=0.5, max_delay=8.0) and exponential backoff min(base_delay · 3ⁿ, max_delay). Unreadable answers get one corrective retry (n_retry_malformed) with the failure appended to the conversation. ProviderError propagates after all questions have settled, in question insertion order.

Verification

pytest -q                      # the whole suite runs against a local stub HTTP server; no network, no API keys
ruff check src tests           # clean except three PYI034 hints (see docs/internals.md)

The suite drives a real openai SDK client at a stdlib ThreadingHTTPServer stub, so the SDK's own serialization path is exercised; see docs/internals.md.

Optional live check, skipped unless both variables are set:

LLM_MODEL=gpt-5.6-terra OPENAI_API_KEY=... pytest -q tests/test_live.py

Optional MLflow check, skipped unless MLflow is installed — tracing, hosting jevper as a model, the AI Gateway, and mlflow.genai.evaluate (see docs/mlflow.md):

uv pip install -e '.[test,mlflow]' && pytest -q tests/test_mlflow.py

Docs

• docs/api.md — constructor and system_one parameters, answer/usage/debug shapes, errors
• docs/methods.md — the four methods, request bodies, readout rules, surface selection
• docs/local-servers.md — ollama, llama.cpp, vLLM and SGLang: what to pass, turning thinking off, what fits a small GPU
• docs/reasoning.md — native vs two-step reasoning, traces, encrypted content
• docs/few-shot.md — example levels, precedence, rendering, structured examples
• docs/internals.md — module map, call flow, concurrency, retries, testing
• docs/mlflow.md — MLflow 3.16.1: autolog tracing of jevper's calls, hosting jevper as a model, the AI Gateway, mlflow.genai.evaluate

License

Apache-2.0 — see LICENSE.