jevify

Make any model you already have answer typed questions in one pass, the way Jev does, and measure how well it does.

You bring a model behind an OpenAI-compatible endpoint (vLLM, llama.cpp, TensorFold, anything that returns logprobs for a reply token or for the prompt) or a small model that runs in-process. jevify sends it a state once, then asks typed questions about it: a choice among options, a score on ordered levels, a yes/no with a probability. Each answer is read straight off the model's next-token distribution, no generation, no parsing, one request per question. The result comes back in the exact shape of TypeSafe's Jev API, so the unmodified typesafe-sdk works against it.

jevify owns no weights and trains nothing. A recipe says how a model is asked, a probe proves what the deployment can do, and a scorecard says how it did, with the command, hashes and raw file that produced every number.

   state (text, images)          one warm request, cached prefix
        │
        ├── "Which team handles this?"  ──►  {billing: 0.94, support: 0.05, sales: 0.01}
        ├── "How urgent?"               ──►  {low: 0.05, medium: 0.10, high: 0.85}
        └── "Threatens to cancel."      ──►  p(true) = 0.95
                                              each ≈ 30–140 ms on a local GPU

Quickstart

uv tool install jevify                # or: pipx install jevify
git clone https://github.com/fidecastro/jevify && cd jevify

# 1. point a recipe at your server (copy the nearest one under recipes/)
jevify probe recipes/gemma-4-e4b-it.llamacpp.yaml

# 2. ask
jevify ask recipes/gemma-4-e4b-it.llamacpp.yaml \
  --state "Customer: I was charged twice. Fix it today or I cancel." \
  --choice "dept:Which team should handle this?=billing,support,sales" \
  --score  "urgency:How urgent is this?=low,medium,high" \
  --noul   "cancel:The customer threatens to cancel." \
  [--image screenshot.png]

# 3. serve Jev's API and use the stock SDK
jevify serve recipes/gemma-4-e4b-it.llamacpp.yaml --port 8600

Any model behind an OpenAI-compatible server works, not only the ones under recipes/: copy the nearest recipe, set endpoint.base_url and model.name, and run the probe, which re-verifies the answer token ids for that model and writes what it found into the file. vLLM and llama.cpp get their dialect extras; any other server is driven through the plain OpenAI surface. A server that refuses logprobs on the reply token but returns vLLM's prompt_logprobs, such as TensorFold's native server with GLM-5.3-Flash, is read through the prompt_logprobs rung on /v1/completions: with a raw-mode recipe (the template written out with the model's own control tokens), or with a chat-template recipe where the server also takes messages on that route, as TensorFold's does. The chat form is the one that carries images.

serve binds to 127.0.0.1. To reach it from other machines pass --host 0.0.0.0, and set JEVIFY_API_KEY first so that every request must carry that bearer key; to leave it running, detach it and keep its log:

jevify serve recipes/<name>.yaml --host 0.0.0.0 --port 8600 > ~/jevify.log 2>&1 &
curl -s http://<this-host>:8600/health

Every answer carries Jev's fields (choice, confidence, probabilities, score, legend, noul) plus an x_jevify block naming the readout method, the cached tokens, the latency and the recipe hash that produced it.

From a clone, make ask and make serve wrap the two commands; docs/playbook.md walks through them and explains every answer field.

How it works

1. Recipe. A hashed YAML file: the endpoint, the prompt template (chat messages or a raw prompt with the model's own control tokens), the answer token spellings, and the readout policy. The only way a model is ever asked.
2. Probe. jevify probe measures the deployment: dialect (vLLM, llama.cpp, generic), context, which logprob rungs it honours, whether thinking is really off, whether an image really adds tokens, how much of a warmed prefix a question reuses. The findings are written into the recipe.
3. Warm, then ask. The state goes once; each question is a short suffix that reuses the server's prefix cache. A 32k-token state costs one warm and then a few hundred milliseconds per question.
4. Readout ladder. Named-token logprobs, grammar with post-sampling probabilities, top-k, prompt logprobs (each answer token appended to the prompt and its logprob read back, for servers that give none on the reply token), equal bias, and a floored top-k as the last resort. Auto walks down the proven rungs; the answer names the rung that read it and whether it was degraded.
5. Evidence. jevify eval runs a frozen suite with controls (shuffled states, permuted options) and writes a scorecard. jevify calibrate fits temperatures and refuses tables it cannot defend.

Model kinds

The in-process kinds need uv tool install 'jevify[encoder]'.

Results

Four frozen suites, every recipe on each, one run each, latencies per question on one machine on one day. Every row links to the scorecard that records the command, the recipe hash, the suite hash and the raw file digest.

Latency per question

The median of each suite, combined across the suites a model ran; one run each. A question is one request that reads the answer off the next-token probabilities, after the state was sent once and cached.

Image questions cost the llama.cpp models nothing extra, since the frame is paid once in the warm; DeepSeek pays about half again on images. The two in-process models are the fastest and the least accurate.

Accuracy by suite

policy-29, the diagnostic suite carried over from the author's earlier experiments (choice only):

policy-hard-52, written for jevify before any model saw it: instruction flips, negations, two-threshold policies, distractors, and long transcripts with the decisive line buried (tools/author_policy_hard.py):

Every generative model misses the same inverted instructions ("the action the customer asked NOT to take"), a weakness the author's earlier experiments found across model generations. Negations in the state and long transcripts are solved.

doom-frames-41, 164 questions over 41 ViZDoom screenshots at 640×480, every label from the engine's labels buffer or a fixed expert rule (tools/record_doom_suite.py; frames regenerate byte-identically and their hashes are pinned in the manifest):

Presence and count are perception; the expert button asks the model to agree with a rule it is not told, so that column is a floor for a scripted policy, not a skill score. Enemy side is where the vision models diverge most.

long-state-24, one decisive customer line at the start, middle or end of a filler transcript of about 2k, 8k, 16k or 32k tokens (tools/author_long_state.py). This is the suite the warm step exists for: the state is sent once, and each question then costs only its own tokens.

"context" means the launch's slot context was shorter than the state and the case recorded a context error, which is the finding for that launch (Gemma was launched with two 8k slots; Bonsai first ran with two 16k slots and failed 16k and 32k the same way, then with one 64k slot for the row above). No model that could read the state missed the decisive line at any position.

These suites rank models; they do not measure calibration, which is why jevify calibrate refused to write a table for any of them.

Play Doom

tools/play_doom.py lets a recipe play ViZDoom from screenshots: one frame per decision, one question per frame, the chosen button pressed, a video with the decision and its latency burned in. tools/record_doom_suite.py records the frozen perception suite the same way. Both need uv sync --extra games and ffmpeg.

Work with your agent

jevify is written to be operated through a coding agent. Point yours at SKILL.md: the workflow, recipe anatomy per kind, how to read x_jevify, and what a result may claim. Typical asks:

• "Write a recipe for the model behind http://host:8000/v1, probe it, and tell me which readout rung it landed on and why."
• "Run policy-hard-52 against this recipe with controls and read me the scorecard against the claims contract."
• "Serve this recipe and show me an unmodified typesafe-sdk call against it."

Repository map

To work on jevify: uv sync --group dev (add --extra encoder --extra games as needed) and uv run pytest. Live and in-process tests skip unless you opt in.

Not yet

Named so that nobody has to discover it: a direct-call control, so no "faster than generation" claim exists yet; clearing Doom's deadly_corridor (in defend_the_center Bonsai's perception plus a fixed rule scores 10 kills in 16 seconds, but in the corridor even the rule fed the engine's own labels dies before the armour, so the corridor needs a policy with memory, not better perception); the rerank kind and the embedding route against a live server; a marker-slot encoder layout; the grammar rung on the PrismML llama.cpp fork, which accepts a grammar without constraining the reported probabilities; the prompt_logprobs rung on any server but TensorFold's native one with GLM-5.3-Flash, where it has been probed and asked by hand (raw and chat recipes, text and images) but no evaluation suite has been scored on it yet (that server skips its prefix cache for such requests, so every question reads the whole state again, images included, which the probe measures and notes); Windows. Choice menus on the endpoint kind are bounded by the single-token identifier alphabet the probe verifies (26 on the DeepSeek recipe); Jev's 255-option ceiling is not reachable there.

Licence

MIT.