jevify

Make any model you already have answer typed questions in one pass, the way Jev does, and measure how well it does.

You bring a model behind an OpenAI-compatible endpoint (vLLM, llama.cpp, anything with logprobs) or a small model that runs in-process. jevify sends it a state once, then asks typed questions about it: a choice among options, a score on ordered levels, a yes/no with a probability. Each answer is read straight off the model's next-token distribution, no generation, no parsing, one request per question. The result comes back in the exact shape of TypeSafe's Jev API, so the unmodified typesafe-sdk works against it.

jevify owns no weights and trains nothing. A recipe says how a model is asked, a probe proves what the deployment can do, and a scorecard says how it did, with the command, hashes and raw file that produced every number.

   state (text, images)          one warm request, cached prefix
        │
        ├── "Which team handles this?"  ──►  {billing: 0.94, support: 0.05, sales: 0.01}
        ├── "How urgent?"               ──►  {low: 0.05, medium: 0.10, high: 0.85}
        └── "Threatens to cancel."      ──►  p(true) = 0.95
                                              each ≈ 30–140 ms on a local GPU

Quickstart

uv tool install jevify                # or: pipx install jevify
git clone https://github.com/fidecastro/jevify && cd jevify

# 1. point a recipe at your server (copy the nearest one under recipes/)
jevify probe recipes/gemma-4-e4b-it.llamacpp.yaml

# 2. ask
jevify ask recipes/gemma-4-e4b-it.llamacpp.yaml \
  --state "Customer: I was charged twice. Fix it today or I cancel." \
  --choice "dept:Which team should handle this?=billing,support,sales" \
  --score  "urgency:How urgent is this?=low,medium,high" \
  --noul   "cancel:The customer threatens to cancel." \
  [--image screenshot.png]

# 3. serve Jev's API and use the stock SDK
jevify serve recipes/gemma-4-e4b-it.llamacpp.yaml --port 8600

Every answer carries Jev's fields (choice, confidence, probabilities, score, legend, noul) plus an x_jevify block naming the readout method, the cached tokens, the latency and the recipe hash that produced it.

How it works

1. Recipe. A hashed YAML file: the endpoint, the prompt template (chat messages or a raw prompt with the model's own control tokens), the answer token spellings, and the readout policy. The only way a model is ever asked.
2. Probe. jevify probe measures the deployment: dialect (vLLM, llama.cpp, generic), context, which logprob rungs it honours, whether thinking is really off, whether an image really adds tokens, how much of a warmed prefix a question reuses. The findings are written into the recipe.
3. Warm, then ask. The state goes once; each question is a short suffix that reuses the server's prefix cache. A 32k-token state costs one warm and then a few hundred milliseconds per question.
4. Readout ladder. Named-token logprobs, grammar with post-sampling probabilities, top-k, equal bias, and a floored top-k as the last resort. Auto walks down the proven rungs; the answer names the rung that read it and whether it was degraded.
5. Evidence. jevify eval runs a frozen suite with controls (shuffled states, permuted options) and writes a scorecard. jevify calibrate fits temperatures and refuses tables it cannot defend.

Model kinds

The in-process kinds need uv tool install 'jevify[encoder]'.

Results

Four frozen suites, every recipe on each, one run each, latencies per question on one machine on one day. Every row links to the scorecard that records the command, the recipe hash, the suite hash and the raw file digest.

policy-29, the diagnostic suite carried over from the author's earlier experiments (choice only):

policy-hard-52, written for jevify before any model saw it: instruction flips, negations, two-threshold policies, distractors, and long transcripts with the decisive line buried (tools/author_policy_hard.py):

Every generative model misses the same inverted instructions ("the action the customer asked NOT to take"), a weakness the author's earlier experiments found across model generations. Negations in the state and long transcripts are solved.

doom-frames-41, 164 questions over 41 ViZDoom screenshots at 640×480, every label from the engine's labels buffer or a fixed expert rule (tools/record_doom_suite.py; frames regenerate byte-identically and their hashes are pinned in the manifest):

Presence and count are perception; the expert button asks the model to agree with a rule it is not told, so that column is a floor for a scripted policy, not a skill score. Enemy side is where the vision models diverge most.

long-state-24, one decisive customer line at the start, middle or end of a filler transcript of about 2k, 8k, 16k or 32k tokens (tools/author_long_state.py). This is the suite the warm step exists for: the state is sent once, and each question then costs only its own tokens.

"context" means the launch's slot context was shorter than the state and the case recorded a context error, which is the finding for that launch (Gemma was launched with two 8k slots; Bonsai first ran with two 16k slots and failed 16k and 32k the same way, then with one 64k slot for the row above). No model that could read the state missed the decisive line at any position.

These suites rank models; they do not measure calibration, which is why jevify calibrate refused to write a table for any of them.

Play Doom

tools/play_doom.py lets a recipe play ViZDoom from screenshots: one frame per decision, one question per frame, the chosen button pressed, a video with the decision and its latency burned in. tools/record_doom_suite.py records the frozen perception suite the same way. Both need uv sync --extra games and ffmpeg.

Work with your agent

jevify is written to be operated through a coding agent. Point yours at SKILL.md: the workflow, recipe anatomy per kind, how to read x_jevify, and what a result may claim. Typical asks:

• "Write a recipe for the model behind http://host:8000/v1, probe it, and tell me which readout rung it landed on and why."
• "Run policy-hard-52 against this recipe with controls and read me the scorecard against the claims contract."
• "Serve this recipe and show me an unmodified typesafe-sdk call against it."

Repository map

To work on jevify: uv sync --group dev (add --extra encoder --extra games as needed) and uv run pytest. Live and in-process tests skip unless you opt in.

Not yet

Named so that nobody has to discover it: a direct-call control, so no "faster than generation" claim exists yet; a Doom player that survives (both models die within seconds of deadly_corridor); the rerank kind and the embedding route against a live server; a marker-slot encoder layout; the grammar rung on the PrismML llama.cpp fork, which accepts a grammar without constraining the reported probabilities; Windows. Choice menus on the endpoint kind are bounded by the single-token identifier alphabet the probe verifies (26 on the DeepSeek recipe); Jev's 255-option ceiling is not reachable there.

Licence

MIT.