polyjev

Typed, calibrated decisions from any LLM.
Ask yes/no, multiple-choice, rating and extraction questions; get back typed answers with honest probabilities,
from Claude, GPT, Gemini, or open models on your own GPUs.

LLMs are great judges, classifiers and extractors, but they give you text, and when they sound sure they are often wrong. polyjev turns any model into a decision model:

• Always valid. Answers are read as probabilities over your options, so there is nothing to parse and no answer outside your schema.
• Honest numbers. Every answer has a probability distribution and a confidence. Option shuffling reduces position bias, and one-command calibration brings "90% sure" close to right 90% of the time (benchmark).
• Grounded extraction. Span answers are always substrings of the input, with character offsets. The model cannot invent a value.
• Any model. Anthropic, OpenAI, Gemini, OpenRouter and every OpenAI-compatible server (vLLM, SGLang, Ollama, llama.cpp), or a Hugging Face model loaded straight onto your GPU. One string switches between them.
• A drop-in Jev API. polyjev serve speaks POST /v1/systemone, the API of TypeSafe's Jev, so Jev clients work against your own models.

import polyjev as pj

judge = pj.Polyjev("vllm/local")   # or "anthropic/claude-opus-5", "openai/gpt-5.6-luna", "hf/Qwen/Qwen3-4B-Instruct-2507"
d = judge.decide(
    {"ticket": "Everything is down and we have a demo at noon."},
    {
        "urgent": pj.Noul("Does the customer need a reply within the hour?"),
        "team": pj.Choice("Which team owns this?", {"billing": None, "outage": "service down", "feature": None}),
        "tone": pj.Score("How angry is the customer?", ["calm", "annoyed", "furious"]),
        "order": pj.Span("the order number, if the customer gives one"),
    },
)
d["urgent"].p              # 0.999
d["team"].value            # 'outage'   (pass an Enum class and you get Enum members back)
d["team"].probabilities    # {'billing': 8e-14, 'outage': 1.0, 'feature': 2e-12}
d["tone"].level            # 'annoyed'
d["order"].found           # False: there is no order number, and polyjev will not invent one

_{Raw output of Qwen3-4B-Instruct-2507 on vLLM (one RTX 5000 Ada): about 60 ms for all four questions. Instruction-tuned models are this sure of themselves; the benchmark shows how often they should be.}

Install

pip install "polyjev[all]"     # library + server + Claude + Gemini
pip install "polyjev[hf]"      # + in-process Hugging Face models (torch)

No key or GPU? fake/demo is a deterministic stand-in model, and the Colab notebook runs a real model on a free GPU.

polyjev ask fake/demo "Invoice #A-1042, total due \$1,234.56 by 2024-03-15" \
  --noul "overdue=Is the invoice overdue?" --span "id=the invoice id" --spans "dates=every date"

What people use it for

Questions and answers

Questions can depend on each other (depends_on), be asked only when an earlier answer matches (ask_if), and take images. Extra reads (samples) average over shuffled option orders; think lets the model reason first. See how it works.

Honest probabilities: the benchmark

polyjev-bench runs 1,777 questions from six public datasets (BoolQ, RTE, MNLI, AG News, MMLU, SST-5) through 7 open models on one GPU, calibrating on one half and reporting on the other:

What we learned:

• Models are over-confident, and one fitted temperature fixes most of it.
• Reading logprobs beats asking a model for probabilities. Verbalized confidence cost Qwen3-8B 6 points of accuracy and ran ~10× slower.
• Shuffling the options cut raw calibration error by a third.
• polyjev's logprob reading matches the model's exact logits: the same answer on 99.5% of items.

Full leaderboard · charts · rerun it with bench/run.py.

Calibrate your own model on your own data:

polyjev calibrate vllm/local --data labelled.jsonl --out calibration.json   # then `calibration: calibration.json` in polyjev.yaml

Any model

strategy: auto probes each model once and falls back on its own. Adding a provider is one small class (docs). The open-model paths (vLLM, in-process Hugging Face) have run on real GPUs. The hosted-API adapters are tested offline: every request is checked against the vendor SDK's real signature, at both the newest and the oldest supported SDK version. The benchmark so far covers open models; hosted results are welcome as PRs.

Serve it

polyjev serve --model vllm/local --playground            # http://localhost:8011
curl -s localhost:8011/v1/systemone -H 'content-type: application/json' -d @examples/requests/readme_example.json

The request and response shapes, extensions and error bodies follow Jev and djev, so existing clients only change the URL. pj.Remote(url) is the typed Python client for any such server. The built-in playground has a JSON editor, image upload and webcam capture, and probability bars. Server docs.

Run open models on your own GPUs

• Workstation: docker compose --profile gpu up (deploy/) runs vLLM and polyjev together.
• HPC / SLURM:
  • setup-env.sh installs the vLLM/torch build that matches your driver.
  • polyjev.sbatch serves a model and prints the SSH tunnel command.
  • Both were tested on a university cluster, including the gotchas: CUDA 12 drivers, GPUs shared with a display, compute nodes without nvcc, and home-directory quotas.
• Inside any GPU job: provider: hf, no server.

Deployment guide, with a GPU-memory → model table.

How it compares

FAQ

Is this Jev? No. polyjev is independent and not affiliated with TypeSafe; "Jev" is their trademark. polyjev implements the publicly documented /v1/systemone request and response shapes, so it can stand in for Jev with models you choose. Jev itself is one trained model with built-in calibration.

Why not just ask the model for JSON? You get a label but not how sure the model is. The numbers a model writes about itself are less reliable than its token probabilities (see the benchmark), and a bare label cannot tell you when to escalate to a human.

Does it work with Claude? Yes. Claude has no logprobs, so polyjev asks for a probability per option through structured output ("verbalized"), and prompt caching keeps the repeated state cheap. The adapter is tested offline against Anthropic's SDK, but Claude has not been benchmarked here yet. Calibrate it on a few hundred of your own examples.

How fast is it? With an open model on vLLM, one read is a few output tokens: about 100 ms per decision, and 20–100 decisions per second on one workstation GPU (see the benchmark). Hosted APIs take their usual round trip per question, asked in parallel.

What does samples: "auto" do? It reads once, and re-reads with shuffled options only when the first answer is uncertain. Confident questions cost one call.

Project layout

src/polyjev/   library: types, validation, read strategies, engine, spans, client, backends/, server/, CLI
bench/         polyjev-bench: tasks, runner, report, results, SLURM scripts
deploy/        Dockerfile, compose (+ vLLM), SLURM recipes
docs/          documentation site (mkdocs)
examples/      runnable use cases, configs, Colab notebook
tests/         offline tests with every provider SDK faked, plus a real-model test for `hf`

Credits

polyjev follows the API of TypeSafe's Jev and the details of Matt Mastracci's djev / djev-spark (validation rules, samples: "auto", dependencies, span semantics). Its option-shift debiasing follows AnyJev's cyclic shifts.

If polyjev helps your work, a ⭐ helps others find it, and you can cite it.

Roadmap

• Benchmark results for hosted models (Claude, GPT-5.x, Gemini)
• pj.Form: decision schemas declared as classes
• Joint mode: every question in one call, for APIs that bill per request
• A TypeScript client
• Hidden-state probes for in-process models (AnyJev-style)

Contributing

Bug reports, new providers, benchmark results for more models, and docs fixes are very welcome. See CONTRIBUTING.md.

License

Apache-2.0