pyprocessors-jev

Processor based on the Jev System One API: it asks typed questions about a document and gets back calibrated probabilities instead of text, so nothing has to be parsed out of a completion.

Three providers of the same questions and answers, and a reranker for the linking:

jev and openjev speak the System One protocol (POST {base_url}/v1/systemone). cygnet needs no System One server: src/pyprocessors_jev/readout.py turns each question into one chat request whose options are lettered A., B., ..., asks for one token constrained to those letters (logprobs and vLLM's structured_outputs, which the endpoint must pass through), and reads the probability of each letter, with a calibration temperature (3.4 for Gemma). The questions of a document go out together (parallel), so the server batches them and computes their shared text once. The prompt and the readout are those of the shim once run in front of vLLM (tests/eval/cygnet/), which tests/test_readout.py checks, so the figures measured through it hold.

rerank (src/pyprocessors_jev/rerank.py) serves link_annotations only, with Qwen3-Reranker-0.6B fine-tuned on Jev's AFP decisions (87.5 % F1 on MSNBC and AQUAINT, as recalibrated Jev) behind a /rerank endpoint: RERANK_API_BASE, else LITELLM_OPENAI_API_BASE, else OPENAI_API_BASE; model RERANK_MODEL, else qwen3-reranker-afp-jev; key RERANK_API_KEY, else LITELLM_OPENAI_API_KEY, else OPENAI_API_KEY. Each group of mentions is one call: the context of its first mention (600 characters each side, the mention between [ ]) against its candidates ("name. definition"), the input of the training. The log-odds are turned into a distribution with a temperature, the entity-fishing prior and a NIL logit (temperature, prior_weight, nil_logit: 3, 0.5, -3, fitted on MSNBC and AQUAINT), then written as for the other deciders. vLLM serves the merged model (tools/merge_rerank.py of pyannotators_entityfishing) with the score template deploy/vllm/qwen3_reranker_linking.jinja, which renders the training prompt exactly; the command is in that script.

What it does, in v1

The v1 does two things, still one HTTP request per document:

• add_categories — one choice question over the project labels, whose probabilities become the document categories;
• link_annotations — the decision step of entity linking (ADR-0001 of pyannotators_entityfishing): one choice per group of mentions over the candidates an upstream step wrote in their terms (the entityfishing_candidates processor), plus "none of them". Only the chosen candidates stay in terms.

The API allows more, and the processor already carries it, commented out in src/pyprocessors_jev/jev.py (and in tests/test_jev.py, and in the table below): free-form questions on any target, one noul question per label for a multilabel decision, filtering existing annotations, picking the best alternative text. They come back by uncommenting them — the v1 keeps few options to explain and behaviours to support.

Jev chooses and rates; it does not write. That bounds what any of it will ever produce:

Because the API answers many questions in one call, a document that will one day be classified and have its metadata filled still costs one HTTP request — there is no --- METADATA --- section to split off.

Usage

from pymultirole_plugins.v1.schema import Document
from pyprocessors_jev.jev import JevDecisionProcessor, JevDecisionParameters

processor = JevDecisionProcessor()
parameters = JevDecisionParameters(
    labels={"billing": "Payments, invoicing, refunds", "technical": "Bugs, outages, integrations"},
    instructions="Which team should handle this?",
)

docs = processor.process([Document(text="Help! My payouts have been failing for 3 days.")], parameters)
for cat in docs[0].categories:
    print(cat.labelName, cat.score, cat.properties)

Linking: link_annotations

The mentions come from a NER, their candidates from the entityfishing_candidates processor of pyannotators_entityfishing (wikidata terms, the prior as score, a definition and facts in their properties). The mentions of the same entity — same properties.entity_group, written by pyprocessors_coreference, else same label and surface — are one question: the union of their candidates, described by name, definition and facts, plus NONE. The prior is never shown to Jev.

For each group, every mention keeps the candidates whose probability clears multi_label_threshold, most probable first: one link at the default 0.5, the ambiguity kept below it. keep_best keeps the most probable candidate when none clears the bar, unless Jev answered NONE. Terms of other lexicons than wikidata (a referential matched upstream) are not candidates and stay. With no candidate left, only those remain and the annotation — a detection of the NER — stays. A kept term has the Jev probability as score, confidence in its properties, the prior still in prob_c, and loses its definition and facts unless keep_descriptions.

parameters = JevDecisionParameters(function="link_annotations")
docs = processor.process(documents, parameters)  # documents: the output of entityfishing_candidates

Measured on MSNBC and AQUAINT (1,383 linked mentions): F1 86.2 % against 68.0 % for entity-fishing's /disambiguate, about 10,600 input tokens and 0.6 s of Jev per document (ADR-0001, §2 and §7).

Options

A document whose call fails comes back with its categories and terms as they went in, rather than taking the rest of the batch down with it, which is what process used to do. It says why: the error (JevError: Jev returned 401 for …) is written in its decision_error metadata, and in the decision_altText trace under error when one is set — otherwise an undecided document looks exactly like one that was never sent, and the cause only shows in the logs of the plugin server. A document decided later drops that metadata. When every document of a batch fails, process raises instead: that is a key, an url or a model, not a document, and Sherpa shows the error. The same holds for rerank_decision.

Answers land in a predictable shape: a category carries the probability as its score and the question confidence in its properties. (v1+: a noul in metadata writes a boolean under the question id plus its probability under <id>_probability; a choice and a score write their value plus <id>_confidence.) An answer naming a choice outside the criteria that were sent is dropped with a warning rather than written as a label the project does not have.

The official typesafe_sdk is deliberately not a dependency: it cannot talk to an Open-Jev server, which would mean two code paths for one protocol.

Development

The build is driven by Task and uv, with the shared stages coming from the python-archetype submodule.

Getting started

The stages live in a Git submodule, so clone with --recurse-submodules:

git clone --recurse-submodules git@bitbucket.org:kairntech/pyprocessors_jev.git
cd pyprocessors_jev
sh -c "$(curl -sSL https://taskfile.dev/install.sh)" -- -d -b ~/.local/bin
task

Already cloned without it? The submodule directory is empty, and task fails on:

task: No Taskfile found at ".../submodules/python-archetype/resources/Taskfile.yml"

which means exactly that, and nothing worse:

git submodule update --init

Task is the only manual prerequisite. An archetype cannot bootstrap itself: uv and the Python interpreter install themselves on demand (every task that runs uv depends on an internal install-python task), but the thing that runs them does not. Make sure ~/.local/bin is on your PATH — that is where task and uv both land.

Running the pipeline

task stages          # print the pipeline stages, in order
task                 # run the pipeline up to (but excluding) py:publish
task -- --skip-tests # same, without the test stage
task up-to -- py:lint # run the pipeline up to and including one stage
task jenkins         # run every stage, exactly what Jenkins runs

task with no argument is safe by construction: it runs every stage but the last, and that bound is computed from the STAGES list rather than written down. The last stage is the only one with an effect outside your machine.

STAGES, declared once in Taskfile.yml, is the single definition of the pipeline order — so what you run locally is what Jenkins runs.

Individual stages

uv.lock is not versioned here, so py:sync always resolves from scratch (--upgrade): a stale lock lying around on a machine would otherwise make you test and audit versions the CI never sees.

Tests, and where the api key goes

The unit tests never open a socket: the recorder fixture replaces JevClient.system_one, so the whole suite runs without a key and without a server. They check what the processor sends (the questions built, $labels substituted, one call for every label) and how it reads back the typed answers — not that Jev answers well. That last part is the job of the single integration test.

Keys for that one live in tests/.env, which .gitignore keeps out of git (pytest-dotenv loads it, same convention as pyprocessors_openai_completion):

# tests/.env
JEV_API_KEY=sk-...
# or, for a self-hosted server:
OPENJEV_API_BASE=http://127.0.0.1:8791

A key sitting there does not make task py:test hit the network: addopts carries -m 'not integration', so the default run stays hermetic and the live test is asked for explicitly (the -m of the command line wins over addopts):

task py:test-marker -- integration

Without a key and without OPENJEV_API_BASE, that command skips instead of failing.

Measuring a real label set

tests/eval/ holds an evaluation harness for the Cairn question classifier: a frozen dev/holdout split over 574 manually labelled questions, five label-description variants with what each one scored, paired McNemar comparison, calibration and coverage curves, and the saved model outputs so the numbers can be rechecked without spending tokens. It is not part of the test suite — no file there is named test_*, so task py:test ignores it. See tests/eval/README.md.

tests/test_cairn_routing.py pins the delivery configuration of that project — twelve real questions of the corpus, the probabilities the API actually answered for them, and what the processor must make of them at multi_label_threshold=0.25. It runs offline, like the rest of the suite.