pyprocessors-jev

Processor based on the Jev System One API: it asks typed questions about a document and gets back calibrated probabilities instead of text, so nothing has to be parsed out of a completion.

Two providers, one protocol (POST {base_url}/v1/systemone):

entry point	provider	base url	model	api key
jev	hosted TypeSafe	JEV_API_BASE, else TYPESAFE_BASE_URL, else https://api.typesafe.ai	JEV_MODEL, else jev-latest	JEV_API_KEY, else TYPESAFE_API_KEY
openjev	self-hosted Open-Jev	OPENJEV_API_BASE, else http://127.0.0.1:8791	OPENJEV_MODEL, else open-jev	OPENJEV_API_KEY (usually none)

What it does, in v1

The v1 does two things, still one HTTP request per document:

• add_categories — one choice question over the project labels, whose probabilities become the document categories;
• link_annotations — the decision step of entity linking (ADR-0001 of pyannotators_entityfishing): one choice per group of mentions over the candidates an upstream step wrote in their terms (the entityfishing_candidates processor), plus "none of them". Only the chosen candidates stay in terms.

The API allows more, and the processor already carries it, commented out in src/pyprocessors_jev/jev.py (and in tests/test_jev.py, and in the table below): free-form questions on any target, one noul question per label for a multilabel decision, filtering existing annotations, picking the best alternative text. They come back by uncommenting them — the v1 keeps few options to explain and behaviours to support.

Jev chooses and rates; it does not write. That bounds what any of it will ever produce:

output	how	not possible
categories	choice over the project labels — the v1	—
metadata (v1+)	typed values only: noul → boolean, choice → key of a closed set, score → number	free-form extraction (dates, amounts, names) — use pyprocessors_openai_completion
annotations.terms	link_annotations: choose among the candidates already in the terms of the mentions	find candidates: they come from the upstream step
annotations (v1+)	filter existing candidate spans	create spans: no offsets come back
altTexts (v1+)	select among the texts already on the document (rerank / judge)	generate a summary or a translation

Because the API answers many questions in one call, a document that will one day be classified and have its metadata filled still costs one HTTP request — there is no --- METADATA --- section to split off.

Usage

from pymultirole_plugins.v1.schema import Document
from pyprocessors_jev.jev import JevProcessor, JevParameters

processor = JevProcessor()
parameters = JevParameters(
    labels={"billing": "Payments, invoicing, refunds", "technical": "Bugs, outages, integrations"},
    instructions="Which team should handle this?",
)

docs = processor.process([Document(text="Help! My payouts have been failing for 3 days.")], parameters)
for cat in docs[0].categories:
    print(cat.labelName, cat.score, cat.properties)

Linking: link_annotations

The mentions come from a NER, their candidates from the entityfishing_candidates processor of pyannotators_entityfishing (wikidata terms, the prior as score, a definition and facts in their properties). The mentions of the same entity — same properties.entity_group, written by pyprocessors_coreference, else same label and surface — are one question: the union of their candidates, described by name, definition and facts, plus NONE. The prior is never shown to Jev.

For each group, every mention keeps the candidates whose probability clears multi_label_threshold, most probable first: one link at the default 0.5, the ambiguity kept below it. keep_best keeps the most probable candidate when none clears the bar, unless Jev answered NONE. With nothing left, terms is emptied and the annotation — a detection of the NER — stays. A kept term has the Jev probability as score, confidence in its properties, the prior still in prob_c, and loses its definition and facts unless keep_descriptions.

parameters = JevParameters(function="link_annotations")
docs = processor.process(documents, parameters)  # documents: the output of entityfishing_candidates

Measured on MSNBC and AQUAINT (1,383 linked mentions): F1 86.2 % against 68.0 % for entity-fishing's /disambiguate, about 10,600 input tokens and 0.6 s of Jev per document (ADR-0001, §2 and §7).

Options

Option	Default	Description
base_url	provider default (see above)	Jev endpoint base url
model	provider default (see above)	model route
function	add_categories	the question to ask. add_categories: a single choice over labels. link_annotations: one choice per group of mentions over the candidates in their terms (see Linking); labels and instructions are not used. (v1+: add_multilabel_categories — one noul per label, same call; filter_annotations — one noul per annotation, offsets preserved; select_altText — choice over the alternative texts.)
instructions	Choose the best category for this text.	instructions of the add_categories question
labels	–	label name → description mapping, injected from the project label set. The description is what Jev reads to decide; the categories written on the document carry labelName only, never label — a description has no place on the document, and the project label set already holds the display name.
state_altText	–	which text Jev reads to answer — what the API calls the state. Empty, it is the text of the document. Set to the name of an alternative text, it is that text: how a document is classified on what an upstream processor produced (a cleaning, a translation, a summary, the segment a retrieval step kept, a rendering of the metadata) without copying anything or running a second pipeline. A document that has no alternative text of that name is classified on its own text, with a warning, rather than skipped. max_chars truncates whichever is sent.
decision_altText	–	keep questions, answers and token usage in that alternative text — the audit trail
multi_label_threshold	0.5	the bar a probability has to clear to become a category, and with it how many labels a document comes back with. At 0.5 the decision is single-label — the probabilities of a choice are exclusive and sum to 1, so at most one clears the bar. Lower it and the same question becomes multilabel: every label above the bar becomes a category, sorted by decreasing probability
keep_best	false	when no answer of a choice reaches multi_label_threshold, keep the most probable one anyway, so the document never comes back without a category. A no-op with 3 labels (the winner is mechanically above 1/3), a safety net with a large label set
keep_descriptions	false	link_annotations only: keep the definition and facts of the kept candidates — by default they are dropped, being there for Jev to decide
max_chars	0	truncate the state, 0 sends it whole
timeout	60.0	HTTP timeout, in seconds
max_retries	3	retries of a throttled (429) or overloaded (529) call, exponential backoff, obeying Retry-After. A timeout or a dropped connection — the call that never reached a status code — is retried the same way
concurrency	1	how many documents are sent at the same time. One document is one call whatever this is set to: it changes the wall clock, never what the model reads, so the decision trace and the blast radius of a failed call stay per document. A call is almost pure network wait, so N at a time divides the duration by about N until the API throttles, which max_retries absorbs. Measured on 40 questions of the Cairn corpus: 298 ms per document sequentially, 76 ms at 4, 44 ms at 8, 25 ms at 16

A document whose call fails is logged and left untouched — it comes back without a category rather than taking the rest of the batch down with it, which is what process used to do.

Answers land in a predictable shape: a category carries the probability as its score and the question confidence in its properties. (v1+: a noul in metadata writes a boolean under the question id plus its probability under <id>_probability; a choice and a score write their value plus <id>_confidence.) An answer naming a choice outside the criteria that were sent is dropped with a warning rather than written as a label the project does not have.

The official typesafe_sdk is deliberately not a dependency: it cannot talk to an Open-Jev server, which would mean two code paths for one protocol.

Development

The build is driven by Task and uv, with the shared stages coming from the python-archetype submodule.

Getting started

The stages live in a Git submodule, so clone with --recurse-submodules:

git clone --recurse-submodules git@bitbucket.org:kairntech/pyprocessors_jev.git
cd pyprocessors_jev
sh -c "$(curl -sSL https://taskfile.dev/install.sh)" -- -d -b ~/.local/bin
task

Already cloned without it? The submodule directory is empty, and task fails on:

task: No Taskfile found at ".../submodules/python-archetype/resources/Taskfile.yml"

which means exactly that, and nothing worse:

git submodule update --init

Task is the only manual prerequisite. An archetype cannot bootstrap itself: uv and the Python interpreter install themselves on demand (every task that runs uv depends on an internal install-python task), but the thing that runs them does not. Make sure ~/.local/bin is on your PATH — that is where task and uv both land.

Running the pipeline

task stages          # print the pipeline stages, in order
task                 # run the pipeline up to (but excluding) py:publish
task -- --skip-tests # same, without the test stage
task up-to -- py:lint # run the pipeline up to and including one stage
task jenkins         # run every stage, exactly what Jenkins runs

task with no argument is safe by construction: it runs every stage but the last, and that bound is computed from the STAGES list rather than written down. The last stage is the only one with an effect outside your machine.

STAGES, declared once in Taskfile.yml, is the single definition of the pipeline order — so what you run locally is what Jenkins runs.

Individual stages

Task	Description
task py:sync	Install the project and its dependencies (uv sync)
task py:lint	ruff check and ruff format --check
task py:format	Reformat the code with ruff
task py:test	Run the test suite
task py:test-marker -- <m>	Run the tests carrying one pytest marker
task py:sbom	Generate a CycloneDX SBOM of the resolved environment
task py:check-vulnerabilities	Check for known CVEs
task py:check-updates	Check for dependency updates
task py:build	Build the wheel and sdist (uv build)
task py:publish	Publish the distributions (uv publish)
task py:version-file	Print the path of the file carrying __version__
task py:set-version VERSION=x	Write that version into it

uv.lock is not versioned here, so py:sync always resolves from scratch (--upgrade): a stale lock lying around on a machine would otherwise make you test and audit versions the CI never sees.

Tests, and where the api key goes

The unit tests never open a socket: the recorder fixture replaces JevClient.system_one, so the whole suite runs without a key and without a server. They check what the processor sends (the questions built, $labels substituted, one call for every label) and how it reads back the typed answers — not that Jev answers well. That last part is the job of the single integration test.

Keys for that one live in tests/.env, which .gitignore keeps out of git (pytest-dotenv loads it, same convention as pyprocessors_openai_completion):

# tests/.env
JEV_API_KEY=sk-...
# or, for a self-hosted server:
OPENJEV_API_BASE=http://127.0.0.1:8791

A key sitting there does not make task py:test hit the network: addopts carries -m 'not integration', so the default run stays hermetic and the live test is asked for explicitly (the -m of the command line wins over addopts):

task py:test-marker -- integration

Without a key and without OPENJEV_API_BASE, that command skips instead of failing.

Measuring a real label set

tests/eval/ holds an evaluation harness for the Cairn question classifier: a frozen dev/holdout split over 574 manually labelled questions, five label-description variants with what each one scored, paired McNemar comparison, calibration and coverage curves, and the saved model outputs so the numbers can be rechecked without spending tokens. It is not part of the test suite — no file there is named test_*, so task py:test ignores it. See tests/eval/README.md.

tests/test_cairn_routing.py pins the delivery configuration of that project — twelve real questions of the corpus, the probabilities the API actually answered for them, and what the processor must make of them at multi_label_threshold=0.25. It runs offline, like the rest of the suite.