pyprocessors-jev

Processor based on the Jev System One API: it asks typed questions about a document and gets back calibrated probabilities instead of text, so nothing has to be parsed out of a completion.

Two providers, one protocol (POST {base_url}/v1/systemone):

entry point	provider	base url	model	api key
jev	hosted TypeSafe	JEV_API_BASE, else TYPESAFE_BASE_URL, else https://api.typesafe.ai	JEV_MODEL, else jev-latest	JEV_API_KEY, else TYPESAFE_API_KEY
openjev	self-hosted Open-Jev	OPENJEV_API_BASE, else http://127.0.0.1:8791	OPENJEV_MODEL, else open-jev	OPENJEV_API_KEY (usually none)

What it does, in v1

The v1 does one thing: add_categories — one choice question over the project labels, whose probabilities become the document categories. One document, one question, one HTTP request.

The API allows more, and the processor already carries it, commented out in src/pyprocessors_jev/jev.py (and in tests/test_jev.py, and in the table below): free-form questions on any target, one noul question per label for a multilabel decision, filtering existing annotations, picking the best alternative text. They come back by uncommenting them — the v1 keeps one option to explain and one behaviour to support.

Jev chooses and rates; it does not write. That bounds what any of it will ever produce:

output	how	not possible
categories	choice over the project labels — the v1	—
metadata (v1+)	typed values only: noul → boolean, choice → key of a closed set, score → number	free-form extraction (dates, amounts, names) — use pyprocessors_openai_completion
annotations (v1+)	filter existing candidate spans	create spans: no offsets come back
altTexts (v1+)	select among the texts already on the document (rerank / judge)	generate a summary or a translation

Because the API answers many questions in one call, a document that will one day be classified and have its metadata filled still costs one HTTP request — there is no --- METADATA --- section to split off.

Usage

from pymultirole_plugins.v1.schema import Document
from pyprocessors_jev.jev import JevProcessor, JevParameters

processor = JevProcessor()
parameters = JevParameters(
    labels={"billing": "Payments, invoicing, refunds", "technical": "Bugs, outages, integrations"},
    instructions="Which team should handle this?",
)

docs = processor.process([Document(text="Help! My payouts have been failing for 3 days.")], parameters)
for cat in docs[0].categories:
    print(cat.labelName, cat.score, cat.properties)

Options

Option	Default	Description
base_url	provider default (see above)	Jev endpoint base url
model	provider default (see above)	model route
function	add_categories	the question built from labels. One value in v1: add_categories, a single choice over the labels. (v1+: add_multilabel_categories — one noul per label, same call; filter_annotations — one noul per annotation, offsets preserved; select_altText — choice over the alternative texts.)
instructions	Choose the best category for this text.	instructions of the question built from function
labels	–	label name → description mapping, injected from the project label set. The description is what Jev reads to decide; the categories written on the document carry labelName only, never label — a description has no place on the document, and the project label set already holds the display name.
state_altText	–	which text Jev reads to answer — what the API calls the state. Empty, it is the text of the document. Set to the name of an alternative text, it is that text: how a document is classified on what an upstream processor produced (a cleaning, a translation, a summary, the segment a retrieval step kept, a rendering of the metadata) without copying anything or running a second pipeline. A document that has no alternative text of that name is classified on its own text, with a warning, rather than skipped. max_chars truncates whichever is sent.
decision_altText	–	keep questions, answers and token usage in that alternative text — the audit trail
multi_label_threshold	0.5	the bar a probability has to clear to become a category, and with it how many labels a document comes back with. At 0.5 the decision is single-label — the probabilities of a choice are exclusive and sum to 1, so at most one clears the bar. Lower it and the same question becomes multilabel: every label above the bar becomes a category, sorted by decreasing probability
keep_best	false	when no answer of a choice reaches multi_label_threshold, keep the most probable one anyway, so the document never comes back without a category. A no-op with 3 labels (the winner is mechanically above 1/3), a safety net with a large label set
max_chars	0	truncate the state, 0 sends it whole
timeout	60.0	HTTP timeout, in seconds
max_retries	3	retries of a throttled (429) or overloaded (529) call, exponential backoff, obeying Retry-After. A timeout or a dropped connection — the call that never reached a status code — is retried the same way
concurrency	1	how many documents are sent at the same time. One document is one call whatever this is set to: it changes the wall clock, never what the model reads, so the decision trace and the blast radius of a failed call stay per document. A call is almost pure network wait, so N at a time divides the duration by about N until the API throttles, which max_retries absorbs. Measured on 40 questions of the Cairn corpus: 298 ms per document sequentially, 76 ms at 4, 44 ms at 8, 25 ms at 16

A document whose call fails is logged and left untouched — it comes back without a category rather than taking the rest of the batch down with it, which is what process used to do.

Answers land in a predictable shape: a category carries the probability as its score and the question confidence in its properties. (v1+: a noul in metadata writes a boolean under the question id plus its probability under <id>_probability; a choice and a score write their value plus <id>_confidence.) An answer naming a choice outside the criteria that were sent is dropped with a warning rather than written as a label the project does not have.

The official typesafe_sdk is deliberately not a dependency: it cannot talk to an Open-Jev server, which would mean two code paths for one protocol.

Development

The build is driven by Task and uv, with the shared stages coming from the python-archetype submodule.

Getting started

The stages live in a Git submodule, so clone with --recurse-submodules:

git clone --recurse-submodules git@bitbucket.org:kairntech/pyprocessors_jev.git
cd pyprocessors_jev
sh -c "$(curl -sSL https://taskfile.dev/install.sh)" -- -d -b ~/.local/bin
task

Already cloned without it? The submodule directory is empty, and task fails on:

task: No Taskfile found at ".../submodules/python-archetype/resources/Taskfile.yml"

which means exactly that, and nothing worse:

git submodule update --init

Task is the only manual prerequisite. An archetype cannot bootstrap itself: uv and the Python interpreter install themselves on demand (every task that runs uv depends on an internal install-python task), but the thing that runs them does not. Make sure ~/.local/bin is on your PATH — that is where task and uv both land.

Running the pipeline

task stages          # print the pipeline stages, in order
task                 # run the pipeline up to (but excluding) py:publish
task -- --skip-tests # same, without the test stage
task up-to -- py:lint # run the pipeline up to and including one stage
task jenkins         # run every stage, exactly what Jenkins runs

task with no argument is safe by construction: it runs every stage but the last, and that bound is computed from the STAGES list rather than written down. The last stage is the only one with an effect outside your machine.

STAGES, declared once in Taskfile.yml, is the single definition of the pipeline order — so what you run locally is what Jenkins runs.

Individual stages

Task	Description
task py:sync	Install the project and its dependencies (uv sync)
task py:lint	ruff check and ruff format --check
task py:format	Reformat the code with ruff
task py:test	Run the test suite
task py:test-marker -- <m>	Run the tests carrying one pytest marker
task py:sbom	Generate a CycloneDX SBOM of the resolved environment
task py:check-vulnerabilities	Check for known CVEs
task py:check-updates	Check for dependency updates
task py:build	Build the wheel and sdist (uv build)
task py:publish	Publish the distributions (uv publish)
task py:version-file	Print the path of the file carrying __version__
task py:set-version VERSION=x	Write that version into it

uv.lock is not versioned here, so py:sync always resolves from scratch (--upgrade): a stale lock lying around on a machine would otherwise make you test and audit versions the CI never sees.

Tests, and where the api key goes

The unit tests never open a socket: the recorder fixture replaces JevClient.system_one, so the whole suite runs without a key and without a server. They check what the processor sends (the questions built, $labels substituted, one call for every label) and how it reads back the typed answers — not that Jev answers well. That last part is the job of the single integration test.

Keys for that one live in tests/.env, which .gitignore keeps out of git (pytest-dotenv loads it, same convention as pyprocessors_openai_completion):

# tests/.env
JEV_API_KEY=sk-...
# or, for a self-hosted server:
OPENJEV_API_BASE=http://127.0.0.1:8791

A key sitting there does not make task py:test hit the network: addopts carries -m 'not integration', so the default run stays hermetic and the live test is asked for explicitly (the -m of the command line wins over addopts):

task py:test-marker -- integration

Without a key and without OPENJEV_API_BASE, that command skips instead of failing.

Measuring a real label set

tests/eval/ holds an evaluation harness for the Cairn question classifier: a frozen dev/holdout split over 574 manually labelled questions, five label-description variants with what each one scored, paired McNemar comparison, calibration and coverage curves, and the saved model outputs so the numbers can be rechecked without spending tokens. It is not part of the test suite — no file there is named test_*, so task py:test ignores it. See tests/eval/README.md.

tests/test_cairn_routing.py pins the delivery configuration of that project — twelve real questions of the corpus, the probabilities the API actually answered for them, and what the processor must make of them at multi_label_threshold=0.25. It runs offline, like the rest of the suite.