pyprocessors-jev

Processor based on the Jev System One API: it asks typed questions about a document and gets back calibrated probabilities instead of text, so nothing has to be parsed out of a completion.

Two providers, one protocol (POST {base_url}/v1/systemone):

What it can and cannot do

Jev chooses and rates; it does not write. That bounds the four outputs:

Because the API answers many questions in one call, classifying a document and filling its metadata costs one HTTP request — there is no --- METADATA --- section to split off.

Usage

from pymultirole_plugins.v1.schema import Document
from pyprocessors_jev.jev import JevProcessor, JevParameters

processor = JevProcessor()
parameters = JevParameters(
    labels={"billing": "Payments, invoicing, refunds", "technical": "Bugs, outages, integrations"},
    instructions="Which team should handle this?",
)

docs = processor.process([Document(text="Help! My payouts have been failing for 3 days.")], parameters)
for cat in docs[0].categories:
    print(cat.labelName, cat.score, cat.properties)

Options

Answers land in a predictable shape: a category carries the probability as its score and the question confidence in its properties; a noul in metadata writes a boolean under the question id plus its probability under <id>_probability; a choice and a score write their value plus <id>_confidence. A question whose criteria are unknown labels, or an answer naming a label outside labels, is dropped with a warning rather than invented.

The official typesafe_sdk is deliberately not a dependency: it cannot talk to an Open-Jev server, which would mean two code paths for one protocol.

Development

The build is driven by Task and uv, with the shared stages coming from the python-archetype submodule.

Getting started

The stages live in a Git submodule, so clone with --recurse-submodules:

git clone --recurse-submodules git@bitbucket.org:kairntech/pyprocessors_jev.git
cd pyprocessors_jev
sh -c "$(curl -sSL https://taskfile.dev/install.sh)" -- -d -b ~/.local/bin
task

Already cloned without it? The submodule directory is empty, and task fails on:

task: No Taskfile found at ".../submodules/python-archetype/resources/Taskfile.yml"

which means exactly that, and nothing worse:

git submodule update --init

Task is the only manual prerequisite. An archetype cannot bootstrap itself: uv and the Python interpreter install themselves on demand (every task that runs uv depends on an internal install-python task), but the thing that runs them does not. Make sure ~/.local/bin is on your PATH — that is where task and uv both land.

Running the pipeline

task stages          # print the pipeline stages, in order
task                 # run the pipeline up to (but excluding) py:publish
task -- --skip-tests # same, without the test stage
task up-to -- py:lint # run the pipeline up to and including one stage
task jenkins         # run every stage, exactly what Jenkins runs

task with no argument is safe by construction: it runs every stage but the last, and that bound is computed from the STAGES list rather than written down. The last stage is the only one with an effect outside your machine.

STAGES, declared once in Taskfile.yml, is the single definition of the pipeline order — so what you run locally is what Jenkins runs.

Individual stages

uv.lock is not versioned here, so py:sync always resolves from scratch (--upgrade): a stale lock lying around on a machine would otherwise make you test and audit versions the CI never sees.

Tests, and where the api key goes

The unit tests never open a socket: the recorder fixture replaces JevClient.system_one, so the whole suite runs without a key and without a server. They check what the processor sends (the questions built, $labels substituted, one call for every label) and how it reads back the typed answers — not that Jev answers well. That last part is the job of the single integration test.

Keys for that one live in tests/.env, which .gitignore keeps out of git (pytest-dotenv loads it, same convention as pyprocessors_openai_completion):

# tests/.env
JEV_API_KEY=sk-...
# or, for a self-hosted server:
OPENJEV_API_BASE=http://127.0.0.1:8791

A key sitting there does not make task py:test hit the network: addopts carries -m 'not integration', so the default run stays hermetic and the live test is asked for explicitly (the -m of the command line wins over addopts):

task py:test-marker -- integration

Without a key and without OPENJEV_API_BASE, that command skips instead of failing.

Measuring a real label set

tests/eval/ holds an evaluation harness for the Cairn question classifier: a frozen dev/holdout split over 574 manually labelled questions, five label-description variants with what each one scored, paired McNemar comparison, calibration and coverage curves, and the saved model outputs so the numbers can be rechecked without spending tokens. It is not part of the test suite — no file there is named test_*, so task py:test ignores it. See tests/eval/README.md.

tests/test_cairn_routing.py pins the delivery configuration of that project — twelve real questions of the corpus, the probabilities the API actually answered for them, and what the processor must make of them at threshold=0.25. It runs offline, like the rest of the suite.