LLM2Jev — Less generation. Faster decisions.

Classify. Filter. Score. Route. Jev-style decisions from local language models.

An email pipeline often needs a category, a document pipeline needs a relevance decision, and an agent router needs a destination. Generating an explanation and a JSON object can add work that these applications never use. LLM2Jev tests a narrower execution path: define the possible answers, read binary model scores, and construct the result in Python.

The goal is to reduce decoding and parsing overhead in repeated semantic decisions. Local measurements show a modest speedup in one configured workload; quality and application-level savings remain workload-dependent. Candidate scoring repeats input processing, so the approach can also be slower than generating a short label. See the measured results below, including incorrect decisions.

The idea from Jev

LLM2Jev’s public API design is inspired by TypeSafe’s Jev. The state plus typed questions interface and the Choice, Score, and Noul vocabulary follow TypeSafe’s published API primitives. Credit for these interface concepts belongs to TypeSafe; they are not new primitives introduced by LLM2Jev.

TypeSafe's Jev makes typed decisions over supplied state. It motivates a useful application question: how much work can software avoid when it needs a bounded judgment rather than generated text?

LLM2Jev explores that question using existing language models. It implements Jev-style Choice, Score, and Noul responses through next-token binary scoring. It does not reproduce Jev's architecture, training, calibrated probabilities, or parallel sampler, and external API conformance has not been established. This is an independent project.

TypeSafe publishes large speed and cost improvements for its own workloads and service. Those measurements do not transfer to this adapter. Here, Transformers performs a forward pass with zero generated tokens; Ollama requests one generated token per candidate to obtain logprobs and reports actual usage. Neither path parses generated text into an answer.

Concrete workloads

The strongest starting hypothesis is short text, a small fixed answer space, repeated decisions, and an application that consumes the result directly.

These are integration patterns; the refund predicate has a small measured smoke experiment below, while the other workloads remain unmeasured. There is no bundled reproduction of a 500-email speed benchmark or a 7.1-second flight-search agent. For routing, describe operational task requirements in the criteria; model names alone do not teach the scorer which downstream model will succeed.

In a data pipeline, Noul can supply a semantic filter, Choice a bounded semantic map, and Score a rubric-based ranking signal. Applications retain record IDs, apply thresholds, sort, group, and count. LLM2Jev currently provides the per-record decision primitive; it has no dataframe/SQL integration, relational optimizer, or dataset-level operator API. Comparing document pairs is possible by supplying both in state, but a naive semantic join still needs one evaluation per pair.

Install from PyPI

Version 0.4.1 uses llms2jev for both the distribution and Python imports. Commands are llms2jev, llms2jev-serve, and llms2jev-mcp; environment variables use the LLMS2JEV_ prefix. Update imports, shell commands, and MCP configurations when upgrading.

python -m pip install 'llms2jev[server,mcp]==0.4.1'

LLM2Jev is available on PyPI. Install the core Python package directly:

python -m pip install llms2jev

To install the published version used by this README with the dependencies for your runtime:

Wheel and source archives are also available from the PyPI release files.

Try a decision

The core package has no mandatory third-party runtime dependencies. Install the Transformers extra for local inference; model weights are supplied separately. See the runtime compatibility details when choosing an interpreter and model family:

pip install 'llms2jev[transformers]==0.4.1'

Supply downloaded model weights; the runtime loads local files only. On Apple Silicon, select device="mps" explicitly; the example below uses CPU for portability. The measured refund example includes the pinned model download, explicit encoder, and Yes/No label configuration used in the benchmark.

From a source checkout, start with the deterministic, model-free example:

uv sync --group dev
uv run python examples/bound_evaluation.py

That example checks the application path with synthetic scores. For source development, install uv sync --extra transformers --group dev. The following is a general API walkthrough with a compatible local model, not a validated model/encoder configuration:

from llms2jev import Choice, LLM2Jev, Noul, TransformersRuntime

with TransformersRuntime("/path/to/local/model", device="cpu") as runtime:
    triage = LLM2Jev(runtime=runtime, model_identity=runtime.identity).bind(
        model=runtime.identity.name,
        questions={
            "department": Choice(
                instructions="Which queue should handle this customer message?",
                criteria={
                    "billing": "Payments, invoices, or refunds",
                    "delivery": "Shipping, tracking, or missing packages",
                    "other": "Requests outside billing and delivery",
                },
            ),
            "refund_requested": Noul(
                instructions="Does the customer explicitly request a refund?",
            ),
        },
    )
    response = triage.evaluate(
        state="My package never arrived. Please refund my payment.",
    )
    print(response.choices["department"].choice)
    print(response.nouls["refund_requested"].noul)
    print(response.to_json(indent=2))

This executes four binary candidates. Actual values depend on the model and prompt; the message intentionally contains both delivery and billing evidence. Binding snapshots reusable rules once. Each evaluation supplies one record, and iter_evaluate() processes records lazily and sequentially. Passing a whole list of emails as one state asks questions about that list; it does not classify each email separately. Question IDs are result keys: put the predicate in instructions, rather than relying on a name such as refund_requested.

Choice and Score report confidence as distribution concentration, not calibrated correctness. For example, raw candidate support [0.009, 0.001] and [0.9, 0.1] both become [0.9, 0.1] with confidence 0.8. The API does not automatically abstain. Select an application policy on held-out data; an other option can help represent scope but does not guarantee rejection. Probability semantics · Python workflow.

Does it actually make local inference faster?

In the measured configuration, yes—by a modest amount against one-token generation. It is not a general acceleration switch, and the tested small model is not accurate enough for unattended refund decisions.

Historical measurement of wheel llms2jev==0.1.0; these timings are not a fresh benchmark of 0.2.0. Version 0.2.0 has separately passed installed-wheel inference with Qwen2.5 on Python 3.8. See the component validation record. Benchmark hardware: Apple M4, 24 GB RAM; Qwen2.5-0.5B-Instruct; MPS, float32; 12 hand-authored English records × 3 repetitions. The configured predicate uses Yes/No and the explicit example renderer. The one-token baseline uses the same prompt. Timings include tokenization, model execution, synchronization, and output handling; model loading is excluded.

Tail projection reduced median scoring time by 18.1% versus full logits (1.22×), and this scoring path took 13.7% less time than one-token generation (1.16×). JSON was slower and less accurate here; that comparison does not establish equal-quality application savings. It is prompted JSON, not grammar-constrained decoding.

These are smoke measurements, not a held-out quality study. The configured binary path missed two of six refund requests and incorrectly accepted two of six negative records. Default rendering with lowercase labels scored every record positive with both tested small Qwen models. No calibration, downstream savings, energy reduction, or universal speedup is claimed. Raw observations, configuration failures, and reproduction commands.

Watch the actual run

Play/download the 13-second MP4. This historical 0.1.x recording shows a real subprocess at wall-clock speed, including model loading, real probabilities, measured per-call times, and expected versus predicted decisions. Its three demo timings are a separate run from the repeated benchmark. Timestamped output.

Where the latency and cost can go

rules ── bind once ──> candidate plan
                            + one record
                            ↓
                     C binary prompts
                            ↓
                 next-token yes/no scores
                            ↓
                   typed decision results
                            ↓
                 application filter / route / count

For one record, C is the sum of Choice options, Score levels, and Noul candidates. Each prompt contains the state. Transformers batches candidate prompts, but recomputes their input representations; binding is not a KV or prefix cache. Ollama sends candidates sequentially, including through its async adapter.

A useful break-even comparison is C candidate prefills and their serving overhead versus one prefill plus the baseline's output decoding. Batching can reduce wall-clock time without eliminating input work. Compare against a minimal generated label or constrained structured output, not only a verbose reasoning response. Input length, candidate count, label availability, hardware, and downstream mistakes all matter. Evaluation protocol.

Failure workloads and limits

The adapters accept text/JSON state; they do not provide OCR, audio understanding, browser control, or evidence retrieval. Deployment and correctness findings are recorded in the architecture review.

Existing techniques and this project's contribution

Binary next-token scoring is established: Qwen3-Reranker uses yes/no logits for relevance scoring. Semantic data operators are also established in LOTUS. TypeSafe publishes a System One Adapter for obtaining its decision interface from language models. LLM2Jev does not claim to invent these ideas.

The current contribution is their integration into a small decision runtime: immutable rule compilation, per-execution state snapshots, typed probability assembly, shared Python/HTTP execution, and explicit runtime ownership and capability failures. Unlike the generated-answer adapter, this runtime reads binary token scores and assembles answers without parsing generated probabilities. Its practical value must be established by showing useful decisions at lower total cost or latency at a fixed error budget. No new model or training method is claimed. The measured execution advantage below is limited to its stated configuration and does not establish general application savings. See the workload evaluation protocol.

Execution and development

The root llms2jev exports are the public Python API. application/ orchestrates evaluation through scoring contracts; runtime/ owns model execution and resources; inference/ owns the scoring-independent algorithms. transport/ parses and routes HTTP, while serving/ composes these parts and owns process lifespan. Runtimes do not import application or inference implementations. Architecture · API · Privacy · Documentation index.

src/llms2jev/
├── core/                    Domain values and wire serialization
├── contracts.py             Runtime protocols, labels, identities, BinaryScores
├── application/             Services, bound evaluators, per-call preparation
├── inference/               Compilation, rendering, normalization, assembly
├── runtime/
│   ├── ollama/              HTTP execution, wire policy, configuration
│   ├── transformers/        Tensor execution and provider tokenization
│   └── lifecycle.py         Shared resource state machines
├── transport/               Framework-free parsing and HTTP routes
├── serving/                 App composition, authentication, lifespan, CLI
└── utils/                   Internal JSON and probability primitives

Customize prompts with ProbeEncoder.encode(CandidateProbe) and encoder=, and probability policies with distribution=. See the API and packaging decision.

Use LLM2Jev(runtime=...) to inject model execution. Custom runtimes return BinaryScores; question/answer JSON contracts are defined in the API reference.

uv run python -m unittest discover -s tests -v
uv run python -m compileall -q src tests
uv run mypy src/llms2jev
uv build

Tests cover deterministic scoring, state isolation, lifecycle, HTTP, and diagnostic privacy. CPU tensor tests need the optional PyTorch dependency and otherwise skip. These checks establish software contracts; the small local-model experiment establishes bounded execution evidence; held-out accuracy, calibration, scaling, and downstream cost remain open evaluation work in the implementation plan.

CLI and MCP

Version 0.4.1 provides llms2jev health, llms2jev models, llms2jev evaluate, and llms2jev-mcp (stdio). Install them with pip install 'llms2jev[server,mcp]==0.4.1'. They share one HTTP client and call the existing Ollama-backed llms2jev-serve; MCP needs Python 3.10+ and the optional SDK 2.x dependency. See the complete CLI/MCP tutorial for source installation, JSON examples, client configuration, and real-run evidence.

Contributions — PRs welcome

PRs are welcome: bug fixes, clearer documentation, reproducible evaluations, and runtime improvements tied to a concrete use case. For architectural or public API changes, open an issue first to discuss the behavior and tradeoffs.

Describe the problem, the resulting behavior, and the checks you ran. Keep changes focused, preserve documented contracts, and include regression tests for behavior changes. Performance claims need reproducible measurements with hardware, dependency versions, baselines, and correctness results. Retain source attribution when adapting ideas or code.

See the contribution guide for setup, architecture rules, and the PR checklist. Submit PRs to llms2jev-releases.

Licensed under MIT.