oev 0.4.0

At a glance

• 22.2 ms per question on a T4 (GPU); on CPU the ONNX INT8 build runs at 54.2 ms p50 on 8 threads (228 MB artifact, 3.2x smaller)
• 0.8594 on 77-label Banking77 from a single soup checkpoint (the 4-checkpoint ensemble holds best ECE 0.0583): each option is embedded as its own anchor with full tokens, so accuracy scales with label count (gap to Jev 1.06 pts)
• 184M params, Apache 2.0 weights
• Kev (0.8B / 4B) publishes no in-domain numbers on these datasets, so it is not in the tables; see BENCHMARKS.md for the like-for-like comparison plan

Architecture

flowchart LR
    subgraph input["Input"]
        direction TB
        S["state<br/>text or JSON"]
        Q["questions<br/>choice, noul, score"]
    end

    P["packer<br/>state + questions + anchors<br/>one packed sequence"]

    subgraph pass["One forward pass, 22 ms on T4"]
        direction TB
        E["encoder<br/>DeBERTa-v3-base, 184M"]
        H["shared linear head<br/>scores every anchor"]
    end

    D["softmax per question<br/>calibrated distribution"]
    O["outputs<br/>choice / noul / score"]

    S --> P
    Q --> P
    P --> E
    E --> H
    H --> D
    D --> O

• One anchor mechanism covers all three primitives - options, yes/no pairs, and score levels are each embedded as anchors in one packed sequence
• No text generation - nothing to parse, nothing to hallucinate
• New question types need no new heads - new options are just new anchors

A from-scratch character-level encoder also exists for the no-transformers path (see oev/tokenizer.py); the diagrams and benchmarks here all use the DeBERTa backbone.

Why not an LLM?

A generative model answers a structured question like this:

input -> generate text -> parse the output -> validate it -> maybe get a decision

OEV is built for the case where the system already knows the candidate answers:

state + candidates -> score every candidate -> calibrated probability distribution

If you need open-ended text, use an LLM. If you need many small, bounded decisions - which department, is this refund requested, how severe, escalate or continue - scoring known options directly is cheaper, faster, and structurally immune to output-parsing failures: the outputs are probabilities over the inputs you supplied. Different tool for a different layer of the stack.

Benchmarks: OEV vs the published field

Fine-tuned on each benchmark's train split, following the same protocol as Laya's published runs. Selected results and evaluation notes are in BENCHMARKS.md.

Quickstart

pip install oev          # from PyPI
# or from source:
pip install -e .

Optional extras:

• pip install -e ".[dev]" - pytest
• pip install -e ".[data]" - dataset converters
• pip install -e ".[backbone]" - DeBERTa fine-tuning
• pip install -e ".[serve]" - FastAPI server (oev-serve)
• pip install -e ".[app]" - Gradio Space dependencies

from oev import OEV

agent = OEV("checkpoints_td5/oev-tiny.pt", device="cpu")

result = agent.decide("We were charged twice for the same order.", {
    "department": {"type": "choice", "options": ["billing", "technical", "sales", "other"],
                   "instructions": "Which department should handle this?"},
    "refund_requested": {"type": "noul"},
    "severity": {"type": "score", "levels": [1, 2, 3, 4, 5]},
})

# Native schema above; the Jev/TypeSafe `criteria` schema (the one laya and Kev
# speak) works unchanged - drop-in for existing Jev clients:
result = agent.decide("We were charged twice for the same order.", {
    "department": {"type": "choice", "instructions": "Which department should handle this?",
                   "criteria": {"billing": "invoices, payments", "technical": "bugs, outages"}},
    "refund_requested": {"type": "noul"},
    "severity": {"type": "score", "criteria": ["minor", "soon", "blocking"]},
})

# The candidate set is an inference-time input - swap it per call, same weights:
state = "play my morning playlist and take a note"
actions = ["open_spotify", "search_web", "open_vscode", "create_note"]
result = agent.decide(state, {"action": {"type": "choice", "options": actions}})

{
  "department": {
    "choice": "billing",
    "probabilities": {
      "billing": 0.94,
      "technical": 0.04,
      "sales": 0.01,
      "other": 0.01
    },
    "confidence": 0.94
  },
  "refund_requested": 0.91,
  "severity": {
    "value": 3,
    "probabilities": { "1": 0.02, "2": 0.08, "3": 0.61, "4": 0.22, "5": 0.07 }
  }
}

Confidence gating - automate when confident, escalate when not:

from oev.presets import triage_questions, gate

for name, payload, confident in gate(result, threshold=0.85):
    automate(name, payload) if confident else escalate_to_human(name)

HTTP server (native + Jev-compatible /v1/systemone endpoint - TypeSafe clients work by changing baseUrl):

pip install -e ".[serve]"
oev-serve --checkpoint checkpoints_td5/oev-tiny.pt --port 8000
curl -X POST localhost:8000/decide -H "Content-Type: application/json" \
  -d '{"state": "My payment failed twice", "questions": {"urgency": {"type": "score", "criteria": ["not urgent", "soon", "blocking"]}}}'

Docker:

docker compose up   # checkpoint at ./checkpoints/oev-tiny.pt

Training

python -m oev.convert_typed

python -m oev.train --backbone microsoft/deberta-v3-base --epochs 4 --batch-size 8 --max-len 768 --data-dir data/typed --out checkpoints_td5

python -m oev.rlcd --checkpoint checkpoints_td5/oev-tiny.pt --data-dir data/typed --epochs 2 --batch-size 8 --out checkpoints_rlcd

python -m oev.ensemble --ckpts checkpoints_td5/oev-tiny.pt,checkpoints_rlcd/oev-tiny.pt,checkpoints_rlcd_soup/oev-tiny.pt,checkpoints_rlcd_seed1/oev-tiny.pt --data-dir data/typed

python -m pytest -q

train_colab.ipynb runs the entire pipeline end to end. Evaluation rules and scope notes are in BENCHMARKS.md.

Roadmap

• Round 3 distillation: 3 teachers, 5 domains including NLI - typed 0.6895 (r2b 0.6480), Banking77 0.8370, emotion zero-shot 0.8650, checkpoint on the Hub
• Push Banking77 past Jev's 0.870: soup4 landed 0.8594; the lever is lineage diversity, data augmentation or distillation, not more warm-starts
• Round 4: 10-domain distill from the round-3 student - typed 0.6985, emotion zero-shot 0.8700, b77 0.8188 (stated trade), 5 h2h fix domains in
• MNLI specialist rebuilt: WANLI zero-shot 0.5265 (lost original 0.5645), Hub file fixed
• ANLI R1 fine-tune: first adversarial NLI row, 0.5750 in-domain (chance 0.333), WANLI transfer up to 0.5690
• INT8 / ONNX CPU deployment: 54.2 ms p50 on 8 threads, 228 MB artifact (scripts/bench_latency.py)
• Multi-question shared-state encoding (one pass, many questions)
• Robustness: reduce mild overconfidence on out-of-distribution inputs
• Non-English checkpoints (the interface is language-agnostic; the weights are not yet)

Full list: ROADMAP.md.

Documentation

• Benchmarks - methodology, caveats, reproduction commands, checkpoint hashes
• Model card - checkpoints and usage
• Roadmap - what is next
• Contributing - how to open issues and PRs
• Changelog - release history
• Security - supported versions and private reporting
• Hugging Face demo - try it in the browser
• PyPI - pip install oev
• Server API - oev-serve --help, schema docs in oev/serve.py

Credits

The interface and benchmark protocol follow Laya, Kev and the System One model category introduced by TypeSafe's Jev. Their published numbers are quoted here for comparison and remain their measurements.