oev 0.3.0

At a glance

• 22.2 ms per question on a T4 (GPU); on CPU the ONNX INT8 build runs at 54.2 ms p50 on 8 threads (228 MB artifact, 3.2x smaller)
• 0.8584 on 77-label Banking77 from a single soup checkpoint (the 3-checkpoint ensemble still holds best ECE 0.0595): each option is embedded as its own anchor with full tokens, so accuracy scales with label count (gap to Jev 1.16 pts)
• 184M params, Apache 2.0 weights
• Kev (0.8B / 4B) publishes no in-domain numbers on these datasets, so it is not in the tables; see BENCHMARKS.md for the like-for-like comparison plan

Architecture

flowchart LR
    subgraph input["Input"]
        direction TB
        S["state<br/>text or JSON"]
        Q["questions<br/>choice, noul, score"]
    end

    P["packer<br/>state + questions + anchors<br/>one packed sequence"]

    subgraph pass["One forward pass, 22 ms on T4"]
        direction TB
        E["encoder<br/>DeBERTa-v3-base, 184M<br/>or the from-scratch char model"]
        H["shared linear head<br/>scores every anchor"]
    end

    D["softmax per question<br/>calibrated distribution"]
    O["outputs<br/>choice / noul / score"]

    S --> P
    Q --> P
    P --> E
    E --> H
    H --> D
    D --> O

• One anchor mechanism covers all three primitives - options, yes/no pairs, and score levels are each embedded as anchors in one packed sequence
• No text generation - nothing to parse, nothing to hallucinate
• New question types need no new heads - new options are just new anchors

Benchmarks: OEV vs the published field

Fine-tuned on each benchmark's train split, following the same protocol as Laya's published runs. Selected results and evaluation notes are in BENCHMARKS.md.

Quickstart

pip install oev          # from PyPI
# or from source:
pip install -e .

Optional extras:

• pip install -e ".[dev]" - pytest
• pip install -e ".[data]" - dataset converters
• pip install -e ".[backbone]" - DeBERTa fine-tuning
• pip install -e ".[serve]" - FastAPI server (oev-serve)
• pip install -e ".[app]" - Gradio Space dependencies

from oev.infer import OEV

agent = OEV("checkpoints_td5/oev-tiny.pt", device="cpu")

result = agent.decide("We were charged twice for the same order.", {
    "department": {"type": "choice", "options": ["billing", "technical", "sales", "other"],
                   "instructions": "Which department should handle this?"},
    "refund_requested": {"type": "noul"},
    "severity": {"type": "score", "levels": [1, 2, 3, 4, 5]},
})

# Native schema above; the Jev/TypeSafe `criteria` schema (the one laya and Kev
# speak) works unchanged - drop-in for existing Jev clients:
result = agent.decide("We were charged twice for the same order.", {
    "department": {"type": "choice", "instructions": "Which department should handle this?",
                   "criteria": {"billing": "invoices, payments", "technical": "bugs, outages"}},
    "refund_requested": {"type": "noul"},
    "severity": {"type": "score", "criteria": ["minor", "soon", "blocking"]},
})

{
  "department": {
    "choice": "billing",
    "probabilities": {
      "billing": 0.94,
      "technical": 0.04,
      "sales": 0.01,
      "other": 0.01
    },
    "confidence": 0.94
  },
  "refund_requested": 0.91,
  "severity": {
    "value": 3,
    "probabilities": { "1": 0.02, "2": 0.08, "3": 0.61, "4": 0.22, "5": 0.07 }
  }
}

Confidence gating - automate when confident, escalate when not:

from oev.presets import triage_questions, gate

for name, payload, confident in gate(result, threshold=0.85):
    automate(name, payload) if confident else escalate_to_human(name)

HTTP server (native + Jev-compatible /v1/systemone endpoint - TypeSafe clients work by changing baseUrl):

pip install -e ".[serve]"
oev-serve --checkpoint checkpoints_td5/oev-tiny.pt --port 8000
curl -X POST localhost:8000/decide -H "Content-Type: application/json" \
  -d '{"state": "My payment failed twice", "questions": {"urgency": {"type": "score", "criteria": ["not urgent", "soon", "blocking"]}}}'

Docker:

docker compose up   # checkpoint at ./checkpoints/oev-tiny.pt

Training

python -m oev.convert_typed

python -m oev.train --backbone microsoft/deberta-v3-base --epochs 4 --batch-size 8 --max-len 768 --data-dir data/typed --out checkpoints_td5

python -m oev.rlcd --checkpoint checkpoints_td5/oev-tiny.pt --data-dir data/typed --epochs 2 --batch-size 8 --out checkpoints_rlcd

python -m oev.ensemble --ckpts checkpoints_td5/oev-tiny.pt,checkpoints_rlcd/oev-tiny.pt,checkpoints_rlcd_soup/oev-tiny.pt,checkpoints_rlcd_seed1/oev-tiny.pt --data-dir data/typed

python -m pytest -q

train_colab.ipynb runs the entire pipeline end to end. Evaluation rules and scope notes are in BENCHMARKS.md.

Reading the numbers

• Each benchmark number comes from a checkpoint fine-tuned on that benchmark's train split, matching the baselines' published protocol
• Zero-shot emotion is a separate head-to-head win: shipped student 0.6505 vs laya 0.595 (both zero-shot)
• The 0.7760 headline is a 4-checkpoint ensemble; the best single file is 0.7705
• The Banking77 single-file best is the soup at 0.8584; 0.8403 is a historical re-tune, not the shipped artifact
• Adversarial NLI (ANLI) stays near chance for now; WANLI reaches 0.5645 and an ANLI specialist is next on the roadmap
• The shipped generalist trails the typed specialist (0.6480 vs 0.7705); closing that spread is round 4
• CPU: torch fp32 measured 447 ms p50 at release; on the current workstation the same fp32 model runs 252.5 ms and the ONNX INT8 build 54.2 ms (8 threads, scripts/bench_latency.py). Headline timings are GPU
• English only

Roadmap

• Round 3 distillation: 6 teachers, 5 domains including NLI
• 4-member Banking77 ensemble: the live shot past 0.8584
• Round 4: one file near specialist numbers everywhere
• INT8 / ONNX CPU deployment (export + quantization scripts in scripts/, bench pending)
• Multi-question shared-state encoding (one pass, many questions)
• Robustness: reduce mild overconfidence on out-of-distribution inputs
• Non-English checkpoints (the interface is language-agnostic; the weights are not yet)

Full list: ROADMAP.md.

Credits

The interface and benchmark protocol follow Laya, Kev and the System One model category introduced by TypeSafe's Jev. Their published numbers are quoted here for comparison and remain their measurements.