oev 0.2.0

At a glance

• 22.2 ms per question on a T4 (GPU); 447 ms p50 on CPU (8 threads, 184M soup checkpoint)
• 0.8584 on 77-label Banking77 from a single soup checkpoint (the 3-checkpoint ensemble still holds best ECE 0.0595): each option is embedded as its own anchor with full tokens, so accuracy scales with label count (gap to Jev 1.16 pts)
• 184M params, Apache 2.0 weights
• Kev (0.8B / 4B) publishes no in-domain numbers on these datasets, so it is not in the tables; see BENCHMARKS.md for the like-for-like comparison plan

Architecture

flowchart LR
    subgraph input["Input"]
        direction TB
        S["state<br/>text or JSON"]
        Q["questions<br/>choice, noul, score"]
    end

    P["packer<br/>state + questions + anchors<br/>one packed sequence"]

    subgraph pass["One forward pass, 22 ms on T4"]
        direction TB
        E["encoder<br/>DeBERTa-v3-base, 184M<br/>or the from-scratch char model"]
        H["shared linear head<br/>scores every anchor"]
    end

    D["softmax per question<br/>calibrated distribution"]
    O["outputs<br/>choice / noul / score"]

    S --> P
    Q --> P
    P --> E
    E --> H
    H --> D
    D --> O

flowchart TB
    subgraph input["Input"]
        direction TB
        S["state<br/>text or JSON"]
        Q["questions<br/>choice, noul, score"]
    end

    P["packer<br/>state + questions + anchors<br/>one packed sequence"]

    subgraph pass["One forward pass, 22 ms on T4"]
        direction TB
        E["encoder<br/>DeBERTa-v3-base, 184M<br/>or the from-scratch char model"]
        H["shared linear head<br/>scores every anchor"]
    end

    D["softmax per question<br/>calibrated distribution"]
    O["outputs<br/>choice / noul / score"]

    S --> P
    Q --> P
    P --> E
    E --> H
    H --> D
    D --> O

• One anchor mechanism covers all three primitives - options, yes/no pairs, and score levels are each embedded as anchors in one packed sequence
• No text generation - nothing to parse, nothing to hallucinate
• New question types need no new heads - new options are just new anchors

Benchmarks: OEV vs the published field

Fine-tuned on each benchmark's train split, following the same protocol as Laya's published runs. Selected results and evaluation notes are in BENCHMARKS.md.

Quickstart

pip install oev          # from PyPI
# or from source:
pip install -e .

Optional extras:

• pip install -e ".[dev]" - pytest
• pip install -e ".[data]" - dataset converters
• pip install -e ".[backbone]" - DeBERTa fine-tuning
• pip install -e ".[serve]" - FastAPI server (oev-serve)
• pip install -e ".[app]" - Gradio Space dependencies

from oev.infer import OEV

agent = OEV("checkpoints_td5/oev-tiny.pt", device="cpu")

result = agent.decide("We were charged twice for the same order.", {
    "department": {"type": "choice", "options": ["billing", "technical", "sales", "other"],
                   "instructions": "Which department should handle this?"},
    "refund_requested": {"type": "noul"},
    "severity": {"type": "score", "levels": [1, 2, 3, 4, 5]},
})

{
  "department": {
    "choice": "billing",
    "probabilities": {
      "billing": 0.94,
      "technical": 0.04,
      "sales": 0.01,
      "other": 0.01
    },
    "confidence": 0.94
  },
  "refund_requested": 0.91,
  "severity": {
    "value": 3,
    "probabilities": { "1": 0.02, "2": 0.08, "3": 0.61, "4": 0.22, "5": 0.07 }
  }
}

Confidence gating - automate when confident, escalate when not:

from oev.presets import triage_questions, gate

for name, payload, confident in gate(result, threshold=0.85):
    automate(name, payload) if confident else escalate_to_human(name)

HTTP server (native + Jev-compatible /v1/systemone endpoint - TypeSafe clients work by changing baseUrl):

pip install -e ".[serve]"
oev-serve --checkpoint checkpoints_td5/oev-tiny.pt --port 8000
curl -X POST localhost:8000/decide -H "Content-Type: application/json" \
  -d '{"state": "My payment failed twice", "questions": {"urgency": {"type": "score", "levels": [1, 2, 3]}}}'

Docker:

docker compose up   # checkpoint at ./checkpoints/oev-tiny.pt

Training

python -m oev.convert_typed

python -m oev.train --backbone microsoft/deberta-v3-base --epochs 4 --batch-size 8 --max-len 768 --data-dir data/typed --out checkpoints_td5

python -m oev.rlcd --checkpoint checkpoints_td5/oev-tiny.pt --data-dir data/typed --epochs 2 --batch-size 8 --out checkpoints_rlcd

python -m oev.ensemble --ckpts checkpoints_td5/oev-tiny.pt,checkpoints_rlcd/oev-tiny.pt,checkpoints_rlcd_soup/oev-tiny.pt,checkpoints_rlcd_seed1/oev-tiny.pt --data-dir data/typed

python -m pytest -q

train_colab.ipynb runs the entire pipeline end to end. Evaluation rules and selected limitations are in BENCHMARKS.md.

Limitations

• every benchmark number is from a checkpoint fine-tuned on that benchmark's train split (the 0.9300 emotion figure included). Zero-shot emotion is a separate win: the shipped distilled student (0.6505, both models zero-shot) against laya's 0.595, up from the round-1 starting point of 0.4265
• ANLI remains near chance (0.3380 for the round-1 student), while the WANLI specialist reaches 0.5645; the NLI result is split-dependent, not uniformly at chance
• pure-mimicry distillation transfers breadth, not depth. The round-1 student hit emotion 0.6875 (+26 pts zero-shot) but lost typed skill (0.5385). Adding gold-CE loss (round 2) collapsed to uniform, a documented negative result. Round 2b (pure-KL, balanced domains) rescued it: typed 0.6480, emotion 0.6505, b77 0.7964, probes all PASS
• the headline ensemble result is an average of four checkpoints; the best single model is 0.7705
• CPU inference is roughly 20x slower than the T4 (447 ms p50, 8 threads) - all headline timings are GPU
• the b77 headline includes a 3-checkpoint probability ensemble and a single-file soup checkpoint at 0.8584; 0.8403 is a historical warm-start re-tune, not the current best single artifact
• English only

Roadmap

• round 3 distillation: 6 teachers, 5 domains including NLI
• 4-member Banking77 ensemble: the live shot past 0.8584
• round 4: one file near specialist numbers everywhere
• INT8 / ONNX CPU deployment (export + quantization scripts in scripts/, bench pending)
• multi-question shared-state encoding (one pass, many questions)
• robustness: reduce mild overconfidence on out-of-distribution inputs
• non-English checkpoints (the interface is language-agnostic; the weights are not yet)

Full list: ROADMAP.md.

Credits

The interface and benchmark protocol follow Laya, Kev and the System One model category introduced by TypeSafe's Jev. Their published numbers are quoted here for comparison and remain their measurements.