oev 0.1.0

At a glance

• 22.2 ms per question on a T4 (GPU); CPU latency not yet characterized
• ECE 0.0298 on typed-decisions after temperature fitting - measured confidence tracks actual accuracy, so it can gate automation
• 0.8303 on 77-label Banking77: each option is embedded as its own anchor with full tokens, so accuracy scales with label count (Jev still leads there)
• 184M params, Apache 2.0 weights

Architecture

flowchart LR
    S["state\n(text / JSON)"] --> P["packer:\nstate + questions + anchors\none sequence"]
    P --> E["encoder\nDeBERTa-v3-base (184M)\nor char transformer"]
    E --> H["one linear head\nscores every ANCHOR"]
    H --> D["softmax per question\n= calibrated distribution"]
    D --> O["choice / noul / score"]

• One anchor mechanism covers all three primitives - options, yes/no pairs, and score levels are each embedded as anchors in one packed sequence
• No text generation - nothing to parse, nothing to hallucinate

New question types need no new heads

Benchmarks: OEV vs the published field

Fine-tuned on each benchmark's train split, following the same protocol as Laya's published runs. Complete tables in BENCHMARKS.md.

Quickstart

pip install -e .

Optional extras:

• pip install -e ".[dev]" - pytest
• pip install -e ".[data]" - dataset converters
• pip install -e ".[backbone]" - DeBERTa fine-tuning

from oev.infer import OEV

agent = OEV("checkpoints_td5/oev-tiny.pt", device="cpu")

result = agent.decide("We were charged twice for the same order.", {
    "department": {"type": "choice", "options": ["billing", "technical", "sales", "other"],
                   "instructions": "Which department should handle this?"},
    "refund_requested": {"type": "noul"},
    "severity": {"type": "score", "levels": [1, 2, 3, 4, 5]},
})

{
  "department": {
    "choice": "billing",
    "probabilities": {
      "billing": 0.94,
      "technical": 0.04,
      "sales": 0.01,
      "other": 0.01
    },
    "confidence": 0.94
  },
  "refund_requested": 0.91,
  "severity": {
    "value": 3,
    "probabilities": { "1": 0.02, "2": 0.08, "3": 0.61, "4": 0.22, "5": 0.07 }
  }
}

Confidence gating - automate when confident, escalate when not:

from oev.presets import triage_questions, gate

for name, payload, confident in gate(result, threshold=0.85):
    automate(name, payload) if confident else escalate_to_human(name)

HTTP server (native + Jev-compatible /v1/systemone endpoint - TypeSafe clients work by changing baseUrl):

pip install -e ".[serve]"
oev-serve --checkpoint checkpoints_td5/oev-tiny.pt --port 8000
curl -X POST localhost:8000/decide -H "Content-Type: application/json" \
  -d '{"state": "My payment failed twice", "questions": {"urgency": {"type": "score", "levels": [1, 2, 3]}}}'

Docker:

docker compose up   # checkpoint at ./checkpoints/oev-tiny.pt

Training

python -m oev.convert_typed

python -m oev.train --backbone microsoft/deberta-v3-base --epochs 4 --batch-size 8 --max-len 768 --data-dir data/typed --out checkpoints_td5

python -m oev.rlcd --checkpoint checkpoints_td5/oev-tiny.pt --data-dir data/typed --epochs 2 --batch-size 8 --out checkpoints_rlcd

python -m oev.ensemble --ckpts checkpoints_td5/oev-tiny.pt,checkpoints_rlcd/oev-tiny.pt,checkpoints_rlcd_soup/oev-tiny.pt --data-dir data/typed

python -m pytest -q   # 38 tests passing

train_colab.ipynb runs the entire pipeline end to end. Full training docs in BENCHMARKS.md.

Limitations

• every benchmark number is from a checkpoint fine-tuned on that benchmark's train split; zero-shot performance is much weaker
• the headline ensemble result is an average of four checkpoints; the best single model is 0.7705
• CPU latency is uncharacterized - all timings are T4 GPU
• Banking77 calibration (ECE 0.186) is markedly worse than typed-decisions (0.0298)
• English only

Roadmap

• distill the ensemble into one 184M model (single-model general skills currently erode after per-benchmark fine-tuning)
• INT8 / ONNX export for CPU deployment
• multi-question shared-state encoding (one pass, many questions)
• b77 confidence-sharpening sweep (ECE 0.186 -> target < 0.10)
• robustness: reduce mild overconfidence on out-of-distribution and garbage inputs
• non-English checkpoints (the interface is language-agnostic; the weights are not yet)

Credits

The interface and benchmark protocol follow Laya and the System One model category introduced by TypeSafe's Jev. Their published numbers are quoted here for comparison and remain their measurements.