laya-mnn

Run the Laya System 1 decision model locally on MNN — no PyTorch, no GPU, no API key.

Laya takes a state (email, ticket, JSON) plus typed questions and returns typed answers with calibrated probabilities in one forward pass. It never generates text, so there is nothing to parse and nothing to hallucinate. This package runs a converted Laya checkpoint on Alibaba's MNN runtime.

uvx laya-mnn decide \
  --state "Hi, we were billed twice for March. Please refund the duplicate or we will cancel our plan." \
  --questions '{"department":{"type":"choice","instructions":"Which department should handle this?","criteria":{"billing":"invoices, payments, refunds","technical":"bugs, outages","other":"everything else"}},
                "churn_risk":{"type":"noul","instructions":"Does the user threaten to cancel?"}}'

{
  "model": "laya-mnn",
  "answers": {
    "department": {
      "type": "choice", "choice": "billing",
      "probabilities": { "billing": 0.9631, "technical": 0.0165, "other": 0.0204 },
      "confidence": 0.8067, "action": { "act_probability": 0.9977 }
    },
    "churn_risk": { "type": "noul", "noul": 0.9476, "confidence": 0.9476, "action": { "act_probability": 0.9977 } }
  }
}

Weights download automatically on first use (about 850 MB for fp16).

Question types

criteria is defined at request time — new schemas do not need retraining. Up to 64 options per question with the shipped checkpoints.

CLI

laya-mnn decide --state S --questions Q [--precision fp16|int8] [--threads N] [--quiet]
laya-mnn download [--precision ...]      # pre-fetch weights
laya-mnn info     [--precision ...]      # cache dir + manifest

--state accepts literal text, inline JSON, or @path. --questions accepts inline JSON or @path.

Python

from laya_mnn import LayaMNN, ensure_model

model = LayaMNN(ensure_model("fp16"), threads=4)
result = model.system_one(
    {"subject": "Duplicate charge", "body": "We were billed twice..."},
    {"department": {"type": "choice", "instructions": "Which department?", "criteria": ["billing", "technical", "other"]}},
)
print(result["answers"]["department"]["choice"], result["answers"]["department"]["probabilities"])

Checkpoints

Both are conversions of convaiinnovations/laya (English, ModernBERT-large backbone, 421M parameters). The model was exported to ONNX (opset 17, eager attention, static 512-token sequence) and converted with the MNN converter 3.6.1.

Hosted on Hugging Face and ModelScope; modelscope is tried first, then Hugging Face (override with HF_ENDPOINT). Set LAYA_MNN_CACHE to change the cache directory (default ~/.cache/laya-mnn).

Notes and limits

• CPU is the supported backend. The MNN Metal backend was measured to return all-zero logits for this graph in some builds of the 3.6.1 wheel; --backend metal is exposed but not recommended.
• A 512-token forward pass costs roughly 1.3 s on an Apple Silicon CPU. Shorter states are cheaper — the graph is exported at a fixed 512 tokens, so the CLI pads and masks.
• Max 64 options per question, 512 tokens of context, English only. The multilingual checkpoint (mmBERT-base) is not converted here.
• Percentile-latency and calibration in the upstream model card are for the original PyTorch model; quantized results will differ slightly.

License and attribution

Apache-2.0. The model, its prompt format and the calibration scheme come from NandhaKishorM/laya by Convai Innovations (Apache-2.0); this package is an independent MNN port and is not affiliated with or endorsed by them. See NOTICE.