Laya-MLX

Open-weight typed decisions, running natively on Apple Silicon.

中文 · Usage reference · Benchmarks · Hugging Face weights · Changelog

Laya answers constrained questions (pick an option, score against a rubric, judge true or false) with calibrated probabilities from one bidirectional forward pass. It generates no tokens and returns no JSON to parse. This package runs the original Laya weights in MLX, without PyTorch, Transformers or a cloud API.

13.4 ms median for a short English decision, 7.4 ms with the multilingual checkpoint (M3 Max, FP16).

Quick start

Requires Apple Silicon, macOS 14+ and Python 3.11+.

pip install laya-mlx

import laya_mlx as laya

agent = laya.load("aac6fef/laya-mlx")
result = agent.predict(
    "I was billed twice. Please refund the duplicate.",
    {
        "department": {
            "type": "choice",
            "instructions": "Which department should handle this request?",
            "criteria": {
                "billing": "Payments, invoices, refunds, and duplicate charges.",
                "technical": "Broken features, errors, and product troubleshooting.",
                "sales": "Pricing, upgrades, and new purchases.",
            },
        },
        "refund": {
            "type": "noul",
            "instructions": "Does the customer ask for money back?",
        },
    },
)
print(result["answers"]["department"]["choice"])  # billing

Describing each option, as above, works much better than bare labels. The first load downloads the checkpoint; later inference is fully local.

Question types

States can be text, JSON-like dicts or conversation lists. Each answer carries calibrated probabilities and confidence fields; confidence does not guarantee accuracy on a new task.

Checkpoints

laya.load accepts either column. Context counts instructions, options and state together. Each published checkpoint includes its model card, validation results, provenance and checksums (pinned revisions and hashes).

What else it does

All of these are covered in the usage reference:

• Language routing: Router() detects the language, picks a checkpoint and keeps up to two resident. You can register your own checkpoints.
• Batches and long documents: predict_batch shares forward passes across identical question schemas, and long-text scanning keeps answer attribution.
• Confidence gate: min_confidence= marks low-confidence answers for review without changing them.
• Large choice sets: predict_shortlist (embedding pre-filter) and predict_tournament (no embedder).
• Throughput options: compile=True, pad_to_multiple=16 and cache_prompts=True for repeated workloads.
• Tools and services: CLI (laya-mlx predict), HTTP (laya-mlx-serve), MCP (laya-mlx-mcp-server), evaluation and calibration, and LangChain/LlamaIndex/CrewAI/Agno integrations, installed as extras.
• Export: laya-mlx convert writes a standalone MLX checkpoint.

Performance

M3 Max (40-core GPU, 128 GiB), FP16, end to end:

Timings include prompt preparation, tokenization, inference, calibration and formatting; model loading is excluded. The throughput row uses batch_size=64 (the default is 16). Latency varies with input length, question count and system load.

All three checkpoints matched upstream's selected answer on 63/63 validation questions in FP32 and FP16 (378/378 comparisons), with deterministic repeated calls and no measured memory growth. That measures port fidelity on those fixtures, not task accuracy. Method, every timing sample and probability errors.

Snake demo

The GIF above is a real local run at original speed: every move asks Laya three questions, and a visible safety layer can override unsafe moves.

pip install 'laya-mlx[demo]'
hf download aac6fef/laya-multilingual-mlx
laya-snake                      # paced; add --max-speed to remove pacing
laya-snake --optimize --max-speed

The terminal must be at least 104 × 35. Space pauses, ↑/↓ changes speed, R resets, Q quits. With --optimize, the measured M3 Max run reached 75.4 moves/s over 2,400 moves with zero deaths. Demo guide · speed and stability · optimization.

Relationship to upstream

This is an independent port, not an official Convai Innovations release. Prompt formatting, calibration, routing, output schema and host APIs follow NandhaKishorM/laya v0.4.17; the encoder and decision heads are reimplemented in MLX. CUDA, ONNX, PyTorch backend controls and training are out of scope. RLCD training and fine-tuning stay upstream. Native boundaries · v0.4.17 migration record · all documentation.

Contributing

Bug reports and pull requests are welcome. Upstream-compatible fixes take priority over new features. See CONTRIBUTING.md for the development setup, tests, benchmarks and release process.

License

Apache-2.0; see LICENSE and NOTICE. Laya and its pretrained weights are by Convai Innovations and upstream contributors. The neural architecture follows Laya and Hugging Face ModernBERT.