laya-apple

Correctness-validated heterogeneous Laya runtime for Apple silicon. It runs the MLX GPU and the Apple Neural Engine at the same time, and uses the Neural Engine only where it has been proven to give the same decisions as upstream Laya.

The gain comes from running both engines at once, not from raw ANE latency. This is the v1.0 benchmark on one Apple M4 Max with macOS 26.6.2: one short and one long request stream through one Laya(execution="workers") instance (see GPU + ANE heterogeneous serving). Other Macs are untested, and you can add yours. The method and raw data are in benchmarks/v1.0.md.

Why this exists

Laya answers typed questions about a context (choice, score, noul) in one forward pass. On a Mac there are two engines that can run it, with different strengths.

1. Correct ANE execution. A fast Core ML export is not necessarily correct. On the tested Mac, the ordinary Core ML export ran on the Neural Engine without any error and changed up to 85 decisions against upstream PyTorch. laya-apple ships an ANE artifact only after it passes a parity gate on the machine that uses it (docs/correctness.md).
2. Automatic routing. Short, validated single-question requests go to the ANE. Long or multi-question requests go to the MLX GPU. The router decides before a request runs and records why.
3. Concurrent GPU + ANE serving. Both engines serve independent requests at the same time, so short requests stop queueing behind long ones.

Install

Apple silicon, Python 3.11–3.13:

pip install laya-apple

Optional extras:

pip install "laya-apple[ane]"       # + the Neural Engine runtime (coremltools 9.0)
pip install "laya-apple[convert]"   # + building ANE artifacts on this Mac (torch 2.7.0)
laya-apple artifacts build laya-typed-decisions   # optional: build + parity-validate ANE artifacts here (~5 min)

Without the ane extra, or without a built artifact, everything runs on the MLX GPU. To run from source or develop laya-apple, see CONTRIBUTING.md.

Quickstart (30 seconds)

from laya_apple import Laya

model = Laya.from_pretrained(
    "convaiinnovations/laya-typed-decisions",
    device="auto",
)

result = model.predict(
    context="The customer was charged twice for the same invoice and is frustrated.",
    questions={
        "urgency": {
            "type": "choice",
            "instructions": "How urgent is this?",
            "criteria": ["low", "medium", "high"],
        }
    },
)
print(result.answers["urgency"]["choice"], result.answers["urgency"]["probabilities"])

rt = result.runtime
print(rt.backend, rt.device, rt.routing_reason, f"{rt.latency_ms:.1f} ms")

On the tested machine:

high {'low': 0.1713, 'medium': 0.3358, 'high': 0.4929}
coreml ane validated_short_single_question_path 11.2 ms

• The first call downloads the pinned checkpoint. After that it works offline (local_files_only=True).
• Without ANE artifacts, the same request runs on MLX and routing_reason says why.
• More: examples/ (basic.py, auto_routing.py, heterogeneous_serving.py) and the user guide.

How auto routing works

The production threshold is more conservative than the measured crossover. laya-multilingual is slightly faster on the ANE at exactly 256 tokens (8.3 ms against 8.6 ms), but that bucket stays explicit-only, because it does not beat MLX at the previous bucket, 128 tokens. Every result carries routing_reason.

How the thresholds are derived: docs/support-matrix.md. How the pieces fit together: docs/architecture.md.

GPU + ANE heterogeneous serving

with Laya.from_pretrained("convaiinnovations/laya-typed-decisions", execution="workers") as model:
    futures = [model.submit(context=c, questions=q) for c, q in requests]   # thread-safe

• The GPU runs in a worker process, and the ANE on its own dispatcher.
• Each request runs on one device, chosen by the router.
• Under load, the router also compares queue backlogs.

Short-request P99 under open-loop bursty arrivals, measured from arrival with queueing included (v1.0, same arrival sequence for both):

A single short request is not dramatically faster on the ANE (for example 9.9 against 12.2 ms). The gain comes from using both engines at once.

Correctness

Parity against upstream Laya on PyTorch CPU FP32, over the shipped golden rows (v1.0). Each cell gives hard mismatches, then the max probability error.

• The FP16 gate: probability error ≤ 0.02 and 0 hard mismatches.
• Near-tie flips (upstream's top-two margin < 0.04) are listed, not hidden.
• Explicit ANE requests never fall back. They run the validated artifact or raise, and every loaded artifact is also timed against CPU_ONLY to catch a silent CPU placement.
• Definitions, every configuration tested, and the fallback audit are in docs/correctness.md and docs/no-silent-fallback.md.

Supported models and platforms

Tested:

• Apple M4 Max, macOS 26.6.2, MLX 0.32.2, coremltools 9.0;
• Python 3.11–3.13.

Other Apple silicon:

• MLX is expected to work.
• auto stays on MLX until artifacts are built and calibrated on that machine (laya-apple calibrate).

See docs/compatibility.md and the community matrix in docs/community-benchmarks.md.

Reproduction

Each headline number above traces to a report, raw data, a command and an environment in docs/reproducibility.md. The full v1.0 suite, which compares PyTorch CPU/MPS, the ordinary Core ML export, MLX and laya-apple, is in benchmarks/v1.0.md. The quick check for your own Mac:

uv run python scripts/hardware_report.py --quick

Contributing

The most useful first contribution is a benchmark from a Mac other than an M4 Max: run the command above and open a PR with hardware-results/ (how).

• CONTRIBUTING.md covers setup, test tiers (which tests a change actually needs), parity checks and backend changes.
• Open work is labelled good first issue, help wanted and research.

Limitations

• One test machine. Every benchmark is from one Apple M4 Max on macOS 26.6.2. Routing thresholds are not assumed to hold on other Apple SoCs.
• Long contexts stay on MLX, which is faster there. The ANE path is batch 1 only.
• Isolation is partial. Under concurrency, each stream's P99 is above its solo value.
• Cold start on a fresh artifact location costs 3–5 minutes of Core ML compile per model. ane_startup="background" serves on MLX in the meantime.
• choice decisions can depend on option order. This comes from upstream Laya, and laya-apple reproduces it exactly (research/option-order/).
• Not measured yet: energy use, quantized artifacts and cross-SoC validation.

More

• User guide: docs/guide.md.
• Stable API: docs/api.md.
• Architecture: docs/architecture.md.
• Changes: CHANGELOG.md.
• Security: SECURITY.md.

Apache-2.0; see LICENSE and NOTICE. Model weights are downloaded from their pinned Hugging Face revisions and are not redistributed. This is an independent project, not an official release of Convai Innovations, Apple or MLX.