English | 简体中文

laya-cuda

A lightweight inference library for Laya

Latency of one request repeated 200 times:

Accuracy on 200 frozen requests (laya-cuda / SDK): laya 52.25% / 52.25%, laya-multilingual 43.25% / 43.25%, laya-typed-decisions 69.25% / 69.75%. Jev, a larger remote model, scores 65.00% on the laya requests at 378 / 466 ms, which includes network latency.

• Setup: RTX 4090, Windows 11. The short request is a real AG News request with one four-option question; the full-context request asks the same question about real text truncated at the model's limit. Each request ran 200 times per process, in 3 processes per backend with alternating order.
• What is timed: the full predict call, including tokenization, GPU work and decoding. Jev's time also includes the network.
• Mixed requests: on 200 different frozen requests (40–900 tokens, 1–5 questions, 4–77 options, each new shape's first request included), laya measures 4.1 / 8.5 ms against 20.1 / 22.1 ms for the SDK: 4.9× / 2.6×. Mixing request sizes raises P95; the latency report explains it.
• SDK precision: the official SDK runs at its default BF16. On the 2,000-decision fixed set, laya-cuda agrees with the SDK at FP32 more closely than BF16 does.
• Protocol: benchmarks/; the latency report records the frozen inputs.

Install

pip install laya-cuda   # core: CuPy + CUDA components, no Torch
laya-cuda doctor laya   # check driver, GPU and CUDA components, then time one prediction

Windows or Linux x86-64, Python 3.12–3.14 and an NVIDIA GPU with compute capability 8.0 or newer (Ampere or later, such as RTX 30/40/50, A100, L4 or H100); no CUDA Toolkit is needed. Measured on an RTX 4090. To work on the source, use uv:

git clone https://github.com/Alexw1111/laya-cuda
cd laya-cuda
uv sync
uv run laya-cuda doctor laya

Use

from laya_cuda import Engine

questions = {
    "team": {"type": "choice", "instructions": "Which team should handle this request?",
             "criteria": {"billing": "payments and refunds", "technical": "software errors"}},
    "urgent": {"type": "noul", "instructions": "Does this need urgent attention?"},
}
with Engine("laya") as model:
    print(model.predict("Please refund my duplicate payment.", questions)["answers"])

laya-cuda predict "Please refund my duplicate payment." -q questions.json
laya-cuda predict -i requests.jsonl -q questions.json -o results.jsonl

Answers keep the SDK's structure: probabilities, confidence, action probability and token usage. BatchEngine batches concurrent callers. backend="official" runs the unmodified SDK for comparison and needs the full or reference extra.

Learn more

• Guide: Engine, BatchEngine, model registry, platform and driver details, development, architecture.
• Examples: decision workflows, the Jev client and Snake Lab, a live demo where every move is a Laya prediction.
• Benchmarks: how comparisons, gates and reports work.
• FP32-reference validation: the full record of the numerical, long-input and accuracy gates.
• Latency tail: what made P95 twice P50, the fixes, and their check on untuned workloads.
• Adding a model: the adapter recipe.

The library is about 1,270 lines of Python and CUDA. CuPy owns memory, streams and CUDA Graphs; cuBLAS does the matrix multiplication; a few NVRTC kernels handle the rest, including one fused tensor-core attention kernel.

Acknowledgements

• First, thanks to Laya and its developers. Without Laya, this project would not exist.
• Thanks to the benchmarks: typed-decisions, AG News, DAIR Emotion, Banking77 and MASSIVE.
• Thanks to laya-mlx, which inspired this project.

License

Source code and derived request semantics are Apache-2.0; see LICENSE and laya_cuda/NOTICE. Model and dataset licenses apply separately.