Sakuradite

Structured decisions from open-weight language models, embedded in Python or served over HTTP. Sakuradite scores answer candidates with native model forwards; it is not a chat or text-generation server.

Questions return typed results:

• Noul: a probability for a true/false question.
• Choice: a selected key, candidate probabilities, and confidence.
• Score: an expected zero-based position in an ordered criteria list, candidate probabilities, a legend, and confidence.

Probabilities are normalized over the supplied candidates, not calibrated guarantees of correctness.

Install

Use Python 3.12+ on Linux or macOS. Install a published release with python -m pip install 'sakuradite[server,torch]', or install from a source checkout:

python -m pip install .                  # HTTP client and Typer CLI
python -m pip install '.[torch]'         # Embedded PyTorch / Transformers
python -m pip install '.[mlx]'           # Embedded MLX, Apple Silicon macOS only
python -m pip install '.[server,torch]'  # Uvicorn daemon with PyTorch

Choose the runtime you need; the base install includes neither model weights nor an inference runtime. Use .[server,mlx] for an MLX daemon. CUDA execution also needs a compatible PyTorch installation, GPU, and driver.

JSON uses orjson. Native inference requires Rust-backed fast tokenizers. Serving uses the package's ASGI application and Uvicorn, without FastAPI or SGLang in its request path. The optional MLX vision runtime installs its own upstream dependencies, which may include web-framework packages.

Embedded Python

Pass an actual Hugging Face repository ID or a local checkpoint directory. Hub models are downloaded on first use; access-controlled models require an existing authorized Hugging Face login or HF_TOKEN.

from sakuradite import Choice, Noul, Sakuradite, Score

with Sakuradite("Qwen/Qwen3-0.6B", backend="torch") as client:
    result = client.decide(
        state="The package arrived damaged and the customer needs a replacement.",
        questions={
            "replacement": Noul(instructions="Does the customer need a replacement?"),
            "route": Choice(criteria={"shipping": "Delivery issues", "billing": "Payments"}),
            "urgency": Score(criteria=["Low urgency", "Medium urgency", "High urgency"]),
        },
    )
    print(result.model_dump_json(indent=2))

state, instructions, and criteria entries accept text or structured JSON objects/lists. Reuse an open client for repeated requests: an embedded client owns one model and releases it when the context exits.

The same API is awaitable inside an event loop:

import asyncio

from sakuradite import Noul, Sakuradite


async def main():
    async with Sakuradite("Qwen/Qwen3-0.6B", backend="torch") as client:
        result = await client.decide(
            state="The delivery is late.",
            questions={"late": Noul(instructions="Is the delivery late?")},
        )
        print(result.nouls["late"].noul)


asyncio.run(main())

Use with in synchronous code and async with in asynchronous code. If you manage the lifecycle manually, call close() or await close() respectively. A closed client cannot be reopened. Closing an HTTP client does not unload models from the daemon.

For an image/text checkpoint on Apple Silicon:

from pathlib import Path

from sakuradite import Noul, Sakuradite

with Sakuradite("LiquidAI/LFM2-VL-450M", backend="mlx") as client:
    result = client.decide(
        state="Inspect the supplied photograph.",
        images=[Path("photo.png")],
        questions={"red": Noul(instructions="Is a red object visible?")},
    )
    print(result.nouls["red"].noul)

Daemon and CLI

Start the native Uvicorn server in one terminal:

sakuradite serve --host 127.0.0.1 --port 7777

In another terminal, inspect, download, and load a model:

sakuradite inspect Qwen/Qwen3-0.6B
sakuradite pull Qwen/Qwen3-0.6B
sakuradite load Qwen/Qwen3-0.6B --backend torch
sakuradite list
sakuradite ps

inspect reads metadata without downloading weights. pull downloads without loading; load downloads if necessary and retains the model in device memory. These commands connect to the running daemon. Use the global --base-url option before the command when it runs elsewhere.

Save this question map as questions.json:

{
  "late": {
    "type": "noul",
    "instructions": "Is the delivery late?"
  }
}

sakuradite run Qwen/Qwen3-0.6B --backend torch \
  --state "The delivery is late." --questions questions.json
sakuradite unload Qwen/Qwen3-0.6B
sakuradite rm Qwen/Qwen3-0.6B

run loads the model if necessary and prints a JSON decision result. Use --state-file for UTF-8 text or JSON, or pipe state through standard input. unload releases device memory; rm deletes an unloaded managed checkpoint. Bare sakuradite starts an interactive shell in a terminal. Commands are also available through python -m sakuradite; use --help for options.

HTTP client

Load the model on the daemon first, then use the same decision API:

from sakuradite import Noul, Sakuradite

with Sakuradite(
    "Qwen/Qwen3-0.6B",
    base_url="http://127.0.0.1:7777",
) as client:
    result = client.decide(
        state="The delivery is late.",
        questions={"late": Noul(instructions="Is the delivery late?")},
    )
    print(result.model_dump_json(indent=2))

Native HTTP requests use POST /v1/decisions:

curl --fail-with-body http://127.0.0.1:7777/v1/decisions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-0.6B","state":"The delivery is late.","questions":{"late":{"type":"noul","instructions":"Is the delivery late?"}}}'

GET /v1/models lists ready models. /healthz reports liveness; /readyz returns 503 until a model is ready. /docs serves the generated API reference; /openapi.json exposes its OpenAPI 3.1 contract. /v1/systemone accepts the text-only TypeSafe System One request contract; native image input uses /v1/decisions.

Non-loopback serving requires SAKURADITE_API_KEY; clients send it as a bearer token or set api_key=. Administrative endpoints use a separate SAKURADITE_ADMIN_KEY. On POSIX, by default, the daemon generates an owner-only admin.key in its home directory, read automatically by local clients using the default loopback URL and the same home. Other URLs need an explicit administration key. Use HTTPS through a trusted reverse proxy for remote use. Windows daemons require an explicit SAKURADITE_ADMIN_KEY; generate a strong random value and set it in the daemon and administrative clients. This remains separate from SAKURADITE_API_KEY.

Model load/download operations return an operation ID; poll with client.models.wait(operation) or GET /admin/v1/operations/{operation_id}. The daemon retains the latest 256 completed operations plus all pending/running operations. Older operation IDs return 404; loaded models remain resident. Persist results you need for auditing. Failed operations raise APIError with the corresponding error code and status, rather than always reporting 503. Model downloads requiring approved Hugging Face access use HF_TOKEN or hf auth login on the daemon host, not the inference or administration key.

API reference

With the default loopback daemon and no inference key configured, open http://127.0.0.1:7777/docs. The reference is generated from the same route declarations and Pydantic schemas used by the daemon, with grouped endpoints, request/response fields, text and image examples, all three answer primitives, status codes, model-operation polling, and SSE event examples. Expand a response or example and follow schema links for field descriptions and constraints.

The page is read-only semantic HTML: no JavaScript, external CDN, frontend build, or network assets are required. It works offline when saved. Installing the Python package includes the library, CLI, and daemon, not a desktop app or Node.js toolchain. Clients can consume /openapi.json directly; operation IDs are stable and the relative server URL supports reverse-proxy path prefixes.

When SAKURADITE_API_KEY is configured, both documentation endpoints require the inference bearer token, just like /v1/*. The separate admin token only authorizes /admin/*; viewing documentation does not grant administrative access. A plain browser visit cannot set the bearer header. Fetch and save the page with an authorized HTTP client instead:

curl --fail-with-body http://127.0.0.1:7777/docs \
  -H "Authorization: Bearer $SAKURADITE_API_KEY" -o sakuradite-api.html
curl --fail-with-body http://127.0.0.1:7777/openapi.json \
  -H "Authorization: Bearer $SAKURADITE_API_KEY" -o openapi.json

Save both files in the same directory to retain the raw-JSON link when reading offline. Never put tokens in URLs. Without an inference key, access remains restricted to loopback clients on a loopback-bound daemon. /healthz and /readyz remain unauthenticated; no cross-origin browser access is enabled.

Streaming decisions

Pass stream=True to receive each completed answer before the full request finishes. This is answer streaming, not token generation: the engine scores candidates in parallel, and answers become available as native batches finish. Output-token usage remains zero.

from sakuradite import Noul, Sakuradite

with Sakuradite("Qwen/Qwen3-0.6B", base_url="http://127.0.0.1:7777") as client:
    with client.decide(
        state="The delivery is late and the package is damaged.",
        questions={
            "late": Noul(instructions="Is the delivery late?"),
            "damaged": Noul(instructions="Is the package damaged?"),
        },
        stream=True,
    ) as stream:
        for event in stream:
            if event.type == "answer":
                print(event.key, event.answer)
        print(stream.result.model_dump_json(indent=2))

For asynchronous code, use async with await client.decide(..., stream=True) and async for event in stream. The same stream API works with an embedded model. A stream supports one consumer. Exhausting it closes its transport automatically; use its context manager when stopping early. Closing the client also closes its active streams. stream.result is available only after the complete event.

Over HTTP, send "stream": true to POST /v1/decisions and use a client that does not buffer the response:

curl --no-buffer --fail-with-body http://127.0.0.1:7777/v1/decisions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-0.6B","state":"The delivery is late.","questions":{"late":{"type":"noul"}},"stream":true}'

The response is text/event-stream. Each SSE frame has an event name and one JSON data payload whose type matches that name:

• answer: model, question key, typed answer, and that question's usage.
• complete: the complete result, identical in shape to a non-streaming response.
• error: an error with code and message, plus status_code; terminal.

Failures before the first answer retain their ordinary HTTP status and JSON error body. Failures after streaming starts use an error event; the Python SDK raises APIError with its code and status. A truncated or inconsistent stream raises TransportError, never a successful partial result. Streams are not automatically retried or resumable. Input-token usage counts logical question inputs, not the physical tokens saved by cache reuse.

Disable response buffering at the reverse proxy and allow the configured model deadline for upstream reads. The daemon sends X-Accel-Buffering: no, bounds pending events, cancels disconnected requests, and limits blocked stream writes to 30 seconds by default (ServerConfig.stream_write_timeout). Native work already running drains before its model or request cache is released. The TypeSafe-compatible /v1/systemone endpoint remains non-streaming.

Runtime limits and configuration

• PyTorch uses native Transformers loaders for text and compatible image/text checkpoints, on CPU, CUDA, or Apple MPS. MLX supports text on Apple Silicon and native SmolVLM/Idefics3 and LFM2-VL image/text checkpoints through MLX-VLM.
• Support depends on the checkpoint's native backend registry, fast tokenizer, chat template, and model processor. Custom repository code is not executed. Registry membership alone does not prove a checkpoint works; there is no claim of support for every architecture or quantization format.
• With only one runtime installed, select backend="torch" or backend="mlx" explicitly. Automatic selection prefers MLX for eligible text and image checkpoints on Apple Silicon and otherwise selects PyTorch.
• For compatible vision checkpoints, pass images=[Path("image.png")] to decide() after importing Path from pathlib, or repeat --image in the CLI. Up to eight JPEG/PNG images are accepted as Path objects, bytes, or base64 data URLs. Remote image URLs are not fetched.
• MLX image inference has been checked against native full forwards with HuggingFaceTB/SmolVLM-256M-Instruct and LiquidAI/LFM2-VL-450M, including multiple images and image-sensitive probabilities. Other MLX vision families are rejected explicitly; select PyTorch for those compatible checkpoints.
• Qwen3.5 is currently disabled in MLX-LM because release 0.31.3 has a known recurrent q/k normalization defect. Use the Torch backend instead; registry availability is not treated as proof of correct native math.
• revision= pins a Hub revision; an immutable commit is best for reproducibility. offline=True or CLI --offline requires locally available model assets. Device and dtype can be selected when opening or loading a model.

Runtime data, downloads, and the model catalog live in ~/.sakuradite by default. Override this with SAKURADITE_HOME, Python home=Path(...), or the global CLI --home option. The SDK reads defaults from config.toml there; explicit constructor values override those defaults. SAKURADITE_MODEL and SAKURADITE_BASE_URL can select a model and HTTP endpoint without hardcoding them; unset SAKURADITE_BASE_URL when using embedded inference.

On CPU-limited GPU servers, prevent image preprocessing from creating more CPU threads than the container can run. Set thread limits before starting Python:

OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 sakuradite serve

This limits CPU preprocessing threads, not CUDA parallelism. CPU-only inference can benefit from more threads. Latency depends on the model, input length, images, question count, concurrent callers, and available hardware; loading and the first request also incur initialization costs.

Request compilation, image preparation and inference share one model-owning worker. This keeps mutable native processor state serialized without competing preparation threads. Admission, deadlines and scoped cleanup cover the complete request, including compilation.

Questions and concurrent requests use bounded native batches. PyTorch selects only the needed output positions when the native model supports indexed logits, without replacing its vocabulary head, scaling, or soft-capping. If a tokenizer has no padding token, its existing end-of-sequence token is used for masked right padding; no vocabulary entries or model weights are added. Eligible native attention models share an identical text prefix across a request's native batches when enough duplicate computation can be removed. Eligibility depends on the actual cache, attention, rotary-position and quantization configuration, not the model's brand. Validated hybrid cache layouts include Qwen3.5 text, LFM2 and Nemotron-H. The Qwen3.5 multimodal wrapper can share prefixes for text-only requests; image prefixes are never shared. BF16 linear-attention/state-space hybrids keep unsplit native prefill because splitting can amplify low-precision rounding. Qwen3.5-family and Nemotron Nano shared-prefix measurements explicitly use dtype="float16". LongRoPE reuse requires the prefix and complete inputs to stay in the same frequency regime. Eligibility uses all matching questions in the request, not just the first batch. Immutable request-owned seeds are cloned for each branch and released after all native rows finish, even for slow stream consumers. They are never shared between requests. Retained seeds require additional memory; batch-token budgeting still uses complete input lengths. Other recurrent layouts, quantized caches, shared-KV layers and dynamically changing rotary frequencies do not use this optimization; native batching and supported output-position selection still apply. Native batch shapes, precision and fused kernels can change floating-point results; probabilities are not promised to be bitwise invariant across them. Repeated images are decoded once per request. Verified Idefics3/SmolVLM and LFM2-VL processors also reuse native resized/normalized pixels and image-token expansions for questions sharing the same ordered images. Homogeneous Torch rows retain native padding and independent output storage; heterogeneous image batches still use the complete native processor. Image data is never cached across requests and is released when the owning native work drains.

MLX text scoring uses each architecture's native prompt cache to prefill long inputs and evaluate only the final token's output head. Short prompts stay on a single forward to avoid extra dispatch overhead. This preserves native tied or quantized embeddings, recurrent state, head scaling and soft-capping. Compatible float32 MLX attention models additionally use native variable-length batches and request-wide prefix forks. The new optimizations are disabled for low-precision weights after measured probability drift; this is not a reason to assume float32 is faster overall. Recurrent/rotating caches keep their native execution path. MLX vision currently processes unpadded question rows sequentially and reuses native image features within a request; it does not claim general batched multimodal execution. Compatible float32 vision inputs of at least 512 processed tokens use native language-cache prefill and evaluate only the final output-head position. BF16/FP16 vision keeps the unsplit forward. Each row still receives its own token positions and masks while request-owned image preprocessing and vision features are reused.

An upstream registry entry is not a pretrained-checkpoint test. In particular, MoE "active parameter" counts are not weight-memory requirements: all experts must fit the selected native runtime's placement and quantization constraints. Smaller checkpoints are the practical starting point for local latency tuning.

CUDA runtime tuning

The validated Linux CUDA stack uses PyTorch 2.10.0 with CUDA 12.8 and Transformers 5.17.0. Install the CUDA wheels before the runtime extra:

python -m pip install torch==2.10.0 torchvision==0.25.0 torchaudio==2.10.0 \
  --index-url https://download.pytorch.org/whl/cu128
python -m pip install '.[torch,server]' 'transformers==5.17.0' \
  'kernels==0.16.2' 'fla-core==0.5.2'

Native Mamba checkpoints additionally benefit from mamba-ssm==2.3.2.post1 and causal-conv1d==1.7.0. Their compiled wheels must match Python, PyTorch, CUDA and the C++ ABI; follow the upstream projects' installation instructions. These CUDA extensions are not installed by the base package or the Torch extra. Missing native kernels can substantially increase latency.

The H200 measurements use a 32,768-token native batch budget. The default is 8,192 to suit smaller devices. Set a larger budget explicitly when loading a daemon model, rather than assuming the measured hardware configuration is the default:

from sakuradite import Sakuradite
from sakuradite.config import ModelConfig

with Sakuradite(base_url="http://127.0.0.1:7777") as client:
    operation = client.models.load(
        ModelConfig(
            model="LiquidAI/LFM2.5-2.6B",
            backend="torch",
            device="cuda",
            max_batch_tokens=32768,
        )
    )
    client.models.wait(operation)

A larger batch budget increases potential memory consumption and is not always faster. Model loading and kernel initialization are separate from warm-request latency; concurrent callers also include queueing time.

Tune the row limit alongside the token budget. For Qwen2.5-0.5B-Instruct in FP16 on an RTX 4090, max_batch_size=64 with max_batch_tokens=32768 gave the best tested mixed-workload completion time; 128 rows with 65,536 tokens was slower. This is workload-specific guidance, not a recommendation for every checkpoint. Measure individual request latency as well as whole-batch completion: larger batches can keep short requests waiting. Defaults remain 16 rows and 8,192 tokens.

Model weights are downloaded separately and remain subject to each model's license and access conditions.

License

Sakuradite is licensed under Apache-2.0; see LICENSE. This does not change the licenses or access requirements of any model weights.