Kestrel

High-performance inference engine for multimodal models.

Kestrel is the inference engine behind Photon, Moondream's on-device deployment option. Most Moondream users should install via pip install moondream; this repository provides the engine directly and supports additional model families.

Kestrel provides async, micro-batched inference with streaming support, paged KV caching, and optimized CUDA and Metal kernels. It's designed for production deployments where throughput and latency matter.

Features

• Async micro-batching — Cooperative scheduler batches heterogeneous requests without compromising per-request latency
• Streaming — Real-time token and transcription progress
• Multi-task — Vision-language generation, spatial reasoning, transcription, and speech synthesis
• Paged KV cache — Efficient memory management for high concurrency
• Prefix caching — Radix tree-based caching for repeated prompts and images
• LoRA adapters — Parameter-efficient fine-tuning support with automatic cloud loading

Requirements

• Python 3.10–3.14.
• One of:
  • NVIDIA GPU on Linux x86_64 / aarch64 or Windows x86_64. Optimized kernels for SM80 (A100), SM86 (A10, RTX 30-series), SM87 (Jetson Orin), SM89 (L4, L40S, RTX 4090), SM90 (H100, H200, GH200), SM100 (B200), SM110 (Jetson Thor), SM120 (RTX PRO 6000). Other CUDA GPUs may work but have not been tested.
  • Apple Silicon Mac (M-series) on macOS 13 (Ventura) or later, with native Metal kernels.
• MOONDREAM_API_KEY (optional) — only needed for Moondream finetuned-model inference (get a key from moondream.ai)

Installation

pip install kestrel

For Jetson Orin (JetPack 6) or Jetson Thor (JetPack 7), see the Jetson setup guide.

Model Access

Kestrel supports these model families:

Quick Start

import asyncio

from kestrel.config import RuntimeConfig
from kestrel.engine import InferenceEngine


async def main():
    # Weights are automatically downloaded from HuggingFace on first run.
    # Use a registered model name or Hugging Face repository ID.
    cfg = RuntimeConfig(model="google/gemma-4-E2B-it")

    # Create the engine (loads model and warms up). No API key needed for
    # local inference; pass api_key="..." only for finetuned models.
    engine = await InferenceEngine.create(cfg)

    # Load an image (JPEG, PNG, or WebP bytes)
    image = open("photo.jpg", "rb").read()

    # Visual question answering
    result = await engine.query(
        image=image,
        question="What's in this image?",
        settings={"temperature": 0.2, "max_tokens": 512},
    )
    print(result.output["answer"])

    # Clean up
    await engine.shutdown()


asyncio.run(main())

Image embeddings

dinov2-small serves the single-pass embed task. Its output contains FP32 last_hidden_state ([1, 257, 384]) and pooler_output ([1, 384]) tensors. The complete encoder loads from kestrel-kernels on supported Hopper devices with BF16 inputs. Unsupported devices, precisions, or missing artifacts fail at startup.

import asyncio
from PIL import Image
from kestrel.config import RuntimeConfig
from kestrel.engine import InferenceEngine


async def main():
    engine = await InferenceEngine.create(RuntimeConfig(model="dinov2-small"))
    try:
        result = await engine.model().embed(image=Image.open("photo.jpg"))
        print(result.output["pooler_output"].shape)
    finally:
        await engine.shutdown()


asyncio.run(main())

DINOv2 image accepts a PIL image or a NumPy array. Integer pixels use [0, 255]; floating-point pixels use [0, 1]. Preprocessed inputs can instead be passed as pixel_values with shape [1, 3, 224, 224].

Supplied GPU pixel_values must already be contiguous BF16 on the model device. CPU preprocessing casts before transfer.

For either vision model, CUDA tensor inputs must be ready on the caller's current stream. Keep them unchanged until the awaited request completes; the engine handles the stream handoff internally.

Object detection with RF-DETR

RF-DETR runs through the single-pass detection API on Hopper with BF16:

import asyncio
from PIL import Image
from kestrel.config import RuntimeConfig
from kestrel.engine import InferenceEngine

async def main():
    engine = await InferenceEngine.create(RuntimeConfig(model="rfdetr-nano"))
    try:
        result = await engine.model().detect(
            image=Image.open("photo.jpg"), threshold=0.5, max_objects=20
        )
        print(result.output["objects"])
    finally:
        await engine.shutdown()


asyncio.run(main())

Available model names are rfdetr-nano, rfdetr-small, rfdetr-medium, rfdetr-base, rfdetr-large, rfdetr-xlarge, and rfdetr-2xlarge. Large means the current single-P4 model, not DeprecatedLarge. Each variant uses its checkpoint's fixed resolution and batch size one. Unsupported devices or precisions fail at startup; this route requires the corresponding artifacts in kestrel-kernels.

image accepts RGB PIL images, encoded image bytes, HWC NumPy arrays, or CPU CHW tensors. Floating-point pixels must be in [0, 1]. Alternatively, pixel_values accepts preprocessed [1, 3, resolution, resolution] tensors. GPU inputs must already be contiguous BF16 on the model device. Resizing and ImageNet normalization run on CPU before the input transfer; the detector itself uses one GPU launch.

Each object contains normalized x_min, y_min, x_max, y_max, score, class_id, and label. Classes use the original sparse COCO IDs, including 90 for toothbrush; unused IDs have an empty label. This is fixed-vocabulary detection, not an open-vocabulary object= prompt. threshold defaults to 0.5; max_objects caps results at 300. There is no NMS.

Use model_path to supply a checkpoint. Nano through Large download the released COCO checkpoint when no path is provided. XL and 2XL currently require an explicit path to their released checkpoint. Custom class vocabularies and segmentation checkpoints are not supported by these declared executables.

Speech transcription

Whisper, Qwen3-ASR, and Parakeet use the same model-bound transcribe capability. Choose a Hugging Face repository ID from the model table; Kestrel downloads its weights and selects that model's optimized runtime.

import asyncio
from pathlib import Path

from kestrel.config import RuntimeConfig
from kestrel.engine import InferenceEngine

MODEL = "Qwen/Qwen3-ASR-0.6B"


async def main():
    engine = await InferenceEngine.create(
        RuntimeConfig(
            model=MODEL,
            max_batch_size=4,
        )
    )
    model = engine.model(MODEL)
    try:
        result = await model.transcribe(
            audio=Path("meeting.m4a"),
            timestamps="word",
        )
        print(result.output["text"])
        for segment in result.output["segments"]:
            for word in segment.get("words", []):
                print(
                    f"{word['start']:7.2f}  {word['end']:7.2f}  {word['word']}"
                )
    finally:
        await engine.shutdown()


asyncio.run(main())

Kestrel accepts encoded paths, bytes, bounded binary streams, raw mono PCM, and asynchronous PCM iterators. Long paths are decoded incrementally. moondream/parakeet-redux also runs on the CPU and on Apple silicon: pass device="cpu" or device="mps". See Speech-to-text models for the shared interface and model capability matrix, model-specific options, formats, and resource limits.

For speech synthesis with Qwen3-TTS or Kokoro, see Text-to-speech models.

Tasks

Kestrel supports several vision-language tasks through dedicated methods on the engine.

Query (Visual Q&A)

Ask questions about an image:

result = await engine.query(
    image=image,
    question="How many people are in this photo?",
    settings={
        "temperature": 0.2,  # Lower = more deterministic
        "top_p": 0.9,
        "max_tokens": 512,
    },
)
print(result.output["answer"])

Caption

Generate image descriptions:

result = await engine.caption(
    image,
    length="normal",  # "short", "normal", or "long"
    settings={"temperature": 0.2, "max_tokens": 512},
)
print(result.output["caption"])

Point

Locate objects as normalized (x, y) coordinates:

result = await engine.point(image, "person")
print(result.output["points"])
# [{"x": 0.5, "y": 0.3}, {"x": 0.8, "y": 0.4}]

Coordinates are normalized to [0, 1] where (0, 0) is top-left. Point prompts can also include normalized spatial references:

result = await engine.point(
    image,
    "gaze",
    spatial_refs=[[0.42, 0.18]],  # e.g. the subject's head or eye location
)

Detect

Detect objects as bounding boxes:

result = await engine.detect(
    image,
    "car",
    settings={"max_objects": 10},
)
print(result.output["objects"])
# [{"x_min": 0.1, "y_min": 0.2, "x_max": 0.5, "y_max": 0.6}, ...]

Bounding box coordinates are normalized to [0, 1].

Segment

Generate a segmentation mask (Moondream 3 only):

result = await engine.segment(image, "dog")
seg = result.output["segments"][0]
print(seg["svg_path"])  # SVG path data for the mask
print(seg["bbox"])      # {"x_min": ..., "y_min": ..., "x_max": ..., "y_max": ...}

Note: Segmentation requires Moondream 3 and separate model weights. Contact moondream.ai for access.

Streaming

For longer responses, you can stream tokens as they're generated:

image = open("photo.jpg", "rb").read()

stream = await engine.query(
    image=image,
    question="Describe this scene in detail.",
    stream=True,
    settings={"max_tokens": 1024},
)

# Print tokens as they arrive
async for chunk in stream:
    print(chunk.text, end="", flush=True)

# Get the final result with metrics
result = await stream.result()
print(f"\n\nGenerated {result.metrics.output_tokens} tokens")

Streaming is supported for query and caption methods.

Response Format

All methods return an EngineResult with these fields:

result.output          # Dict with task-specific output ("answer", "caption", "points", etc.)
result.finish_reason   # "stop" (natural end) or "length" (hit max_tokens)
result.metrics         # Timing and token counts

The metrics object contains:

result.metrics.input_tokens     # Number of input tokens (including image)
result.metrics.output_tokens    # Number of generated tokens
result.metrics.prefill_time_ms  # Time to process input
result.metrics.decode_time_ms   # Time to generate output
result.metrics.ttft_ms          # Time to first token

Using Finetunes

If you've created a finetuned model through the Moondream API, you can use it by passing the adapter ID:

result = await engine.query(
    image=image,
    question="What's in this image?",
    settings={"adapter": "01J5Z3NDEKTSV4RRFFQ69G5FAV@1000"},
)

The adapter ID format is {finetune_id}@{step} where:

• finetune_id is the ID of your finetune job
• step is the training step/checkpoint to use

Adapters are automatically downloaded and cached on first use.

Configuration

RuntimeConfig

RuntimeConfig(
    model="moondream3-preview",  # or "moondream2" / "moondream3.1-9B-A2B"
    max_batch_size=4,            # Max concurrent requests
    decode_path="auto",          # "auto", "native", or fail-closed "generated"
)

decode_path="native" disables generated decode construction on runtimes that provide a native decode path. Qwen 3.5/3.6 requires compatible bundled generated programs for both "auto" and "generated", covering every active batch size up to max_batch_size; construction or decode fails instead of falling back. Gemma 4 applies the same fail-closed coverage rule when "generated" is selected. Moondream currently supports only the default "auto" policy.

To run from local files instead of the registered HuggingFace weights or tokenizer, keep model set to the matching registered architecture and pass local paths:

RuntimeConfig(
    model="moondream3.1-9B-A2B",
    model_path="/models/moondream/model.safetensors",
    tokenizer_path="/models/moondream/tokenizer.json",
)

model_path points to the local checkpoint file and skips the automatic HuggingFace weight download. tokenizer_path is optional and only applies to models whose runtime uses a tokenizer; tokenizer-free models do not need one. When provided, it can point directly to a tokenizer.json file or to a directory containing tokenizer.json. When omitted, Kestrel uses the tokenizer declared by the registered model. Local files must match the selected model architecture and checkpoint format.

Environment Variables

Triton Inference Server

Kestrel can be deployed as a Triton Inference Server backend. See the Triton setup guide.

Benchmarks

Throughput and latency for the query skill are tracked in PERFORMANCE.md, with results broken out by GPU.

Telemetry

Kestrel reports basic usage telemetry to help us decide which hardware platforms to prioritize for support and optimization. Each report includes the model in use, your GPU type and memory, aggregate request/error and token counts, your machine's hostname, and timestamps. Prompts, images, and model outputs are never sent.

License

Local inference is free and requires no API key. Finetuned-model inference requires a Moondream API key — see moondream.ai/pricing.