vLLM MBLT

vllm-mblt is an out-of-tree vLLM plugin that integrates Mobilint NPU runtime support into the vLLM serving and benchmarking stack.

It provides a custom vLLM platform, worker, and model registry hooks so Mobilint-optimized LLM/VLM artifacts can be served through familiar vLLM commands and OpenAI-compatible APIs.

Highlights

• Out-of-tree vLLM plugin: registers the mblt platform without patching vLLM itself.
• Mobilint NPU worker: dispatches text-generation and multimodal execution to Mobilint runtime models.
• Model registry integration: supports Mobilint wrappers for Llama, HyperCLOVAX, EXAONE/EXAONE4, Qwen2/3, and Qwen2/3-VL families.
• Runtime-aware scheduling: reads model-configured npu_prefill_chunk_size and max_batch_size values to tune chunked prefill and scheduler concurrency automatically.
• vLLM benchmark compatibility: works with vllm serve, vllm bench serve, and vllm bench throughput.

Requirements

• Python 3.10+
• vllm==0.11.2
• mblt-model-zoo[transformers] >= 2.1.0
• A Mobilint NPU environment. If you are not yet a Mobilint customer, please contact tech-support@mobilint.com.

The package pins vLLM for compatibility:

vllm>=0.11.2,<=0.11.2

Installation

Install from PyPI:

pip install vllm-mblt

Or install the latest source checkout:

git clone https://github.com/mobilint/vllm-mblt.git
cd vllm-mblt
python -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -e .

Quick Start

1. Verify Plugin Registration

After installation, run:

vllm --help

You should see plugin logs indicating that the Mobilint mblt platform plugin has been discovered and activated.

2. Serve a Text Model

vllm serve mobilint/Llama-3.2-1B-Instruct --trust-remote-code

Then query the OpenAI-compatible endpoint:

curl http://127.0.0.1:8000/v1/models

3. Serve a VLM Model

Qwen2-VL and Qwen3-VL Mobilint models can be loaded through the same vLLM server path:

vllm serve mobilint/Qwen2-VL-2B-Instruct --trust-remote-code

vllm serve mobilint/Qwen3-VL-2B-Instruct --trust-remote-code

Current Mobilint Qwen2/3-VL notes:

• The worker loads VLMs through AutoModelForImageTextToText.
• Image inputs are processed through vLLM's multimodal pipeline and merged into Mobilint language-model prompt embeddings inside the custom worker.
• The NPU path currently supports exactly one image in the initial multimodal request.
• Subsequent turns in the same session must be text-only or reuse the same image-token position.
• Video inputs are not supported by the current Mobilint Qwen2/3-VL NPU path.

Runtime Tuning

Runtime Layout Overrides

By default, vllm-mblt follows the runtime layout encoded in the Mobilint model artifact/config. Use --model-loader-extra-config only when you intentionally want to override runtime placement or testing knobs.

Runtime settings such as dev_no, target_cores, target_clusters, core_mode, and max_batch_size can be provided through --model-loader-extra-config. For detailed core_mode and multicore runtime layout guidance, see the Mobilint multicore documentation.

vllm serve mobilint/Llama-3.2-1B-Instruct \
  --trust-remote-code \
  --model-loader-extra-config '{"dev_no": 0, "target_cores": ["1:0"]}'

For VLMs such as mobilint/Qwen3-VL-2B-Instruct, shared runtime layout keys are applied to both Mobilint submodules by forwarding them as model-zoo VLM subconfig keys (vision_* and text_*). Use explicit prefixed keys when the vision encoder and text model need different placement:

vllm serve mobilint/Qwen3-VL-2B-Instruct \
  --trust-remote-code \
  --model-loader-extra-config '{"dev_no": 0, "core_mode": "global4", "vision_core_mode": "single"}'

For VLM-specific MXQ path overrides, use vision_mxq_path and/or text_mxq_path; a single top-level mxq_path is only meaningful for single-module text models.

Chunked Prefill Auto-Tuning

If a model config includes npu_prefill_chunk_size, vllm-mblt uses it to tune vLLM chunked prefill.

• Integer values are used directly.
• Dict values are selected by core_mode.
• core_mode is resolved from --model-loader-extra-config first, then from the model config default.
• The selected value is applied to vLLM's max_num_batched_tokens for chunked prefill.
• A default (or DEFAULT) key is used when the resolved core_mode has no entry.
• core_mode: "auto" is never a dict key. When the resolved core_mode is "auto" (or is absent) and the dict holds exactly one usable entry, that entry is used — a global-scheme batch mxq ships only the mode it was compiled for. Multiple entries under "auto" are not guessed.
• If no matching value is found, vllm-mblt falls back to 128.
• For batch-compiled models with max_batch_size > 1, the effective chunked prefill limit is clamped to 128 to match the qbruntime batch execution limit used by the worker.

Example model config:

{
  "npu_prefill_chunk_size": {
    "single": 64,
    "global4": 256,
    "global8": 512
  }
}

With this command, vllm-mblt selects 256 for global4:

vllm serve mobilint/YourModel \
  --trust-remote-code \
  --model-loader-extra-config '{"dev_no": 0, "core_mode": "global4", "target_clusters": [0]}'

If you also pass --max-num-batched-tokens, the effective value becomes the smaller of the user-provided value and the model-configured npu_prefill_chunk_size.

Use --block-size only when you intentionally want to override the model-configured/default block size:

vllm serve mobilint/Llama-3.2-1B-Instruct \
  --trust-remote-code \
  --block-size 64

Model-Configured Batch Capacity

If a model config includes max_batch_size, vllm-mblt uses that value to support batch-compiled Mobilint models.

• The worker uses max_batch_size for KV cache memory sizing.
• The platform applies it to vLLM max_num_seqs automatically.
• You do not need to pass --max-num-seqs unless you intentionally want a smaller scheduler cap.
• max_batch_size also supports the same core_mode keyed dict form as npu_prefill_chunk_size.
• For local testing, --model-loader-extra-config '{"max_batch_size": 32}' overrides the model config value.

Example:

vllm serve mobilint/Llama-3.2-1B-Instruct-Batch32 --trust-remote-code

For batch-compiled MXQs such as mobilint/Llama-3.2-1B-Instruct-Batch32, the plugin also caps the effective chunked prefill limit to 128, even when the model config advertises a larger npu_prefill_chunk_size.

Benchmarking

This repository includes sonnet.txt, which can be used with vLLM benchmark commands.

Serve Benchmark

Terminal 1:

vllm serve --model mobilint/Llama-3.2-1B-Instruct --trust-remote-code

Terminal 2:

vllm bench serve --model mobilint/Llama-3.2-1B-Instruct \
  --trust-remote-code \
  --port 8000 \
  --num-warmups 1 \
  --dataset-name sonnet \
  --dataset-path sonnet.txt \
  --num-prompts 10

Throughput Benchmark

vllm bench throughput --model mobilint/Llama-3.2-1B-Instruct \
  --trust-remote-code \
  --dataset-name sonnet \
  --dataset-path sonnet.txt \
  --num-prompts 10

Notes:

• vllm bench serve uses a separate server process; vllm bench throughput runs the engine directly.
• vllm bench serve --max-concurrency is a benchmark client load setting, not the server-side scheduler limit.
• Reported latency and throughput are environment-dependent. Capture results from your target board for documentation or performance comparisons.

NPU Event Tracing

The worker can record NPU activity as a Chrome Tracing log through qbruntime's event tracer, so you can see where time actually goes on the accelerator. It is wired into vLLM's standard worker profiler hook, which means the usual controls apply -- there is no MBLT-specific flag or endpoint.

Set VLLM_TORCH_PROFILER_DIR to a directory before starting the server. This is the switch: without it the OpenAI server does not register the profile routes, and the worker reports that tracing is not enabled.

export VLLM_TORCH_PROFILER_DIR=/tmp/mblt_traces
vllm serve mobilint/Llama-3.2-1B-Instruct --trust-remote-code

Then bracket the work you care about:

curl -X POST http://localhost:8000/start_profile
# send the requests you want to trace
curl -X POST http://localhost:8000/stop_profile

For an offline run, LLM.start_profile() / LLM.stop_profile() do the same, and vllm bench serve --profile brackets the benchmark for you.

Each window writes {hostname}_{pid}.mblt_npu_rank{rank}.{time_ns}.json into that directory. Open it at https://ui.perfetto.dev/. The name follows the convention torch.profiler.tensorboard_trace_handler uses for its own traces, which vLLM and vllm-ascend both build on: the nanosecond timestamp is what keeps successive windows from clashing, the pid separates concurrent processes on a host, and the hostname separates containers that share a mounted trace directory but not a pid namespace. vLLM's front-end trace lands beside it as {hostname}_{pid}.async_llm.{time_ns}.pt.trace.json.gz. The events are the runtime's own device-level spans -- infer, run npu, copy to npu, lock core, read device and the like -- so a window shows what each inference step spent on the accelerator.

vLLM writes its own front-end CPU trace (*.async_llm.*.pt.trace.json.gz) into the same directory, because VLLM_TORCH_PROFILER_DIR also enables the API server's AsyncLLM profiler. The two are complementary: that file covers CPU-side scheduling, the MBLT file covers NPU execution.

Notes:

• Trace a short window. qbruntime buffers the whole log in the process and writes it only when tracing stops, so leaving a trace on for the life of a server grows memory and produces a file too large to be useful. If the worker shuts down while a trace is running it is stopped first so the window is not lost.
• Only one qbruntime trace can record per process. A start while one is already recording is refused with a warning rather than cutting the first window short. qbruntime has no way to report whether a trace is running or who owns it, so this covers traces started through this plugin and, when its module is already imported, through mblt_model_zoo's benchmark helpers. A trace started by any other client cannot be detected.
• If a trace cannot be started, or its log cannot be written, /start_profile and /stop_profile fail rather than reporting success for a trace that will not be on disk. A repeated start or a stop with nothing running is not a failure and only logs. During shutdown a trace that cannot be written is logged and skipped so the rest of the teardown still runs.
• A /stop_profile with no trace running answers 500. That comes from vLLM's front-end AsyncLLM profiler, which raises when stopped before it was started; the worker-side trace is unaffected and only logs that there was nothing to stop.
• A torch profiler on the worker would show nothing useful here: the model runs on the NPU through qbruntime rather than through torch ops, which is why this hook records a qbruntime trace instead.

Supported Model Families

vllm-mblt registers Mobilint model wrappers for:

Model artifacts are available through Mobilint model repositories such as the Mobilint Hugging Face Hub.

Cache Behavior

MbltWorker uses snapshot-based KV cache reuse with these policies:

• Event-driven dump, not every step.
• Reuse live cache for same-request continuous decode.
• Keep finished-session snapshots for prefix reuse.
• Evict finished snapshots with an LRU cap of 16 sessions.
• Load matched snapshots only when the worker-side cost model expects the one-cache-id load to beat recomputing the matched prefix.

The prefix-cache load threshold is enabled by default. During model warmup the worker tries to measure optimistic one-cache-id prefill costs for 1, 2, 4, and 8 KV blocks; for batch MXQs this still submits exactly one active cache_id. Real snapshot loads and dumps update per-cache_id EWMA timings. A snapshot is loaded only when load_ms < prefill_ms * 0.9; otherwise the snapshot remains stored and the prompt is recomputed. The policy can be adjusted with environment variables or equivalent model loader extra config keys:

• VLLM_MBLT_PREFIX_CACHE_AUTO_THRESHOLD / prefix_cache_auto_threshold defaults to enabled. Set 0 to disable measured thresholding.
• VLLM_MBLT_PREFIX_CACHE_MIN_HIT_TOKENS / prefix_cache_min_hit_tokens sets a manual minimum matched-token count before any snapshot load.
• VLLM_MBLT_PREFIX_CACHE_LOAD_MARGIN / prefix_cache_load_margin defaults to 0.9.
• VLLM_MBLT_PREFIX_CACHE_CALIBRATE defaults to enabled. Set 0 to skip startup prefill calibration.

The worker also tracks how many tokens each live runtime cache (or batch cache_id slot) actually holds. A request may continue from a live cache without reloading only when that count matches the scheduler's num_computed_tokens. On a mismatch the worker logs a warning naming the request and cache_id, drops the ownership claim, and rebuilds the prefix instead of decoding against another sequence's KV. Seeing MBLT runtime cache token count holds fewer tokens than the scheduler expects in the log means measurements taken from that server should be re-checked.

Sampling Penalties

frequency_penalty, presence_penalty, and repetition_penalty are applied by default on the CPU-hosted MBLT sampler. Set VLLM_MBLT_ENABLE_SAMPLING_PENALTIES=0 to ignore them instead; the worker then logs once which penalties it dropped. Released MBLT packages ship repetition_penalty in generation_config.json, so ignoring them makes NPU output generated under different sampling than a GPU reference run.

VLM prefix caching currently covers the language-model KV cache only. The worker may load a compatible LM KV prefix snapshot and run only the uncached text/embedding suffix. Image/video feature extraction is not cached by this layer: image requests still rebuild vision features through the model's multimodal feature hooks before the LM prefill/decode step.

Batch-compiled VLM text backends (max_batch_size > 1) are supported for Mobilint Qwen2-VL and Qwen3-VL model types. With mblt-model-zoo>=2.3.0, Qwen3-VL dynamic-vision Batch16 artifacts such as mobilint/Qwen3-VL-8B-Instruct-Batch16 forward packed text embeddings plus the matching packed RoPE and deepstack tensors to the 3-input text MXQ. Unsupported multimodal model types fail before runtime inference with a clear error.

Implementation file: vllm_mblt/mblt_worker.py

Tests

python -m pytest tests

Project Structure

vllm_mblt/
├── __init__.py                 # vLLM plugin and model registration entry points
├── mblt_platform.py            # platform config overrides and runtime-aware defaults
├── mblt_worker.py              # custom worker, prefill/decode flow, KV snapshot logic
├── tracing.py                  # qbruntime NPU event tracing behind vLLM's profiler hook
└── models/                     # Mobilint model wrappers for LLM/VLM families

tests/
├── test_kv_cache_swap_spec.py
├── test_mblt_platform_prefill.py
├── test_mblt_tracing.py
└── test_mblt_worker_optimizations.py