hipEngine

hipEngine is a ROCm-native local inference engine built primarily for AMD Radeon GPUs. It pairs a small Python host with custom HIP kernels for torch-free model loading, generation, and OpenAI-compatible serving on supported hardware.

Current release: v0.6.1. This alpha fixes long-prompt prefix reuse, widens speculative decoding on INT8 KV, and adds /bench and prefix-cache accounting to the terminal chat client. See the release notes for scope and limitations.

Supported models lists the current model families and hardware; the model support reference gives exact formats, checkpoints, and limitations for each.

Why use hipEngine?

• Native AMD support. HIP-first kernels are written from scratch and tuned specifically for RDNA 3 (gfx1100) and Strix Halo RDNA 3.5 (gfx1151) instead of being CUDA ports.
• No PyTorch runtime required. There is no PyTorch dependency, which keeps hipEngine lightweight. Although it is packaged for Python, almost all of the hot path is C++.
• Optimized for agents and concurrent requests. Besides extensive tuning for fast single-request performance, hipEngine is specifically tuned for multiple concurrent request performance. The engine is built with continuous batching, prefix caching, and a shared KV pool.
• Drop-in support for existing clients. The included OpenAI-compatible server supports completion, chat, token-level SSE, logprobs, tools, structured-output validation, Qwen thinking controls, logprob-biased effort control, and request diagnostics. There is also a simple built-in chat interface.
• Rigorous correctness. All implementations are checked against a CPU-side oracle. The strict profile requires exact or parent-parity results, while production defaults must pass correctness gates. Optimizations or routes that fail these gates are rejected or made explicitly opt-in with measured costs stated.

hipEngine is a from-scratch project and does not inherit any unvetted code or legacy design. It is AGPL 3.0 licensed.

Supported models

• Language models: Qwen3.5/3.6/3.8 dense and mixture-of-experts models, Qwen3.8 Flash-Next, Laguna S 2.1, and Maple-Preview.
• Document understanding: Surya OCR 2 for full-page OCR and EVIE 4.5B / 8B for visual document retrieval.
• Speech: VibeVoice ASR for transcription (early support) and VibeVoice TTS for synthesis (experimental). Moonshine ASR has an internal runtime; public audio API support is planned.
• Time-series forecasting: TimesFM 2.5 and TimesFM 3.0.

See the model support reference for exact checkpoints, GGUF quantizations, ParoQuant, MLX, and safetensors formats, plus hardware and API limits. Qwen3.8-27B GGUF Q4_K_M is the dense model to start with on either AMD backend. Most other modalities are tested on Strix Halo; NVIDIA Blackwell (sm_120a) support is limited to Maple's Python API.

An independent survey of Qwen3.8-27B implementations on Strix Halo compares hipEngine against other engines on a single Framework Desktop host.

hipEngine includes DMS support and training code, with a published DMS checkpoint for Qwen3.8-27B Q4_K_M. The DMS analysis records the quality bar and the 8K–232K evidence ladder; measured capacity is in Long context on a 24 GB GPU below.

CPU model generation is not supported. The CPU backend is used for correctness tests. On NVIDIA, load Maple with backend="cuda_sm120a"; automatic hardware selection currently covers AMD only.

Choose the ParoQuant W4 checkpoint for the optimized Qwen3.6 35B-A3B path, or GGUF for the broader model and quantization ecosystem. See the quantization comparison for quality and speed trade-offs, including ROCmFP4/ROCmFPX.

Performance highlights

These are measured results, not estimates. Prompt processing is the speed of reading the input. Text generation is the speed of producing new tokens.

The benchmark summary below is synchronized from the benchmark report.

Tokens/s on each named host: prompt processing is input, text generation is output. MTP is speculative decoding in qualified scopes. Dashes are unmeasured; context limits are capacity tests.

Performance

Radeon Pro W7900 — 48 GB (gfx1100)

BF16 rows: September 20, epyc, 512/128 tokens, warmup + three runs. INT8: September 13. MTP/capacity not refreshed. Laguna: 4K prompts. 35B GGUF MTP is opt-in; 27B MTP is 1.63x its matched 24.36 tok/s AR, not either decode column.

Strix Halo / Radeon 8060S — 120 GB (gfx1151)

September 20 rows: Framework Desktop. Qwen3.8 Q4_K_M MTP uses three drafts and 25 output tokens per request: 1.98x its matched 11.30 tok/s AR, not the 512/128 decode column. This is a short-context measurement. 35B MTP is the July 19 opt-in result. Measurements.

Time-series forecasting (TimesFM 2.5 200M). hipEngine decodes batch=8, context 8192, horizon 512 forecasts in 0.082 s on the HP ZBook Strix Halo host (8.6x the official torch reference) and 0.062 s on a Framework Desktop host — the same GPU on two machines, so the gap is thermal headroom.

VibeVoice-ASR 9B is torch-free on Strix Halo. Q4_K_M beats bf16 — 1.52x prefill, 1.55x decode — at 6.1 vs 16.7 GB, WER 2.01% vs 2.81%; RTF 0.35. Results.

NVIDIA RTX PRO 6000 Blackwell — 96 GB (sm_120a)

Long context on a 24 GB GPU

Qwen3.8-27B Q4_K_M can hold up to 232K tokens on a 24 GB gfx1100 GPU using DMS, a trained KV eviction policy. In these tests, DMS INT8 agreed more closely with BF16 than the direct-INT8 route:

Direct-INT8 failed 9 of 11 quality prompts and is not a default. Capacity evidence

Prefix caching

Multi-turn conversations resend the whole transcript, so hipEngine reuses the KV pages and hybrid state of any 256-token-aligned prefix a later request repeats. Radix is the default for both the direct engine and HTTP serving; --prefix-cache off rolls it back. Different prefill routes can change generated tokens. Qwen3.6-35B-A3B UD-Q4_K_M on Strix Halo (gfx1151), 14 multi-turn lanes of three turns, reusing 42,496 of 87,582 prompt tokens:

Serving several requests at once

Aggregate tokens per second across all active requests, Qwen3.8-27B Q4_K_M on the W7900, September 4, 2026. Peers use F16 KV where hipEngine uses BF16.

On Strix Halo, Maple-Preview 2-bit scales to 214.788 tok/s across eight requests (123.131 at one, 202.038 at four). Where speculative decoding runs automatically in production it is scoped to a qualified shape: Qwen3.6-35B-A3B GGUF reaches 93.644 tok/s public — 1.1565x its own AR — at two concurrent requests on the W7900.

Full commands, software versions, model hashes, memory use, and correctness checks are in the benchmark report.

Coding-Agent Session Replay

Replaying a recorded coding session shows the benefit of avoiding repeated prompt processing: 32 sequential requests, 128 output tokens per request, and a cumulative transcript growing from 740 to 13,762 tokens. Qwen3.6-35B-A3B UD_Q4_K_M on the HP ZBook / Radeon 8060S (gfx1151), using the in-process engine and server chat renderer:

2.79x end-to-end throughput, with 91% of prompt tokens reused rather than recomputed. This is one adapted transcript and one run per setting, not a general agent-quality benchmark; outputs were not compared across arms. System/tool definitions are substituted and long tool results truncated. The 16-snapshot setting uses more memory and is opt-in. Results and retention guidance.

Status and limits

v0.6.1 is alpha. Automatic performance routes remain scoped to qualified model, hardware, and workload combinations:

• Qwen3.6-27B and Qwen3.8-27B GGUF generation and serving on both AMD backends. Both can speculate with the model's own multi-token prediction head, as can Qwen3.6-35B-A3B.
• The server enables speculative decoding for supported model, GPU, storage, and request shapes, labels unmeasured configurations, reports why when it skips speculation, and can be switched off for all new requests by one endpoint call.
• An optional execution-profile selector (strict, production, batch_invariant) that runs a registered kernel plan, checks that its fallbacks are installed, and rejects a combination hipEngine has not been shown to complete.
• Scheduling and memory defaults: the fair prefill/decode policy, a smaller per-process GPU memory reserve on Radeon RDNA 3, and FP16 recurrent state for Qwen3.8 Q4_K_S on Strix Halo.
• Still supported: Qwen3.5/3.6 GGUF and ParoQuant, Laguna S 2.1, Maple-Preview, several requests at once on one resident model, and OpenAI-compatible streaming, sampling, tools, structured-output validation, and cancellation.

Full user-facing change history is in the changelog.

Important limits:

• hipEngine uses one GPU. Multi-GPU inference is not yet implemented.
• There is no desktop GUI, model catalog, or automatic model download.
• CPU model inference is not implemented.
• The concurrency memory figures come from a 48 GB W7900. Single-request context on a 24 GB card is qualified to 232,448 tokens on the DMS and opt-in direct-INT8 routes; concurrent-request shapes on 24 GB are not qualified yet, so keep a conservative context limit there.
• Automatic speculative decoding is bounded by implementation capability and available memory. Qwen3.8 Q4_K_M on gfx1151 supports sampled MTP with BF16 KV for one through four active requests. Context is bounded by the target's allocated capacity. Packed INT8 and compact-DMS MTP are not implemented. See Server API for admission and fallback behavior.
• APIs and supported combinations can still change before 1.0.

Hardware detection

backend="auto" recognizes gfx1100 and gfx1151. These cover the tested Radeon Pro W7900 and Ryzen AI MAX+ 395 / Radeon 8060S systems. Other AMD architecture numbers are not automatically treated as compatible.

You can force a nearby backend, but do so only after checking output quality and performance. hipEngine will not silently use PyTorch when a GPU is unsupported.

Installation

Requirements

ROCm 7.x is the safest choice for the current wheel (ROCm 10.0 has been tested and works fine as well). See the TheRock setup guide for retained ROCm 7.13 and gfx1151 ROCm 10 setup/JIT validation. The first model load compiles and caches kernels, so it takes longer than later starts.

Install from PyPI:

pip install hipengine huggingface_hub

Or install a source checkout:

git clone https://github.com/shisa-ai/hipEngine.git
cd hipEngine
git lfs install
git lfs pull
pip install -e .

Confirm that the command is available:

hipengine --help
hipengine serve --help

Start a local server

hipEngine does not download model weights during startup. Download a supported model first, or use a GGUF file that is already on disk.

For the ParoQuant Qwen checkpoint:

hf download shisa-ai/Qwen3.6-35B-A3B-PARO-packed

hipengine serve \
  --model shisa-ai/Qwen3.6-35B-A3B-PARO-packed \
  --served-model-name qwen-paro

For GGUF, pass the path to the model file:

hipengine serve \
  --model /path/to/Qwen3.6-35B-A3B-Q4_K_M.gguf \
  --served-model-name qwen

The server listens on http://127.0.0.1:8000 by default. Test it with:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen",
    "messages": [{"role": "user", "content": "Why is the sky blue?"}],
    "max_tokens": 128
  }'

Point any client that accepts a custom OpenAI base URL at http://127.0.0.1:8000/v1. See the server guide for API keys, streaming, tools, structured output, and model capability checks.

Chat in your terminal

With the server running, open another terminal:

pip install 'hipengine[chat]'
hipengine chat

The client connects to http://127.0.0.1:8000 and discovers the served model. No model path is needed, and it does not start another server. For a different address, use hipengine chat --server http://127.0.0.1:8001.

Replies stream as Markdown, with optional reasoning display and per-turn stats (prompt prefill rate, decode rate, and tokens served from the prefix cache). Use /status for server limits, /usage for conversation token counts, /bench for a quick prefill, decode, and prefix-cache measurement of the running server (with and without speculative MTP), /think off to disable reasoning, /retry to regenerate, and /clear to start over. /help lists all commands; /quit, Ctrl-C, or Ctrl-D exits. Use hipengine chat --plain for plain-text output.

Use the Python API

from hipengine import LLM, SamplingParams

llm = LLM("shisa-ai/Qwen3.6-35B-A3B-PARO-packed")
outputs = llm.generate(
    ["Hello, hipEngine."],
    SamplingParams(max_tokens=64, temperature=0.0),
)
print(outputs[0])
llm.close()

LLM(...) detects a supported AMD GPU and chooses the model format automatically. You can also pass a local GGUF or Maple path. Advanced users can override the choice with backend= and quant=. The execution_profile="strict"|"production"|"batch_invariant" selector is fail-closed to registered kernel plans with exact fallbacks; omitting it selects production for models with a certified plan and keeps the previous behaviour otherwise.

Documentation

User guides

Development and benchmark details

Project lineage

hipEngine is an independent project that builds on ideas and software from ROCm, HIP, Nano-vLLM, ParoQuant, FastDMS, llama.cpp, and other open-source projects. See the source and model guides for detailed attribution.

Thanks

Special thanks to Framework and AMD for providing Strix Halo test hardware.

License

hipEngine source code is licensed under AGPL-3.0-or-later. Model weights, checkpoints, and external datasets remain under their own licenses.