hipEngine
hipEngine is a ROCm-native local inference engine built primarily for AMD Radeon GPUs. It pairs a small Python host with custom HIP kernels for torch-free model loading, generation, and OpenAI-compatible serving on supported hardware.
Current release: v0.8.1. Fixes GGUF serving startup and memory sizing on 24 GB GPUs, and reduces compact-DMS prefill and speculative rollback overhead. See the release notes for scope and limitations.
Why use hipEngine?
- Native AMD support. HIP-first kernels are written from scratch and tuned specifically for RDNA 3 (gfx1100) and Strix Halo RDNA 3.5 (gfx1151) instead of being CUDA ports.
- No PyTorch runtime required. There is no PyTorch dependency, which keeps hipEngine lightweight.
- Optimized for agents and concurrent requests. hipEngine is tuned for both single-request latency and multiple concurrent requests, with continuous batching, prefix caching, and a shared KV pool.
- Drop-in support for existing clients. The included OpenAI-compatible server
supports completion, chat, token-level SSE, logprobs, tools, structured-output
validation, Qwen thinking controls, logprob-biased effort control, and
request diagnostics. There is also a simple built-in
chatinterface. - Correct output first, then speed. Some engines trade output accuracy for benchmark numbers. hipEngine does not: every kernel is checked against a CPU reference implementation, and a faster kernel becomes the default only after its output passes those checks.
Supported models
- Language models: Qwen3.5/3.6/3.8 dense and mixture-of-experts models,
Qwen3.8 Flash-Next (GGUF
UD-Q4_K_XL: text/QSA, opt-in MTP, ≤1K image/video, 32K served context, c4 serving), Gemma 4 26B-A4B, Laguna S 2.1, and Maple-Preview. - Document understanding: Surya OCR 2 for full-page OCR and EVIE 4.5B / 8B for visual document retrieval.
- Speech: VibeVoice ASR for transcription (early support) and VibeVoice TTS for synthesis (experimental). Moonshine ASR has an internal runtime; public audio API support is planned.
- Music generation: YuE2 for songs from lyrics
and a style prompt, through
LLM.generate_song()orPOST /v1/audio/songs(early support). - Time-series forecasting: TimesFM 2.5 and TimesFM 3.0.
See the model support reference for exact checkpoints,
GGUF quantizations, ParoQuant, MLX, and safetensors formats, plus hardware
and API limits. Qwen3.8-27B GGUF Q4_K_M is the dense model to start with
on either AMD backend. Most other modalities are tested on Strix Halo.
NVIDIA Blackwell (sm_120a) support is limited to Maple's Python API, which
you load explicitly with backend="cuda_sm120a".
An independent survey of Qwen3.8-27B implementations on Strix Halo compares hipEngine against other engines on a single Framework Desktop host.
hipEngine includes DMS support and training code, with a published DMS checkpoint for Qwen3.8-27B Q4_K_M. The DMS analysis has the quality results; measured capacity is in Long context on a 24 GB GPU below.
CPU model generation is not supported. The CPU backend is used for correctness tests.
Choose the ParoQuant W4 checkpoint for the optimized Qwen3.6 35B-A3B path, or GGUF for the broader model and quantization ecosystem. See the quantization comparison for quality and speed trade-offs, including ROCmFP4/ROCmFPX.
Performance highlights
Tokens/s on each named host: prompt processing is input, text generation is output. MTP is speculative decoding. Dashes are unmeasured; Max context is a capacity limit, not a full-length prefill.
Radeon Pro W7900 — 48 GB (gfx1100)
| Model | Quant | Prompt processing | Text generation | With MTP | Max context |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B | ParoQuant W4 | 2879.4 | 113.3 | 115.8 | — |
| Qwen3.6-35B-A3B | GGUF UD-Q4_K_M |
2924.5 | 95.0 | 122.7 | — |
| Qwen3.8-27B Dense | GGUF Q4_K_M (BF16 KV) |
898.6 | 30.8 | 39.7 | — |
| Qwen3.8-27B Dense | GGUF Q4_K_M (INT8 KV) |
868.6 | 27.9 | — | 176,128 |
| Gemma 4 26B-A4B | GGUF UD-Q4_K_XL |
3805.6 | 75.8 | 120.2 | — |
| Laguna S 2.1 | GGUF UD-Q2_K_XL |
440.9 | — | — | — |
Strix Halo / Radeon 8060S — 120 GB (gfx1151)
| Model | Quant | Prompt processing | Text generation | With MTP |
|---|---|---|---|---|
| Maple-Preview | 2-bit | 754.5 | 153.2 | — |
| Qwen3.6-35B-A3B | GGUF UD-Q4_K_M |
1418.1 | 56.7 | 80.1 |
| Gemma 4 26B-A4B | GGUF UD-Q4_K_XL |
1865.6 | 43.3 | 69.4 |
| Qwen3.8 Flash-Next | GGUF UD-Q4_K_XL |
1036.2 | 23.7 | 28.9 |
| Laguna S 2.1 | GGUF Q4_K_M |
654.2 | 23.2 | — |
| Qwen3.8-27B Dense | GGUF Q4_K_S |
396.1 | 13.1 | 23.9 |
| Qwen3.8-27B Dense | GGUF Q4_K_M |
404.5 | 12.15 | 22.4 |
| Qwen3.8-27B Dense | GGUF Q4_K_M (INT8 KV) |
— | 10.84 | 19.3 |
NVIDIA RTX PRO 6000 Blackwell — 96 GB (sm_120a)
| Model | Quant | Prompt processing | Text generation | With MTP | Max context |
|---|---|---|---|---|---|
| Maple-Preview | 2-bit | 1917.5 | 402.4 | — | — |
Long context on a 24 GB GPU
Using a source checkout: Qwen3.8-27B Q4_K_M
on RX 7900 XTX, with --max-active-requests 1. Add the
DMS checkpoint
for long context. Omit --max-context-tokens to let the server calculate the
context from available GPU memory.
| KV configuration | Auto context, MTP on | Auto context, MTP off | Manual context, MTP on |
|---|---|---|---|
| BF16 KV | 28,160 | 55,808 | — |
| INT8 KV | 17,664 | 46,336 | — |
| DMS INT8 | 110,592 | — | 131,072 |
Prefix caching
Multi-turn conversations resend the whole transcript, so hipEngine reuses the
KV pages and hybrid state of any 256-token-aligned prefix a later request
repeats. Radix is the default for both the direct engine and HTTP serving
(--prefix-cache off to disable). Qwen3.6-35B-A3B UD-Q4_K_M on Strix Halo:
| Lane | Cache off | Cache on | Wall time |
|---|---|---|---|
| Coding, transcript resent | 10.09 tok/s | 16.49 tok/s | -38.8% |
| Coding, transcript rebuilt | 10.13 tok/s | 12.67 tok/s | -20.0% |
| Chat (ShareGPT) | 29.50 tok/s | 29.15 tok/s | +1.2% |
| All lanes | 16.93 tok/s | 20.81 tok/s | -18.6% |
Serving several requests at once
Aggregate tokens/s across all active requests, Qwen3.8-27B Q4_K_M on the W7900:
| Requests | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
| hipEngine | 23.6 | 39.1 | 53.1 | 63.9 | 72.8 | 79.5 | 83.2 | 85.9 |
| llama.cpp HIP | 21.0 | 34.4 | 30.6 | 27.7 | 36.7 | 46.4 | 52.1 | 58.4 |
| hipEngine advantage | +12% | +14% | +74% | +130% | +99% | +71% | +60% | +47% |
The summary above is synchronized from the benchmark report, which has the full commands, software versions, model hashes, memory use, and correctness checks.
Coding-Agent Session Replay
Replaying a recorded coding session shows the benefit of avoiding repeated
prompt processing. Qwen3.6-35B-A3B UD_Q4_K_M on Strix Halo (gfx1151):
| Prefix cache | End-to-end output | Average turn | Cache hits |
|---|---|---|---|
| Off | 7.05 tok/s | 18.2 s | 0/32 |
| Radix, default retention | 12.61 tok/s | 10.2 s | 19/32 |
| Radix, 16 retained snapshots | 19.69 tok/s | 6.5 s | 30/32 |
2.79x end-to-end throughput, with 91% of prompt tokens reused rather than recomputed. The 16-snapshot setting uses more memory and is opt-in. Results and retention guidance.
Limits
- hipEngine uses one GPU per process; multi-GPU inference is not implemented.
- Speculative decoding does not support packed INT8 or compact-DMS KV. See Server API for admission and fallback behavior.
- Concurrent long-context requests on a 24 GB card are untested; keep the context limit conservative there.
Hardware detection
backend="auto" recognizes gfx1100 and gfx1151. These cover the tested
Radeon Pro W7900 and Ryzen AI MAX+ 395 / Radeon 8060S systems.
Other AMD architecture numbers are not automatically treated as compatible.
You can force a nearby backend, but do so only after checking output quality and performance.
Installation
Requirements
| Platform | Requirements |
|---|---|
| AMD | Linux x86-64, Python 3.11+ and ROCm with hipcc and libamdhip64.so |
| NVIDIA Blackwell | Linux x86-64, Python 3.11+ and the CUDA toolkit with nvcc; Maple only |
| Published wheel | glibc 2.39 or newer, such as Ubuntu 24.04 |
ROCm 7.x is the safest choice for the current wheel; ROCm 10.0 has also been tested and works. See the TheRock setup guide for ROCm 7.13 and gfx1151 ROCm 10 setup and JIT validation. The first model load compiles and caches kernels, so it takes longer than later starts.
Install from PyPI:
pip install hipengine huggingface_hub
Or install a source checkout:
git clone https://github.com/shisa-ai/hipEngine.git
cd hipEngine
git lfs install
git lfs pull
pip install -e .
Confirm that the command is available:
hipengine --help
hipengine serve --help
Start a local server
hipEngine does not download model weights during startup. Download a supported model first, or use a GGUF file that is already on disk.
For the ParoQuant Qwen checkpoint:
hf download shisa-ai/Qwen3.6-35B-A3B-PARO-packed
hipengine serve \
--model shisa-ai/Qwen3.6-35B-A3B-PARO-packed \
--served-model-name qwen-paro
For GGUF, pass the path to the model file:
hipengine serve \
--model /path/to/Qwen3.6-35B-A3B-Q4_K_M.gguf \
--served-model-name qwen
The server listens on http://127.0.0.1:8000 by default. Test it with:
curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen",
"messages": [{"role": "user", "content": "Why is the sky blue?"}],
"max_tokens": 128
}'
Point any client that accepts a custom OpenAI base URL at
http://127.0.0.1:8000/v1. See the server guide for API keys,
streaming, tools, structured output, and model capability checks.
Chat in your terminal
With the server running, open another terminal:
pip install 'hipengine[chat]'
hipengine chat
The client connects to http://127.0.0.1:8000 and discovers the served model.
No model path is needed, and it does not start another server. For a different
address, use hipengine chat --server http://127.0.0.1:8001.
Replies stream as Markdown, with optional reasoning display and per-turn stats
(prompt prefill rate, decode rate, and tokens served from the prefix cache).
Use /status for server limits, /usage for conversation token counts,
/bench for a quick prefill, decode, and prefix-cache measurement of the
running server (with and without speculative MTP), /think off to disable
reasoning, /retry to regenerate, and /clear to start over. /help lists all
commands; /quit, Ctrl-C, or Ctrl-D exits.
Use hipengine chat --plain for plain-text output.
Use the Python API
from hipengine import LLM, SamplingParams
llm = LLM("shisa-ai/Qwen3.6-35B-A3B-PARO-packed")
outputs = llm.generate(
["Hello, hipEngine."],
SamplingParams(max_tokens=64, temperature=0.0),
)
print(outputs[0])
llm.close()
LLM(...) detects a supported AMD GPU and chooses the model format
automatically. You can also pass a local GGUF or Maple path. Advanced users can
override the choice with backend= and quant=. Pass
execution_profile="strict" for results that match the reference kernels
exactly, or "batch_invariant" for output that does not change with other
concurrent requests. The default is "production", the fastest kernels that
pass hipEngine's correctness checks. If a model has no kernels for the requested
profile, loading fails instead of falling back.
Documentation
User guides
| Guide | Contents |
|---|---|
| Server API | OpenAI-compatible endpoints, clients, authentication, and limits |
| Model support | Exact model families, formats, checkpoints, and hardware |
| GGUF models | Supported Qwen formats and model-specific behavior |
| Laguna S 2.1 | Hardware, memory, context, and serving limits |
| Maple-Preview | AMD and NVIDIA support, memory use, and current limits |
| Environment settings | Runtime settings and overrides |
| Changelog | User-facing changes by release |
Development and benchmark details
| Guide | Contents |
|---|---|
| Architecture and roadmap | Engine design and planned work |
| Kernel catalog | Kernel implementations and source history |
| DMS analysis | DMS quality tests and long-context results |
| Testing | Correctness tests and release checks |
| Benchmark methods | Rules used for performance claims |
| Benchmark results | Full result tables and evidence |
| Contributor guide | Repository workflow |
Project lineage
hipEngine is an independent project that builds on ideas and software from ROCm, HIP, Nano-vLLM, ParoQuant, FastDMS, llama.cpp, and other open-source projects. See the source and model guides for detailed attribution.
Thanks
Special thanks to Framework and AMD for providing Strix Halo test hardware.
License
hipEngine source code is licensed under AGPL-3.0-or-later. Model weights, checkpoints, and external datasets remain under their own licenses.
Metadata
Release files for hipengine 0.8.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| hipengine-0.8.1.tar.gz | 20.7 MB | With provenance | Details for hipengine-0.8.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| hipengine-0.8.1-py3-none-manylinux_2_39_x86_64.whl | Python 3 | none | Linux glibc 2.39+ x86-64 | Details for hipengine-0.8.1-py3-none-manylinux_2_39_x86_64.whl |
Total release size: 48.2 MB
Release files / hipengine-0.8.1.tar.gz
| Download URL | hipengine-0.8.1.tar.gz |
|---|---|
| Size | 20.7 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f0b819468b50dae7cd505f828d2407cdc4ab3efc296b79daded421f2a57f1968
|
|
BLAKE2b-256 checksum How to use checksums |
e96b94170807e75ced9fea1e099314726baff4f908711f1c59d7c419450194f4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.13
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 21, 2026.
Transparency logRelease files / hipengine-0.8.1-py3-none-manylinux_2_39_x86_64.whl
| Download URL | hipengine-0.8.1-py3-none-manylinux_2_39_x86_64.whl |
|---|---|
| Size | 27.5 MB |
| Tags | Linux glibc 2.39+ x86-64 Python 3 |
|
SHA-256 checksum How to use checksums |
da02bdedab837c6d22f61468d68cf608b38b9e85701ba3490574fe5d45ddfe02
|
|
BLAKE2b-256 checksum How to use checksums |
0def4525c31bcaa9eb7ccaeec0d2d89a8a96574a6ee8bbd956ea248997928e37
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.13
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 21, 2026.
Transparency logSecurity for hipengine 0.8.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for hipengine-0.8.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 21, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.