Skip to main content

hipEngine

hipEngine is a ROCm-native local inference engine built primarily for AMD Radeon GPUs. It pairs a small Python host with custom HIP kernels for torch-free model loading, generation, and OpenAI-compatible serving on supported hardware.

Current release: v0.8.3. Fixes Gemma 4 serving and GPU resource cleanup, corrects Flash-Next Q4 decode selection, and reduces redundant Flash-Next work. See the release notes for scope and limitations.

Why use hipEngine?

  • Native AMD support. HIP-first kernels are written from scratch and tuned specifically for RDNA 3 (gfx1100) and Strix Halo RDNA 3.5 (gfx1151) instead of being CUDA ports.
  • No PyTorch runtime required. There is no PyTorch dependency, which keeps hipEngine lightweight.
  • Optimized for agents and concurrent requests. hipEngine is tuned for both single-request latency and multiple concurrent requests, with continuous batching, prefix caching, and a shared KV pool.
  • Drop-in support for existing clients. The included OpenAI-compatible server supports completion, chat, token-level SSE, logprobs, tools, structured-output validation, Qwen thinking controls, logprob-biased effort control, and request diagnostics. There is also a simple built-in chat interface.
  • Correct output first, then speed. Some engines trade output accuracy for benchmark numbers. hipEngine does not: every kernel is checked against a CPU reference implementation, and a faster kernel becomes the default only after its output passes those checks.

Supported models

  • Language models: Qwen3.5/3.6/3.8 dense and mixture-of-experts models, Qwen3.8 Flash-Next (GGUF UD-Q4_K_XL: text/QSA, opt-in MTP, ≤1K image/video, 32K served context, c4 serving), Gemma 4 26B-A4B, Laguna S 2.1, and Maple-Preview.
  • Document understanding: Surya OCR 2 for full-page OCR and EVIE 4.5B / 8B for visual document retrieval.
  • Speech: VibeVoice ASR for transcription (early support) and VibeVoice TTS for synthesis (experimental). Moonshine ASR has an internal runtime; public audio API support is planned.
  • Music generation: YuE2 for songs from lyrics and a style prompt, through LLM.generate_song() or POST /v1/audio/songs (early support).
  • Time-series forecasting: TimesFM 2.5 and TimesFM 3.0.

See the model support reference for exact checkpoints, GGUF quantizations, ParoQuant, MLX, and safetensors formats, plus hardware and API limits. Qwen3.8-27B GGUF Q4_K_M is the dense model to start with on either AMD backend. Most other modalities are tested on Strix Halo. NVIDIA Blackwell (sm_120a) support is limited to Maple's Python API, which you load explicitly with backend="cuda_sm120a".

An independent survey of Qwen3.8-27B implementations on Strix Halo compares hipEngine against other engines on a single Framework Desktop host.

hipEngine includes DMS support and training code, with a published DMS checkpoint for Qwen3.8-27B Q4_K_M. The DMS analysis has the quality results; measured capacity is in Long context on a 24 GB GPU below.

CPU model generation is not supported. The CPU backend is used for correctness tests.

Choose the ParoQuant W4 checkpoint for the optimized Qwen3.6 35B-A3B path, or GGUF for the broader model and quantization ecosystem. See the quantization comparison for quality and speed trade-offs, including ROCmFP4/ROCmFPX.

Performance highlights

Tokens/s on each named host: prompt processing is input, text generation is output. MTP is speculative decoding. Dashes are unmeasured; Max context is a capacity limit, not a full-length prefill.

Radeon Pro W7900 — 48 GB (gfx1100)

Model Quant Prompt processing Text generation With MTP
Qwen3.6-35B-A3B ParoQuant W4 2879.4 113.3 115.8
Qwen3.6-35B-A3B GGUF UD-Q4_K_M 2924.5 95.0 122.7
Qwen3.8-27B Dense GGUF Q4_K_M (BF16 KV) 898.6 30.8 39.7
Qwen3.8-27B Dense GGUF Q4_K_M (INT8 KV) 868.6 27.9 —
Gemma 4 26B-A4B GGUF UD-Q4_K_XL 3805.6 75.8 120.2
Laguna S 2.1 GGUF UD-Q2_K_XL 440.9 — —

Strix Halo / Radeon 8060S — 120 GB (gfx1151)

Model Quant Prompt processing Text generation With MTP
Qwen3.6-35B-A3B GGUF UD-Q4_K_M 1418.1 56.7 80.1
Qwen3.8-27B Dense GGUF Q4_K_M 404.5 12.15 22.4
Qwen3.8-27B Dense GGUF Q4_K_M (INT8 KV) — 10.84 19.3
Qwen3.8 Flash-Next GGUF UD-Q4_K_XL 1036.2 23.7 28.9
Gemma 4 26B-A4B GGUF UD-Q4_K_XL 1865.6 43.3 69.4
Laguna S 2.1 GGUF Q4_K_M 654.2 23.2 —
Maple-Preview 2-bit 754.5 153.2 —

NVIDIA RTX PRO 6000 Blackwell — 96 GB (sm_120a)

Model Quant Prompt processing Text generation With MTP Max context
Maple-Preview 2-bit 1917.5 402.4 — —

Long context on a 24 GB GPU

Using a source checkout: Qwen3.8-27B Q4_K_M on RX 7900 XTX, with --max-active-requests 1. Add the DMS checkpoint for long context. Omit --max-context-tokens to let the server calculate the context from available GPU memory.

KV configuration Auto context, MTP on Auto context, MTP off
BF16 KV 61,440 83,456
INT8 KV 71,680 98,304
DMS INT8 262,144 262,144

Prefix caching

Multi-turn conversations resend the whole transcript, so hipEngine reuses the KV pages and hybrid state of any 256-token-aligned prefix a later request repeats. Radix is the default for both the direct engine and HTTP serving (--prefix-cache off to disable). Qwen3.6-35B-A3B UD-Q4_K_M on Strix Halo:

Lane Cache off Cache on Wall time
Coding, transcript resent 10.09 tok/s 16.49 tok/s -38.8%
Coding, transcript rebuilt 10.13 tok/s 12.67 tok/s -20.0%
Chat (ShareGPT) 29.50 tok/s 29.15 tok/s +1.2%
All lanes 16.93 tok/s 20.81 tok/s -18.6%

Serving several requests at once

Aggregate tokens/s across all active requests, Qwen3.8-27B Q4_K_M on the W7900:

Requests 1 2 3 4 5 6 7 8
hipEngine 23.6 39.1 53.1 63.9 72.8 79.5 83.2 85.9
llama.cpp HIP 21.0 34.4 30.6 27.7 36.7 46.4 52.1 58.4
hipEngine advantage +12% +14% +74% +130% +99% +71% +60% +47%

The summary above is synchronized from the benchmark report, which has the full commands, software versions, model hashes, memory use, and correctness checks.

Coding-Agent Session Replay

Replaying a recorded coding session shows the benefit of avoiding repeated prompt processing. Qwen3.6-35B-A3B UD_Q4_K_M on Strix Halo (gfx1151):

Prefix cache End-to-end output Average turn Cache hits
Off 7.05 tok/s 18.2 s 0/32
Radix, default retention 12.61 tok/s 10.2 s 19/32
Radix, 16 retained snapshots 19.69 tok/s 6.5 s 30/32

2.79x end-to-end throughput, with 91% of prompt tokens reused rather than recomputed. The 16-snapshot setting uses more memory and is opt-in. Results and retention guidance.

Limits

  • hipEngine uses one GPU per process; multi-GPU inference is not implemented.
  • Speculative decoding does not support packed INT8 or compact-DMS KV. See Server API for admission and fallback behavior.
  • Concurrent long-context requests on a 24 GB card are untested; keep the context limit conservative there.

Hardware detection

backend="auto" recognizes gfx1100 and gfx1151. These cover the tested Radeon Pro W7900 and Ryzen AI MAX+ 395 / Radeon 8060S systems. Other AMD architecture numbers are not automatically treated as compatible.

You can force a nearby backend, but do so only after checking output quality and performance.

Installation

Requirements

Platform Requirements
AMD Linux x86-64, Python 3.11+ and ROCm with hipcc and libamdhip64.so
NVIDIA Blackwell Linux x86-64, Python 3.11+ and the CUDA toolkit with nvcc; Maple only
Published wheel glibc 2.39 or newer, such as Ubuntu 24.04

ROCm 7.x or newer is recommended. See the TheRock setup guide for ROCm 7.13 and gfx1151 ROCm 10 setup and JIT validation. The first model load compiles and caches kernels, so it takes longer than later starts.

Install from PyPI:

pip install hipengine huggingface_hub

Or install a source checkout:

git clone https://github.com/shisa-ai/hipEngine.git
cd hipEngine
git lfs install
git lfs pull
pip install -e .

Confirm that the command is available:

hipengine --help
hipengine serve --help

Start a local server

hipEngine does not download model weights during startup. Download a supported model first, or use a GGUF file that is already on disk.

For the ParoQuant Qwen checkpoint:

hf download shisa-ai/Qwen3.6-35B-A3B-PARO-packed

hipengine serve \
  --model shisa-ai/Qwen3.6-35B-A3B-PARO-packed \
  --served-model-name qwen-paro

For GGUF, pass the path to the model file:

hipengine serve \
  --model /path/to/Qwen3.6-35B-A3B-Q4_K_M.gguf \
  --served-model-name qwen

The server listens on http://127.0.0.1:8000 by default. Test it with:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen",
    "messages": [{"role": "user", "content": "Why is the sky blue?"}],
    "max_tokens": 128
  }'

Point any client that accepts a custom OpenAI base URL at http://127.0.0.1:8000/v1. See the server guide for API keys, streaming, tools, structured output, and model capability checks.

Chat in your terminal

With the server running, open another terminal:

pip install 'hipengine[chat]'
hipengine chat

The client connects to http://127.0.0.1:8000 and discovers the served model. No model path is needed, and it does not start another server. For a different address, use hipengine chat --server http://127.0.0.1:8001.

Replies stream as Markdown, with optional reasoning display and per-turn stats (prompt prefill rate, decode rate, and tokens served from the prefix cache). Use /status for server limits, /usage for conversation token counts, /bench for a quick prefill, decode, and prefix-cache measurement of the running server (with and without speculative MTP), /think off to disable reasoning, /retry to regenerate, and /clear to start over. /help lists all commands; /quit, Ctrl-C, or Ctrl-D exits. Use hipengine chat --plain for plain-text output.

Use the Python API

from hipengine import LLM, SamplingParams

llm = LLM("shisa-ai/Qwen3.6-35B-A3B-PARO-packed")
outputs = llm.generate(
    ["Hello, hipEngine."],
    SamplingParams(max_tokens=64, temperature=0.0),
)
print(outputs[0])
llm.close()

LLM(...) detects a supported AMD GPU and chooses the model format automatically. You can also pass a local GGUF or Maple path. Advanced users can override the choice with backend= and quant=. Pass execution_profile="strict" for results that match the reference kernels exactly, or "batch_invariant" for output that does not change with other concurrent requests. The default is "production", the fastest kernels that pass hipEngine's correctness checks. If a model has no kernels for the requested profile, loading fails instead of falling back.

Documentation

User guides

Guide Contents
Server API OpenAI-compatible endpoints, clients, authentication, and limits
Model support Exact model families, formats, checkpoints, and hardware
GGUF models Supported Qwen formats and model-specific behavior
Laguna S 2.1 Hardware, memory, context, and serving limits
Maple-Preview AMD and NVIDIA support, memory use, and current limits
Environment settings Runtime settings and overrides
Changelog User-facing changes by release

Development and benchmark details

Guide Contents
Architecture and roadmap Engine design and planned work
Kernel catalog Kernel implementations and source history
DMS analysis DMS quality tests and long-context results
Testing Correctness tests and release checks
Benchmark methods Rules used for performance claims
Benchmark results Full result tables and evidence
Contributor guide Repository workflow

Project lineage

hipEngine is an independent project that builds on ideas and software from ROCm, HIP, Nano-vLLM, ParoQuant, FastDMS, llama.cpp, and other open-source projects. See the source and model guides for detailed attribution.

Thanks

Special thanks to Framework and AMD for providing Strix Halo test hardware.

License

hipEngine source code is licensed under AGPL-3.0-or-later. Model weights, checkpoints, and external datasets remain under their own licenses.

Metadata

Release files for hipengine 0.8.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hipengine 0.8.3
File Size Uploaded Provenance Actions
hipengine-0.8.3.tar.gz 20.8 MB With provenance Details for hipengine-0.8.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for hipengine 0.8.3
File Interpreter ABI Platform Actions
hipengine-0.8.3-py3-none-manylinux_2_39_x86_64.whl Python 3 none Linux glibc 2.39+ x86-64 Details for hipengine-0.8.3-py3-none-manylinux_2_39_x86_64.whl

Total release size: 48.4 MB

Release files / hipengine-0.8.3.tar.gz

Download URL hipengine-0.8.3.tar.gz
Size 20.8 MB
Tags Source
SHA-256 checksum
How to use checksums
887445080883119a3334f9b2ee73646f61d0551630a8e54f50e01fd887177363
BLAKE2b-256 checksum
How to use checksums
fd1742252dad9ac7d203684acdda25e22bf6f3101bfab7c72c56a9cad6bdaed2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.13

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 23, 2026.

Transparency log

Release files / hipengine-0.8.3-py3-none-manylinux_2_39_x86_64.whl

Download URL hipengine-0.8.3-py3-none-manylinux_2_39_x86_64.whl
Size 27.6 MB
Tags Linux glibc 2.39+ x86-64 Python 3
SHA-256 checksum
How to use checksums
826ad653f12cbf25780db0f758e47b4339c435be7291ddd5dcf3f5be30af74b5
BLAKE2b-256 checksum
How to use checksums
a65672b38ef80696270d40b82adee3e6b8dcfb5b5b43b0610117c73d597150c7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.13

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 23, 2026.

Transparency log

Security for hipengine 0.8.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for hipengine-0.8.3.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 23, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page