hipEngine

hipEngine is a ROCm-native local inference engine built primarily for AMD Radeon GPUs. It pairs a small Python host with custom HIP kernels for torch-free model loading, generation, and OpenAI-compatible serving on supported hardware.

Current release: v0.4.0 alpha. Besides the Qwen 3.6 PARO and GGUF models, the latest version of hipEngine now supports GGUF inference for more model families. These include Laguna S 2.1, Maple ternary, and Moonshine ASR.

Why use hipEngine?

• Native AMD support. HIP-first kernels directly target and tune for specific RDNA 3 (gfx1100) and Strix Halo RDNA 3.5 (gfx1151) instead of being CUDA ports.
• No PyTorch runtime required. There is no PyTorch dependency, which keeps hipEngine lightweight. Although it is packaged for Python, almost all of the hot path is C++.
• Optimized for agents and concurrent requests. Besides extensive tuning for fast single-request performance (especially for prefill), hipEngine also has tuned support for c=N. It is significantly faster than llama.cpp or vLLM for c=8 workloads.
• Drop-in support for existing clients. The included OpenAI-compatible server supports completion, chat, token-level SSE, logprobs, tools, structured-output validation, Qwen thinking controls, logprob-biased effort control, and request diagnostics.

hipEngine is a new, small software project focused on making a select list of models perform well, particularly Qwen 3.x variants and fine-tunes.

Supported models

Yes means that public text generation has been tested. A dash means that the combination is not supported. Features such as batching, sampling, tools, and long context can differ by model.

CPU model generation is not supported. The CPU backend is used for correctness tests. On NVIDIA, load Maple with backend="cuda_sm120a"; automatic hardware selection currently covers AMD only.

Support is specific to the listed model families and formats. hipEngine does not yet run every GGUF model. See the GGUF, Laguna, and Maple guides for model-specific limits.

GGUF or ParoQuant for Qwen?

For Qwen3.6 35B-A3B, the optimized ParoQuant W4 checkpoint is slightly faster and uses less memory in our AMD tests. It is a good choice when that exact model meets your needs.

GGUF has a much larger model and quantization ecosystem. Current development is therefore focused on GGUF compatibility, while the optimized ParoQuant path remains supported.

Installation

Requirements

ROCm 7.x is the safest choice for the current wheel. The first model load compiles and caches kernels, so it takes longer than later starts.

Install from PyPI:

pip install hipengine huggingface_hub

Or install a source checkout:

git clone https://github.com/shisa-ai/hipEngine.git
cd hipEngine
git lfs install
git lfs pull
pip install -e .

Confirm that the command is available:

hipengine --help
hipengine serve --help

Start a local server

hipEngine does not download model weights during startup. Download a supported model first, or use a GGUF file that is already on disk.

For the ParoQuant Qwen checkpoint:

hf download shisa-ai/Qwen3.6-35B-A3B-PARO-packed

hipengine serve \
  --model shisa-ai/Qwen3.6-35B-A3B-PARO-packed \
  --served-model-name qwen-paro

For GGUF, pass the path to the model file:

hipengine serve \
  --model /path/to/Qwen3.6-35B-A3B-Q4_K_M.gguf \
  --served-model-name qwen

The server listens on http://127.0.0.1:8000 by default. Test it with:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen",
    "messages": [{"role": "user", "content": "Why is the sky blue?"}],
    "max_tokens": 128
  }'

Point any client that accepts a custom OpenAI base URL at http://127.0.0.1:8000/v1. See the server guide for API keys, streaming, tools, structured output, and model capability checks.

Use the Python API

from hipengine import LLM, SamplingParams

llm = LLM("shisa-ai/Qwen3.6-35B-A3B-PARO-packed")
outputs = llm.generate(
    ["Hello, hipEngine."],
    SamplingParams(max_tokens=64, temperature=0.0),
)
print(outputs[0])
llm.close()

LLM(...) detects a supported AMD GPU and chooses the model format automatically. You can also pass a local GGUF or Maple path. Advanced users can override the choice with backend= and quant=.

Performance highlights

These are measured results, not estimates. Prompt processing is the speed of reading the input. Text generation is the speed of producing new tokens.

These rows use different hardware and tests. Do not compare one row directly with another.

Multiple requests

Each value is the total tokens per second across all active requests:

Optional speculative modes

These speculative modes are opt-in because their output can differ from normal generation.

Full commands, software versions, model hashes, memory use, and correctness checks are in the benchmark report.

Status and limits

v0.4.0 is a large alpha release. It adds or expands:

• Qwen3.5 and Qwen3.6 GGUF model support on both AMD backends.
• Native parallel request handling for supported Qwen and Maple paths.
• Laguna S 2.1 generation and serving on Radeon 8060S systems.
• Maple-Preview 2-bit generation on AMD, plus an experimental native CUDA path for NVIDIA Blackwell.
• Faster prompt processing and generation across the supported AMD paths.
• OpenAI-compatible streaming, sampling, tools, structured-output validation, request cancellation, and an endpoint that reports available features.

Important limits:

• hipEngine uses one GPU. Multi-GPU inference is not implemented.
• There is no desktop GUI, model catalog, or automatic model download.
• CPU model inference is not implemented.
• NVIDIA support is limited to single-request Maple generation through the Python API. CUDA server and multi-request support are not ready.
• Maple currently uses greedy generation only.
• Advertised model context lengths are not a promise that hipEngine supports the same length. Use the model guide and set a conservative server context limit.
• Speculative generation is optional and off by default when it changes output or does not provide a reliable speed benefit.
• APIs and supported combinations can still change before 1.0.

Hardware detection

backend="auto" recognizes gfx1100 and gfx1151. These cover the tested Radeon RX 7900 XTX / Pro W7900 and Ryzen AI MAX+ 395 / Radeon 8060S systems. Other AMD architecture numbers are not automatically treated as compatible.

You can force a nearby backend, but do so only after checking output quality and performance. hipEngine will not silently use PyTorch when a GPU is unsupported.

Documentation

User guides

Development and benchmark details

Project lineage

hipEngine is an independent project that builds on ideas and software from ROCm, HIP, Nano-vLLM, ParoQuant, FastDMS, llama.cpp, and other open-source projects. See the source and model guides for detailed attribution.

License

hipEngine source code is licensed under AGPL-3.0-or-later. Model weights, checkpoints, and external datasets remain under their own licenses.