auto-round-lib 0.16.0

What is AutoRound Kernel (ARK)?

AutoRound Kernel (ARK) is a low-bit acceleration library for Intel platform, providing three categories of optimized operators for LLM inference.

Validated CPU: Intel Xeon Scalable (Sapphire Rapids / Emerald Rapids), Intel Xeon 6 (Sierra Forest / Granite Rapids)
Validated GPU: Intel Arc B-Series / Arc Pro B-Series (Battlemage)

Highlights — Ecosystem Integration

ARK kernels are integrated into the following projects:

1. Linear (Weight-Only Quantized GEMM)

Low-bit weight-only linear for LLM inference. Both CPU and XPU are supported.

API

Key Features

W4A8 / W2A8 Rescale (QQQ-style) — On XPU, ARK supports ~QQQ-style compute: low-bit weights (INT2/INT4) are re-scaled to INT8 and computed via INT8 GEMM, avoiding FP16 dequantization for better throughput. Enabled automatically via environment variable ARK_AUTO_S8; see xpu_wrapper.hpp.

Supported Data Types

CPU

XPU

^[1] INT8 compute includes dynamic activation quantization; results are dequantized to floating-point.

Example

import auto_round_kernel as ark

# Prepare quantized weight: qweight [K, N] int4/int2, scale [K/G, N] fp16/fp32, zp [K/G, N] int4/int2
packw = ark.repack_quantized_weight(
    qweight,
    scale,
    zp,
    blocksize=128,
    compute_type="fp16",
    weight_type="int4",
    scale_type="fp16",
    asym=False,
)

# Run weight-only quantized GEMM: activation [M, K] → output [M, N]
output = ark.woqgemm(
    activation,  # [M, K] fp16/bf16
    packw,  # packed weight blob (INT8)
    bias,  # [1, N] optional bias
    n,  # output features
    k,  # input features
    groupsize=128,
    compute_type="fp16",
    weight_type="int4",
    scale_type="fp16",
    asym=False,
)

# Decompose back to full precision for verification
decompressed = ark.unpack_weight(
    packw,
    dtype=torch.float16,
    n=n,
    k=k,
    groupsize=128,
    compute_type="fp16",
    weight_type="int4",
    scale_type="fp16",
    asym=False,
)

See test_weightonly.py for an end-to-end example of weight repack, verification, and woqgemm execution on CPU and XPU.

2. MoE (Mixture-of-Experts Grouped GEMM)

Grouped GEMM for MoE layers where different experts process varying numbers of tokens.

API

🚧 INT2 / INT4 / INT8 weight support is under active development. See #PR.

Details

Example

# FP16/BF16 MoE
output = ark.moe_gemm(activations, weights, num_tokens_per_expert)

# INT4 MoE (coming soon)
# output = ark.moe_gemm(activations, q4_weights, num_tokens_per_expert, scales=scales)

Build requirement: ARK_SYCL_TLA=ON. See test_moe.py.

3. SageAttention (XPU SDPA Acceleration)

ARK provides a full family of scaled dot-product attention kernels on XPU, ranging from vanilla FP16 SDPA to INT8-quantized SageAttention variants.

API Overview

Comparison

Drop-in SDPA Replacement

Replace torch.nn.functional.scaled_dot_product_attention globally for lm-eval:

Replace torch SDPA and run lm-eval

ARK exposes a standard SDPA interface through ARK.sdpa(...). The implementation borrows from Neural Speed route logic internally, but the public contract is the standard scaled-dot-product-attention surface. If you want to replace torch.nn.functional.scaled_dot_product_attention globally for evaluation without editing model code, use the helper launcher in tools/lm_eval_with_ark_sdpa.py.

cd /path/to/auto_round_extension/ark
PYTHONPATH=$PWD python tools/lm_eval_with_ark_sdpa.py \
  --model hf \
  --model_args pretrained=/path/to/model,trust_remote_code=True,dtype=bfloat16 \
  --tasks hellaswag,piqa,winogrande \
  --device xpu:0 --batch_size 1

The patching logic (in auto_round_kernel/torch_sdpa_patch.py) routes to ARK on XPU when constraints are met; otherwise falls back to PyTorch SDPA.

Constraints

Installation

Install via pip

pip install auto-round-lib

Install from Source

pip install . --no-build-isolation
# or
python setup.py bdist_wheel; pip install dist/*

Build with MoE / SageAttention support requires ARK_SYCL_TLA=ON.

For oneAPI/SYCL-TLA builds, template compilation can require several gigabytes per compiler process. If the build host runs out of memory, limit concurrent XPU compilations with ARK_SYCL_TLA_JOBS:

ARK_SYCL_TLA_JOBS=1 pip install . --no-build-isolation

On a oneAPI-equipped build host, configure an XPU build with compile commands enabled and measure each translation unit before choosing a higher job count:

python tools/measure_sycl_tla_compile_memory.py \
  --build-dir /path/to/ark/xbuild \
  --output /tmp/ark-sycl-tla-rss.json

The measurement tool runs compiler commands serially and reports peak RSS, direct dispatch count, and reachable template declarations. The current environment must provide icx and a generated compile_commands.json for these measurements to represent the SYCL-TLA build.

Tests