hip-attn 1.2.9

:sunglasses: HiP Attention could extend the model context length training-free and can serve 3 million tokens with a single L40S 48GB GPU while achieving a 7.24 estimated speedup.

| Paper (Arxiv, InfiniteHiP latest) | Paper (ICLR 2025) | SGlang Integration |

[!NOTE] You can try it in our Playground in DeepAuto.ai!

[!IMPORTANT] This is NOT yet free for commercial use. The license is FSL-1.1-MIT, which is free for non-commercial use but will automatically convert to MIT license two years after each release. Please refer to the LICENSE for more details.

News

• 2025.01.26: Version 1.2 is now ready! The preprint is now prepared in arxiv.
• 2025.01.22: HiP Attention is accepted in ICLR 2025!

Usage

hip-attn package is available on PyPI:

pip install hip-attn

or using uv:

uv add hip-attn

After installation, you can access the hip package from any project. hip is the code name of HiP attention.

import torch
from hip_attn import hip_attention_12, HiPAttentionArgs12

device = 'cuda'

batch_size = 1
kv_len = 128 * 1024
q_len = 32 * 1024
num_heads = 32
num_kv_heads = 8
head_dims = 128
dtype = torch.bfloat16

q = torch.randn(
    (batch_size, q_len, num_heads, head_dims),
    dtype=dtype,
    device=device
)
k = torch.randn(
    (batch_size, kv_len, num_kv_heads, head_dims),
    dtype=dtype,
    device=device,
)
v = k.clone()

output, metadata = hip_attention_12(q=q, k=k, v=v, args=HiPAttentionArgs12())
print(output.shape)

# > torch.Size([1, 32768, 32, 128])

Getting Started

Local development

Using uv (Recommended)

It’s recommended to use uv, a very fast Python environment manager, to create and manage Python environments. Please follow the documentation to install uv. After installing uv, you can create a new Python environment and install hip-attention using the following commands:

# Clone this repository
git clone git@github.com:DeepAuto-AI/hip-attention.git
cd hip-attention

# This install all research dev dependencies in .venv/
uv sync --no-dev  # Install base dependencies first
uv sync  # Then install all dependencies including no-build-isolation packages (e.g., flash-attn)
uv run pre-commit install

Then you can run any python program with uv run. uv run automatically picks up .venv/ virtual environment:

• Script: uv run src/hip_research/main/model_eval.py
• Module: uv run -m src.hip_research.main.model_eval

Using pip and conda

# Clone this repository
git clone git@github.com:DeepAuto-AI/hip-attention.git
cd hip-attention

# Make new conda environment
conda create --name hip python=3.11
conda activate hip

# Default install
pip install -e "."
# (Optional) For research benchmarks and unit tests
pip install -e "hip-research"

# Optional, depends on your CUDA environment
export CUDACXX=/usr/local/cuda/bin/nvcc

# Install SGLang with support for HiP Attention
pip install -e ".[sglang]" \
"sglang[all] @ git+https://github.com/DeepAuto-AI/sglang.git@deepauto/release#subdirectory=python" \
--no-build-isolation \
--verbose \
--find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python

Docker

Docker images deepauto/hip-attention are available on Docker Hub. Docker examples are available in Running section.

Running

See the following pages for more details:

• Running OpenAI API server examples (SGlang)

Experiment Reproduce

Check how to reproduce experiment page

Citation

@misc{willette2025_delta_attention,
      title={Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction},
      author={Jeffrey Willette and Heejun Lee and Sung Ju Hwang},
      year={2025},
      eprint={2505.11254},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2505.11254},
}

@misc{lee2025_infinite_hip,
      title={InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU},
      author={Heejun Lee and Geon Park and Jaduk Suh and Sung Ju Hwang},
      year={2025},
      eprint={2502.08910},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.08910},
}

@inproceedings{lee2025_hip_attention,
      title={A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention},
      author={Heejun Lee and Geon Park and Youngwan Lee and Jaduk Suh and Jina Kim and Wonyong Jeong and Bumsik Kim and Hyemin Lee and Myeongjae Jeon and Sung Ju Hwang},
      booktitle={The Thirteenth International Conference on Learning Representations},
      year={2025},
      url={https://openreview.net/forum?id=PTcMzQgKmn}
}

Contributing

Updating dependencies

# This will update git commit hash of sglang
uv lock --upgrade-package sglang
uv sync

Building and publishing

• PyPI

rm -rf dist
uv build --no-sources
uv publish

• Docker

git clone git@github.com:DeepAuto-AI/hip-attention.git
cd hip-attention
docker login

docker build -t deepauto/hip-attention:latest -t deepauto/hip-attention:latest-sglang -t deepauto/hip-attention:$(git rev-parse --short HEAD)-sglang -t deepauto/hip-attention:v$(uv run python -c 'import importlib.metadata; print(importlib.metadata.version("hip-attn"))')-sglang -f Dockerfile.sglang .

docker push deepauto/hip-attention:latest
docker push deepauto/hip-attention:latest-sglang
docker push deepauto/hip-attention:$(git rev-parse --short HEAD)-sglang
docker push deepauto/hip-attention:v$(uv run python -c 'import importlib.metadata; print(importlib.metadata.version("hip-attn"))')-sglang