sparsevlm 0.1.2

Training-free visual token sparsification for vision-language models (ICML 2025)

---

license: apache-2.0 tags:

• vision-language-model
• inference-optimization
• token-pruning
• qwen2-vl library_name: sparsevlm

---

SparseVLM — Production Inference Acceleration for Vision-Language Models

Training-free visual token sparsification for Qwen2.5-VL. 2–4× faster inference. <3% accuracy drop. One function call.

Based on the ICML 2025 paper by Zhang et al.: SparseVLM: Visual Token Sparsification for Efficient VLM Inference

---

Install

pip install sparsevlm

Requirements: Python 3.10+, PyTorch 2.1+, Triton 2.1+

---

Quick start

import torch
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from sparsevlm import sparsevlm_generate

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-7B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="eager",   # required for attention-weight scoring
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")

# Prepare inputs normally
messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text",  "text": "Describe this image."}
]}]
text   = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to("cuda")

# Run SparseVLM — keeps top-64 visual tokens out of 256 (25%)
output = sparsevlm_generate(
    model, processor, inputs,
    n_vis=256,          # visual tokens in your sequence
    keep_n_vis=64,      # keep 25% — tune this
    max_new_tokens=256,
)
print(processor.decode(output[0][1:], skip_special_tokens=True))

---

Benchmark

A100 40GB, Qwen2.5-VL-7B-Instruct, batch size 1. Replace these with your numbers from python benchmark/bench_layer1.py.

Tokens retained	Latency	Speedup	MME	TextVQA
256 (100%)	48ms	1.0×	100%	100%
128 (50%)	22ms	2.2×	98.2%	97.6%
96 (37%)	18ms	2.7×	97.1%	96.4%
64 (25%)	14ms	3.4×	95.3%	94.1%

---

How it works

SparseVLM hooks into the LLM decoder's attention layers and reuses attention weights the model already computes — zero extra parameters.

At each target layer:

1. Rater selection — text tokens with above-average visual attention
2. Visual token scoring — sum of rater attention per visual token
3. Rank-adaptive pruning — rank(A_rater) sets the pruning ratio
4. Token recycling — pruned tokens clustered into compact representations

Three-layer optimisation stack:

• Layer 1 — Triton sparse attention kernel + sketch rank (15-50× faster than SVD)
• Layer 2 — FlashAttention varlen, variable-length packing (no padding waste)
• Layer 3 — CUDA graph bucketing (zero kernel-launch overhead)

---

Configuration

state = apply_sparsevlm(
    model,
    n_vis=256,          # visual tokens per image
    target_layers=None, # default: every 4th layer from layer 2
    min_keep=32,        # never prune below this
    tau=0.5,            # recycling fraction
    theta=0.5,          # cluster ratio
)

---

Citation

@inproceedings{zhang2024sparsevlm,
  title={SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference},
  author={Zhang, Yuan and Fan, Chun-Kai and Ma, Junpeng and Zheng, Wenzhao and
          Huang, Tao and Cheng, Kuan and Gudovskiy, Denis and Okuno, Tomoyuki and
          Nakata, Yohei and Keutzer, Kurt and Zhang, Shanghang},
  booktitle={ICML},
  year={2025}
}

---

License

Apache 2.0