🩻 vit-xray

Look inside any vision transformer backbone in one command

vit-xray photo.jpg

_{Patch-token norms at the final block, seven pretrained backbones. Most of them quietly repurpose a couple of percent of their patch tokens as scratch space; the bright dots, sitting in sky, walls and flat ground.
DINO, MAE, and DINOv2-with-registers do not.}

https://github.com/user-attachments/assets/f02d2808-1d5e-46ac-88f1-36eb0c5487a2

⚡ Quickstart

conda create -n vitxray python=3.11
conda activate vitxray
pip install vit-xray

vit-xray photo.jpg

For the clearest picture, use the model the figures above lead with:

vit-xray photo.jpg --model vit_giant_patch14_dinov2.lvd142m --all-panels

That's DINOv2 ViT-g/14, a 37×37 grid instead of 14×14, and roughly twenty outlier tokens instead of seven. It costs a 4.3 GB download and ~10 GB of RAM on CPU, which is exactly why it isn't the default.

🔍 What am I looking at

Vision transformers cut an image into patches and give each patch a token. You'd expect every token to describe its own patch. Mostly they do, but in a trained ViT, a handful of tokens in boring regions get quietly repurposed.

The model notices that a patch of empty sky looks exactly like its neighbours, decides it has nothing worth storing, and reuses its token as scratch space for global information about the whole image. Those tokens end up with an L2 norm about ten times larger than everything around them, and they no longer describe the patch they came from.

That's the phenomenon Darcet et al. named and measured in Vision Transformers Need Registers (ICLR 2024): roughly 2% of tokens, only in large enough models trained long enough, and fixable by giving the model dedicated scratch tokens registers so it stops stealing patch tokens for the job.

vit-xray gives you three views of it, plus a score:

The score fits a two-component Gaussian mixture to the norms and thresholds at the crossover. There is no hardcoded cutoff: the paper's own was hand-picked for one model, and it says outright that it varies. If the norms aren't meaningfully bimodal, the score is zero and the backbone is reported clean.

🎯 Registers fix it

Same architecture, same width, same training data, one shared colour scale. Registers are the only difference:

📈 The norm distribution

A backbone with artifacts has a visibly bimodal norm distribution, a bulk, a gap, and a small high-norm cluster. That gap is where the fitted threshold lands. DINO has no gap.

💡 Why you should care

If you're freezing a backbone and reading its patch tokens, semantic segmentation, monocular depth, feature fields, open-vocabulary detection, robotic semantic mapping, then artifact tokens are corrupted inputs to your dense head. They carry global information, not local, so whatever sits under them is described wrongly.

Darcet et al. show this measurably hurts dense prediction and breaks unsupervised object discovery: DINOv2 was worse than supervised baselines with LOST until registers were added.

It costs one command to find out which kind of backbone you have. If yours scores clean, ignore all of this. If it doesn't, pick a register-trained variant, or look at the training-free mitigation in Vision Transformers Don't Need Trained Registers.

🧠 Model support

Measured by vit-xray --gallery configs/gallery.yaml, which prints this table as it runs. Percentages are the artifact score on each of the four gallery images.

Any timm ViT works, not just these. The patch grid and prefix-token count are read from the model, never hardcoded, so DINOv2-with-registers (5 prefix tokens) and plain ViTs (1) both come out right.

🐍 Library

from vit_xray import inspect

res = inspect("photo.jpg", model="vit_base_patch16_clip_224.laion2b")

res.norm_map        # (H, W)
res.pca_rgb         # (H, W, 3)
res.attention       # (H, W), or None if the model has no CLS token
res.artifact_score  # float
res.plot()          # matplotlib Figure

🎛 Useful flags

What one run actually puts on screen

🔗 Complementary tools

vit-xray answers one question: are this backbone's patch features clean? For other kinds of looking-inside, use these instead, complementary, and deliberately not reimplemented here:

🔁 Reproducing the figures

The gallery config lives in the repo, so this one needs a checkout:

git clone https://github.com/MikeDegany/vit-xray && cd vit-xray
pip install -e ".[dev]"
vit-xray --gallery configs/gallery.yaml

Downloads the four COCO val2017 images into assets/ (gitignored), runs the zoo, and rewrites everything in docs/. Images are chosen for large low-information regions, sky, walls, flat ground, because that's where artifacts concentrate. Output is deterministic: same figures, pixel for pixel, every run.

@inproceedings{darcet2024vision,
  title     = {Vision Transformers Need Registers},
  author    = {Darcet, Timoth\'ee and Oquab, Maxime and Mairal, Julien and Bojanowski, Piotr},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2024}
}