vlm-occlusion

Find which image regions influence a Vision-Language Model claim through black-box occlusion testing.

Installation

pip install vlm-occlusion

Usage

from vlm_occlusion import analyze_occlusion

result = analyze_occlusion(
    image="example.png",
    predict_fn=describe_image,
    score_fn=claim_score,
    grid_size=(4, 4),
    occlusion="blur",
    output_path="heatmap.png",
)

print(result["most_influential_region"])
print(result["importance_scores"])

predict_fn receives a Pillow image and returns text or a number. score_fn turns that result into a numerical score for the claim you are testing. The package compares the original score with scores produced after occluding each grid cell.

It is model-agnostic and includes no VLM, model adapter, or network client. It measures black-box sensitivity to controlled perturbations; it does not inspect attention or reveal a model's internal reasoning.

Example

The included deterministic example tests the claim:

There is a red desk lamp in the upper-left area.

The mock predictor measures visible red evidence; no model, API key, or network connection is used. The 4×4 analysis identifies the left-side cell containing most of the lamp as the strongest influence.

Input image	Most influential cell occluded	Importance heatmap

The scene is an original generated illustration created for this repository. The occluded image and heatmap are produced locally by the example.

python examples/generate_example_images.py
python examples/basic_usage.py

The README assets are saved under docs/images/.

Features

• Model-agnostic callback API
• Grid-based black, white, mean, and blur occlusion
• Normalized regional importance scores
• Pillow-only visualization
• No network access required

Limitations

Occlusion sensitivity is an approximation. A perturbation may change a response for reasons unrelated to visual grounding, and results can vary with grid size and occlusion strategy. This package does not expose internal model attention.

Issues

Report issues at https://github.com/edujbarrios/vlm-occlusion/issues.

Author

Eduardo J. Barrios — edujbarrios@outlook.com

License

Mozilla Public License 2.0