vlm-eval 0.1.0

Evaluation harness for Vision-Language Models on non-standard axes (calibration under distribution shift, and beyond)

vlm-eval

A lightweight evaluation harness for Vision-Language Models (VLMs) on axes that mainstream benchmarks don't cover. First task module: calibration under distribution shift — how well a model's confidence tracks its actual correctness as images are blurred or corrupted with noise.

Quickstart

uv sync --all-extras --dev
uv run vlm-eval run --task calibration_shift --model mock

This runs the calibration_shift task against the deterministic MockVLMWrapper over the tiny fixture dataset in tests/fixtures/, and writes report/leaderboard.md + report/leaderboard.json.

Add --shift blur or --shift noise to evaluate under a corruption.

Design

• vlm_eval/models/ — thin wrappers around real VLM backends (transformers, open_clip), all implementing one VLMWrapper interface so tasks and scorers are model-agnostic.
• vlm_eval/tasks/ — a task pairs a dataset with a prompt template and a scorer.
• vlm_eval/scorers/ — accuracy, Expected Calibration Error (ECE), Brier score.
• vlm_eval/data/ — dataset loading and distribution-shift transforms.
• vlm_eval/report/ — markdown/JSON leaderboard output.

CI runs entirely offline against MockVLMWrapper and synthetic fixture images — no real model downloads or GPU required. Running real VLMs against real datasets is a manual/local workflow (see models/clip_wrapper.py).

First real result: CLIP

ClipWrapper (vlm_eval/models/clip_wrapper.py) wraps open_clip's ViT-B-32-quickgelu / openai checkpoint and does standard zero-shot classification — cosine similarity between the image embedding and a "a photo of a {label}" text embedding per candidate label, softmaxed into a probability distribution. It ignores the free-form prompt argument (built for instruction-following models); CLIP compares an image against labels directly.

uv sync --extra clip --dev
uv run vlm-eval run --task calibration_shift --model clip
uv run vlm-eval run --task calibration_shift --model clip --shift noise

On the tiny synthetic fixture set (solid dark/light gray squares):

model	shift	accuracy	ece	brier_score
clip	none	0.5000	0.4188	0.7756
clip	blur	0.5000	0.4188	0.7756
clip	noise	1.0000	0.3365	0.2786

Blur is a no-op on uniform-color squares, as expected. CLIP essentially guesses on "dark"/"light" gray squares — they're not natural photos, so "a photo of a dark" isn't a meaningful query — which is itself a useful data point: the harness surfaces a real, non-trivial calibration gap (50% accuracy, but a non-zero ECE means its confidence still doesn't track that chance-level correctness). A richer, more photographic fixture set would give a more representative result; this one mainly demonstrates the harness computing real metrics end-to-end against a real model, not a benchmark claim about CLIP itself.

Development

uv sync --all-extras --dev
uv run pytest tests/unit tests/integration
uv run ruff check .
uv run black --check .
uv run mypy vlm_eval/