VLM Prior Probe

Measure when a vision-language model follows visual evidence versus learned priors.

VLM Prior Probe is a lightweight, provider-independent toolkit for black-box counterfactual evaluation of vision-language models.

No model training. No model-specific integration. No hidden-state access. No LLM judge required.

Installation

pip install vlm-prior-probe

Quick start

from vlm_prior_probe import evaluate

samples = [{
    "id": "banana-001", "image": "blue_banana.png",
    "prompt": "What color is the banana?", "attribute": "color",
    "visual_value": "blue", "prior_value": "yellow",
}]

def target(image, prompt):
    return my_vlm(image=image, prompt=prompt)

report = evaluate(samples, target=target)
print(report["visual_adoption_rate"])
print(report["prior_persistence_rate"])

Why VLM Prior Probe?

A plausible VLM answer may follow a canonical association instead of conflicting visual evidence. This package turns controlled interventions into repeatable behavioral measurements without a server, database, provider SDK, or runtime dependency.

Core idea

Give the model a controlled disagreement—such as a blue banana against the canonical association banana → yellow—and classify its answer as visual, prior, mixed, other, or unclear. Unmatched answers are conservatively unclear; other requires sample-supplied other_values.

Sample format

Required keys are id, prompt, image, attribute, visual_value, and prior_value. Optional visual_aliases, prior_aliases, other_values, category, original_image, counterfactual_image, blank_image, unrelated_image, and prompts add controlled vocabulary or interventions.

Single-sample probing

from vlm_prior_probe import probe

result = probe(samples[0], target=target)
print(result["executed_conditions"])
print(result["classification"])

The visual and text-only conditions run by default. Optional images run when present. Disable image-less execution with text_only=False. Target failures are recorded; use on_error="raise" for fail-fast behavior.

Dataset evaluation

evaluate(samples, target=target) returns JSON-serializable results, overall rates, and breakdowns by attribute and optional category. A valid sample has a successful, non-empty string response in its primary visual condition.

Prior Persistence Rate

prior_persistence_rate is the fraction of valid controlled cases whose primary response matches the canonical prior. It measures prior-consistent behavior, not its internal cause.

Visual Adoption Rate

visual_adoption_rate is the fraction of valid controlled cases whose primary response matches the actual visual value. visual_prior_balance is visual adoption minus prior persistence, a descriptive—not universal—summary from -1 to +1.

Counterfactual Sensitivity

For samples with original_image and counterfactual_image, sensitivity is true when the original response matches prior_value and the changed image response matches visual_value. The rate uses paired cases in which both calls are valid. Visual reliance similarly counts valid pairs where the primary condition is visual and text-only is prior.

Model comparison

from vlm_prior_probe import compare

comparison = compare(samples, targets={"base": base_target, "post-trained": post_target})

Differences are reported as right minus left. Comparing checkpoints can reveal whether prior persistence changed after a training stage; it does not attribute the behavior's origin.

Confidence intervals

Use evaluate(..., bootstrap=1000, seed=42) for deterministic 95% percentile bootstrap intervals. Resampling uses random.Random(seed) and the standard library only. compare also returns paired bootstrap intervals for visual-adoption and prior-persistence differences.

Supported attributes

The deterministic method is intended for controlled values such as color, count, shape, orientation, size, material, state, and spatial relation. Supply experiment-specific aliases; there is intentionally no large synonym database.

Use cases

VLM evaluation, model or checkpoint comparison, robustness and regression testing, benchmark auditing, modality-reliance analysis, and controlled counterfactual research.

Methodology

Matching applies Unicode, case, punctuation, and whitespace normalization followed by whole-phrase matching. Responses containing both controlled values are mixed; neither value is never forced into visual or prior. Library-side results are deterministic for fixed inputs, outputs, configuration, and seed, though provider calls may not be.

Limitations

VLM Prior Probe measures observable behavior under controlled interventions. It does not identify the internal mechanism responsible for a response and does not by itself determine whether a behavior originates from pretraining, fine-tuning, alignment, or another training stage.

A prior-consistent answer does not automatically prove that the image was ignored. Results depend on stimulus quality, prompts, aliases, target nondeterminism, and whether controlled values adequately represent the task. The package does not generate images, judge arbitrary semantics, or perform mechanistic interpretability.

Issues

Report issues in the GitHub issue tracker.

Author

Eduardo J. Barrios — edujbarrios@outlook.com

License

Mozilla Public License 2.0