pathvlm-litebench 0.14.1

A lightweight toolkit for benchmarking and visualizing vision-language models in computational pathology.

PathVLM-LiteBench

PathVLM-LiteBench is a lightweight, CPU-compatible and laptop-GPU accelerated toolkit for benchmarking and visualizing vision-language models in computational pathology under limited computing resources.

This project focuses on patch-level pathology image-text retrieval, zero-shot classification, prompt sensitivity analysis, embedding caching, and visualization reports using frozen CLIP/PLIP-style vision-language models, with CUDA acceleration when available.

_{Zero-shot text→patch retrieval with a frozen PLIP encoder — no fine-tuning — reproducible with pathvlm-litebench demo retrieval --model plip.
Patches: NCT-CRC-HE-100K / CRC-VAL-HE-7K by Kather, J. N. et al., via Zenodo (DOI 10.5281/zenodo.1214456), licensed under CC BY 4.0. Individual patches are unmodified; they are only arranged into a retrieval grid.}

TL;DR Quickstart

Install the toolkit and CLI straight from PyPI (CPU works out of the box; see the PyTorch note for CUDA):

pip install pathvlm-litebench
pathvlm-litebench --help        # list all commands

This gives you the pathvlm-litebench command and the importable library. To run the bundled demos, clone the repo instead — the example scripts ship with the source, not the PyPI wheel:

# 1. Clone and install in editable mode
git clone https://github.com/GuoYixuan0130/PathVLM-LiteBench.git
cd PathVLM-LiteBench
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e .

# 2. Run a demo (downloads CLIP weights on first run)
pathvlm-litebench demo retrieval --model clip --device auto

# 3. List everything else you can do
pathvlm-litebench demo          # list runnable demos

The first demo uses generated RGB images as a smoke test, so it runs on a laptop without a real dataset. Point it at your own patches with --image_dir path/to/your_patch_folder. The rest of this README covers datasets, configs, and reports in detail.

Motivation

Recent computational pathology foundation models and vision-language models have shown strong potential in histopathology image understanding. However, reproducing large-scale pretraining is computationally expensive and often infeasible for students or small research groups.

PathVLM-LiteBench aims to provide a low-compute and reproducible toolkit for evaluating and comparing pathology vision-language models using frozen models and lightweight utilities.

Instead of training large models from scratch, this project focuses on:

• Frozen vision-language model inference
• Patch-level image-text retrieval
• Zero-shot classification
• Prompt sensitivity analysis
• Embedding caching
• Visualization and HTML reporting
• CPU-compatible smoke tests and laptop-friendly experimentation
• CUDA-accelerated inference on consumer-grade laptop GPUs when available

The project does not require A100 GPUs, multi-GPU training, or large-scale VLM pretraining.

Current Stage

The current version supports a minimal but complete patch-level workflow:

• Loading patch images from a folder
• Encoding images with CLIP-style models
• Encoding text prompts with CLIP-style models
• Computing image-text similarity
• Retrieving top-k images for each prompt
• Saving top-k visualization grids
• Generating HTML retrieval reports
• Caching image embeddings
• Running zero-shot classification
• Running prompt sensitivity analysis
• Rendering patch-coordinate heatmaps from existing score artifacts
• Scoring coordinate-aware patch manifests against one text prompt
• Scoring coordinate-aware patch manifests against prompt sets
• Comparing saved prompt-scored patch-coordinate score artifacts

The current demos use simple RGB images for smoke testing. These demo images are not pathology images.

Model Registry

PathVLM-LiteBench uses a lightweight model registry so that demos can accept either a short model key or a Hugging Face model name.

In CLI commands, --model can be either:

• a registered model key, such as clip, plip, or conch
• a full Hugging Face model name, such as openai/clip-vit-base-patch32

The model key is the short name users type in PathVLM-LiteBench commands. The resolved model name is the actual model identifier used by the loader. For example, both clip and clip-vit-base-patch32 resolve to openai/clip-vit-base-patch32.

Currently supported model keys:

Model key	Resolved model name	Status	Notes
clip	openai/clip-vit-base-patch32	Implemented	Default CLIP baseline
clip-vit-base-patch32	openai/clip-vit-base-patch32	Implemented	Alias for CLIP ViT-B/32
plip	vinid/plip	Implemented	Pathology-specific CLIP-compatible PLIP wrapper
conch	MahmoodLab/CONCH	Optional	CONCH wrapper; requires gated Hugging Face access and the optional CONCH package

You can run demos with a registered model key:

python examples/01_patch_text_retrieval_demo.py --model clip

or with a Hugging Face model name:

python examples/01_patch_text_retrieval_demo.py --model openai/clip-vit-base-patch32

PLIP is supported through a CLIP-compatible Hugging Face wrapper. CONCH is supported through the official mahmoodlab/CONCH package and requires gated Hugging Face access. If the optional package or authentication is missing, --model conch raises a clear setup error.

To use CONCH locally, first install its optional package and authenticate with Hugging Face:

pip install git+https://github.com/Mahmoodlab/CONCH.git
hf auth login

Device Support

PathVLM-LiteBench supports explicit device selection for model-based demos:

python examples/01_patch_text_retrieval_demo.py --model clip --device auto
python examples/01_patch_text_retrieval_demo.py --model clip --device cuda
python examples/01_patch_text_retrieval_demo.py --model clip --device cpu

auto is the default mode. It uses CUDA if available and falls back to CPU otherwise.

This design keeps the toolkit compatible with CPU-only environments while allowing faster patch embedding on consumer-grade laptop GPUs.

The retrieval-metrics demo (examples/04_retrieval_metrics_demo.py) does not use a model and therefore does not need a --device argument.

Prompt Templates

PathVLM-LiteBench includes a small built-in prompt template library for common pathology concepts such as tumor, normal, necrosis, inflammation, stroma, lymphocyte, and gland.

These templates are intended for lightweight experimentation with zero-shot classification and prompt sensitivity analysis.

Example:

from pathvlm_litebench.prompts import get_prompt_variants, build_class_prompts

tumor_prompts = get_prompt_variants("tumor")
class_prompts = build_class_prompts(["tumor", "normal", "necrosis"])

The example above produces:

tumor_prompts == [
    "a histopathology image of tumor tissue",
    "a pathology patch showing malignant tissue",
    "a microscopic image of cancerous tissue",
    "a H&E stained tissue patch with tumor region",
]

class_prompts == [
    "a histopathology image of tumor",
    "a histopathology image of normal",
    "a histopathology image of necrosis",
]

get_prompt_variants("tumor") returns several alternative text prompts for the same concept. This is useful when you want to test whether retrieval results change when the wording changes.

build_class_prompts(["tumor", "normal", "necrosis"]) returns one default prompt per class. This is useful for zero-shot classification, where each patch image is compared against one text prompt for each candidate class.

The prompt library is not a clinical ontology. It is a lightweight research utility for reproducible prompt experiments.

Zero-shot with built-in class prompt generation:

python examples/02_zero_shot_classification_demo.py --model clip --device auto --class_names tumor normal necrosis --top_k 2

Here, --class_names tumor normal necrosis defines the candidate labels. Because no --class_prompts are provided, the demo automatically builds one prompt per class with the default template a histopathology image of {class_name}. --top_k 2 prints the two highest-scoring class predictions per patch.

Prompt sensitivity with pathology prompt templates:

python examples/03_prompt_sensitivity_demo.py --model clip --device auto --use_pathology_prompts --concepts tumor normal necrosis --top_k 2

Here, --use_pathology_prompts tells the demo to use the built-in prompt variants instead of a single custom prompt list. --concepts tumor normal necrosis selects which built-in concept groups to test. For each concept, the demo retrieves the top-k patches for several prompt variants, then compares whether those variants return similar patches.

If no real pathology patch folder is passed, the demos still use generated RGB images as smoke tests. For meaningful CPath experiments, provide --image_dir path/to/your_patch_folder.

Benchmark Configuration

PathVLM-LiteBench supports lightweight JSON configuration files for reproducible experiments.

Example retrieval config:

configs/retrieval_demo_config.json

Run retrieval demo with config:

python examples/01_patch_text_retrieval_demo.py --config configs/retrieval_demo_config.json

JSON config can store model, device, image_dir, prompts, top_k, cache, visualization, and report settings. Command-line arguments can override config values.

Zero-shot classification also supports config-driven runs:

python examples/02_zero_shot_classification_demo.py --config configs/zero_shot_demo_config.json

Prompt sensitivity analysis also supports config-driven runs:

python examples/03_prompt_sensitivity_demo.py --config configs/prompt_sensitivity_demo_config.json

Example config:

{
  "task": "prompt_sensitivity",
  "model": "clip",
  "device": "auto",
  "image_dir": "dataset/MHIST/images",
  "top_k": 5,
  "use_pathology_prompts": true,
  "concepts": ["tumor", "normal", "necrosis"],
  "save_report": true,
  "report_dir": "outputs/prompt_sensitivity_demo"
}

The dataset/ folder is local and ignored by Git. The config file is a template and does not include data.

Manifest-specific retrieval evaluation options such as --manifest, --image_root, --split, --label_prompts, and --recall_k are currently passed via CLI.

Common option meanings:

Option	Meaning
--image_dir	Folder of patch images for unlabeled runs.
--manifest	CSV file listing patch paths and optional labels/metadata.
--image_root	Root folder used to resolve relative image_path values in a manifest.
--split	Optional manifest subset to run, such as test.
--prompts	Text queries for retrieval.
--label_prompts	One target label per retrieval prompt, used to compute Recall@K.
--class_names	Candidate zero-shot class labels.
--class_prompts	One text prompt per class name. If omitted, default class prompts are generated.
--top_k	Number of top retrieval results or class predictions to show/save.
--recall_k	Retrieval metric cutoffs, such as Recall@1, Recall@5, and Recall@10.
--use_pathology_prompts	Use built-in prompt variants for prompt sensitivity.
--concepts	Built-in prompt concept groups to evaluate, such as tumor normal necrosis.
--dry-run	Validate and preview a planned run without model inference when supported.
--max-images	Limit the number of manifest records for a quick smoke run.

Patch Manifest Support

PathVLM-LiteBench can also read CSV manifests for real patch datasets.

Example manifest:

image_path,label,split,case_id
patches/patch_001.png,tumor,train,case_001
patches/patch_002.png,normal,train,case_001
patches/patch_003.png,necrosis,test,case_002

Example usage:

from pathvlm_litebench.data import load_patch_manifest, records_to_image_paths, get_unique_labels

records = load_patch_manifest("manifest.csv", image_root="path/to/dataset")
image_paths = records_to_image_paths(records)
labels = get_unique_labels(records)

The manifest loader does not load images or run models. It only prepares structured metadata for downstream retrieval, classification, and evaluation workflows.

Manifest Conversion

PathVLM-LiteBench uses a standard patch manifest format:

image_path,label,split,case_id,slide_id
patch_001.png,tumor,test,case_001,slide_001
patch_002.png,normal,test,case_001,slide_001

For command-line demos, users are encouraged to convert dataset-specific annotation files into this standard format.

Example: convert MHIST annotations:

pathvlm-litebench convert-manifest \
  --preset mhist \
  --input dataset/MHIST/annotations.csv \
  --output dataset/MHIST/manifest.csv \
  --image_root dataset/MHIST/images \
  --require_exists

Generic conversion:

pathvlm-litebench convert-manifest \
  --input annotations.csv \
  --output manifest.csv \
  --path_column "Image Name" \
  --label_column "Majority Vote Label" \
  --split_column "Partition"

Many public datasets (for example NCT-CRC-HE-100K) ship as a class-per-folder ImageFolder tree instead of an annotation CSV. Build a manifest directly from that tree:

pathvlm-litebench build-imagefolder-manifest \
  --image-dir dataset/NCT-CRC-HE-100K \
  --output dataset/NCT-CRC-HE-100K/manifest.csv

See docs/imagefolder_quickstart.md for the full "download a public dataset, build a manifest, run a benchmark" walkthrough.

The local dataset/ folder is intended for private local datasets and is ignored by Git.

Manifest Sampling

For quick low-compute experiments, users can create a smaller balanced manifest:

pathvlm-litebench sample-manifest \
  --input dataset/MHIST/manifest.csv \
  --output dataset/MHIST/manifest_test_50_per_class.csv \
  --split test \
  --samples_per_label 50 \
  --seed 42

This does not copy images. It only creates a smaller CSV manifest.

Then run retrieval or zero-shot evaluation with:

python examples/01_patch_text_retrieval_demo.py \
  --manifest dataset/MHIST/manifest_test_50_per_class.csv \
  --image_root dataset/MHIST/images \
  --model clip \
  --device auto

This is useful for laptop-friendly testing and balanced class evaluation. dataset/ is local and ignored by Git.

Repository Structure

PathVLM-LiteBench/
|-- pathvlm_litebench/
|   |-- config/
|   |-- data/
|   |-- evaluation/
|   |-- models/
|   |-- prompts/
|   |-- retrieval/
|   `-- visualization/
|-- examples/
|   |-- 01_quick_start.ipynb
|   |-- 01_patch_text_retrieval_demo.py
|   |-- 02_zero_shot_classification_demo.py
|   |-- 03_prompt_sensitivity_demo.py
|   |-- 04_retrieval_metrics_demo.py
|   `-- 05_patch_coordinate_heatmap_demo.py
|-- configs/
|-- docs/
|-- tests/
|-- README.md
`-- pyproject.toml

Installation

Clone the repository:

git clone https://github.com/GuoYixuan0130/PathVLM-LiteBench.git
cd PathVLM-LiteBench

Create and activate a virtual environment:

python -m venv .venv

# Windows
.venv\Scripts\activate

# Linux / macOS
source .venv/bin/activate

Install the package in editable mode. This installs all dependencies and the pathvlm-litebench command in one step:

pip install -e .

PyTorch note: pip install -e . installs a default torch build that works on CPU. For CUDA acceleration on a laptop GPU, install the matching torch wheel for your CUDA version first, following the official selector at https://pytorch.org/get-started/locally/, then run pip install -e ..

If you only need the libraries without the CLI, pip install -r requirements.txt also works.

Command-Line Interface

Once installed, you can inspect the toolkit with:

pathvlm-litebench version
pathvlm-litebench models
pathvlm-litebench demos
pathvlm-litebench demo
pathvlm-litebench convert-manifest --help
pathvlm-litebench build-imagefolder-manifest --help
pathvlm-litebench sample-manifest --help
pathvlm-litebench summarize-report --help
pathvlm-litebench compare-reports --help
pathvlm-litebench validate-config --help
pathvlm-litebench run-zero-shot-grid --help
pathvlm-litebench render-coordinate-heatmap --help
pathvlm-litebench score-coordinate-heatmap --help
pathvlm-litebench score-coordinate-heatmap-prompt-set --help
pathvlm-litebench compare-coordinate-heatmap-scores --help
pathvlm-litebench compare-models --help
pathvlm-litebench linear-probe --help

The inspection and help commands above do not download models. Model-based evaluation or scoring commands load model weights only when those runs are executed.

Data Preparation

To run PathVLM-LiteBench on real pathology patches, prepare a folder of patch-level images and pass it with --image_dir.

For the shortest MHIST-style workflow, see docs/small_dataset_quickstart.md. See docs/data_preparation.md for details. For a step-by-step guide on running the toolkit with real pathology patch folders, see docs/real_patch_workflow.md. For the shortest "download a public ImageFolder dataset, build a manifest, run a benchmark" path, see docs/imagefolder_quickstart.md. For an end-to-end public patch benchmark recipe using conversion, sampling, zero-shot, prompt grids, and report comparison, see docs/public_patch_benchmark_workflow.md. For patch-coordinate heatmaps from existing scores, see docs/patch_coordinate_heatmap_workflow.md. For single-prompt patch-coordinate heatmap scoring, see docs/prompt_scored_coordinate_heatmap_workflow.md. For prompt-set patch-coordinate heatmap scoring and comparison, see docs/prompt_set_coordinate_heatmap_workflow.md.

Quick Start

Demo 1: Patch-Text Retrieval

Run the minimal patch-text retrieval demo:

python examples/01_patch_text_retrieval_demo.py --model clip --device auto

If no image folder is provided, the script automatically creates a small demo folder with simple RGB images. These demo images are only used to verify that the pipeline works end-to-end.

Demo 2: Zero-Shot Classification

Run the zero-shot classification demo:

python examples/02_zero_shot_classification_demo.py --model clip --device auto

Run zero-shot evaluation from a CSV manifest:

python examples/02_zero_shot_classification_demo.py \
  --manifest path/to/manifest.csv \
  --image_root path/to/dataset_root \
  --model clip \
  --device auto \
  --split test \
  --top_k 3

If labels are available in the manifest, the demo reports:

• accuracy
• balanced accuracy
• macro precision / recall / F1
• per-class precision / recall / F1
• confusion matrix

If class_names are not provided, they can be inferred from manifest labels. This is intended for lightweight patch-level evaluation, not clinical diagnosis.

Example MHIST-style manifest evaluation command:

python examples/02_zero_shot_classification_demo.py \
  --manifest dataset/MHIST/manifest.csv \
  --image_root dataset/MHIST/images \
  --model clip \
  --device auto \
  --split test \
  --class_names HP SSA \
  --class_prompts \
    "a histopathology image of hyperplastic polyp" \
    "a histopathology image of sessile serrated adenoma" \
  --top_k 2

The current CLIP baseline is not pathology-specific, so low performance is expected and should be interpreted as a baseline.

This demo classifies each patch by comparing its image embedding with class text prompt embeddings.

To save predictions and metrics:

python examples/02_zero_shot_classification_demo.py \
  --manifest dataset/MHIST/manifest.csv \
  --image_root dataset/MHIST/images \
  --model clip \
  --device auto \
  --split test \
  --class_names HP SSA \
  --class_prompts \
    "a histopathology image of hyperplastic polyp" \
    "a histopathology image of sessile serrated adenoma" \
  --top_k 2 \
  --save_report \
  --report_dir outputs/zero_shot_demo

This generates:

• predictions.csv
• errors.csv
• metrics.json

predictions.csv stores per-image predictions. metrics.json stores aggregate classification metrics when labels are available.

When labels are available, --save_report also generates:

• errors.csv: misclassified samples only
• prediction distribution summary in metrics.json
• true label distribution
• predicted label distribution
• error rate
• optional warning when predictions collapse into one class

This is useful for detecting class bias in general CLIP baselines.

To generate a compact Markdown summary from a saved zero-shot report:

pathvlm-litebench summarize-report \
  --task zero-shot \
  --report_dir outputs/zero_shot_demo

This writes experiment_summary.md in the report directory. The summary is generated from saved artifacts only; it does not run model inference or download model weights.

Demo 3: Prompt Sensitivity Analysis

Run the prompt sensitivity demo:

python examples/03_prompt_sensitivity_demo.py --model clip --device auto

This demo evaluates whether different prompt variants for the same concept retrieve similar top-k image results.

To save prompt sensitivity results:

python examples/03_prompt_sensitivity_demo.py \
  --model clip \
  --device auto \
  --image_dir dataset/MHIST/images \
  --use_pathology_prompts \
  --concepts tumor normal necrosis \
  --top_k 5 \
  --save_report \
  --report_dir outputs/prompt_sensitivity_demo

This generates:

• prompt_sensitivity_summary.csv
• prompt_sensitivity_details.csv
• prompt_sensitivity_metrics.json

The summary CSV stores concept-level stability metrics. The details CSV stores prompt-level top-k retrieved indices and scores. The JSON stores full results and experiment metadata.

To generate a compact Markdown summary from a saved prompt sensitivity report:

pathvlm-litebench summarize-report \
  --task prompt-sensitivity \
  --report_dir outputs/prompt_sensitivity_demo

This writes experiment_summary.md in the report directory without rerunning model inference.

Prompt sensitivity analysis can also be configured with JSON:

python examples/03_prompt_sensitivity_demo.py --config configs/prompt_sensitivity_demo_config.json

Command-line arguments can override config values. For meaningful analysis, use real pathology patch images via --image_dir or config.image_dir.

Compare Saved Reports

After generating multiple reports for the same task, you can create a compact Markdown comparison without rerunning inference:

pathvlm-litebench compare-reports \
  --task zero-shot \
  --report_dirs outputs/zero_shot_clip outputs/zero_shot_plip \
  --run_names clip plip \
  --output outputs/zero_shot_comparison.md

The same command supports --task retrieval and --task prompt-sensitivity. The comparison reads saved CSV/JSON artifacts only and writes a local Markdown file. Do not commit generated comparisons if they contain dataset-specific paths or metrics.

Zero-Shot Prompt Grids

To run a small zero-shot prompt grid from JSON config:

pathvlm-litebench validate-config \
  configs/zero_shot_prompt_grid_mhist_sample.json

pathvlm-litebench run-zero-shot-grid \
  --config configs/zero_shot_prompt_grid_mhist_sample.json

Preview the expanded model/prompt runs without loading models:

pathvlm-litebench run-zero-shot-grid \
  --config configs/zero_shot_prompt_grid_mhist_sample.json \
  --dry-run

Each run writes a normal zero-shot report directory under the configured output_root. When save_comparison is enabled, the command also writes a zero-shot comparison Markdown file from saved artifacts. Generated reports stay under outputs/ and are ignored by Git.

For config fields, output layout, and interpretation guidance, see docs/prompt_grid_workflow.md.

Example Commands

Patch-text retrieval with custom prompts:

python examples/01_patch_text_retrieval_demo.py --model clip --device auto --prompts "a red image" "a blue image" "a black image" --top_k 2

Patch-text retrieval on your own patch image folder:

python examples/01_patch_text_retrieval_demo.py --model clip --device auto --image_dir path/to/your/patches --prompts "tumor region" "normal tissue" --top_k 5

Manifest-based patch-text retrieval with Recall@K:

python examples/01_patch_text_retrieval_demo.py \
  --manifest path/to/manifest.csv \
  --image_root path/to/dataset_root \
  --model clip \
  --device auto \
  --split test \
  --prompts \
    "a histopathology image of tumor tissue" \
    "a histopathology image of normal tissue" \
    "a histopathology image showing necrosis" \
  --label_prompts tumor normal necrosis \
  --recall_k 1 5 10 \
  --top_k 5 \
  --save_html_report

--label_prompts maps each text prompt to a manifest label. If labels are available, the demo prints text-to-image Recall@K for lightweight patch-level retrieval benchmarking.

When --label_prompts is used with a labeled manifest, retrieval results in the terminal and HTML report also include:

• the retrieved patch label
• the target label for each text prompt
• whether each retrieved patch is a positive match
• Recall@K metrics in the terminal

To save structured retrieval outputs:

python examples/01_patch_text_retrieval_demo.py \
  --manifest dataset/MHIST/manifest.csv \
  --image_root dataset/MHIST/images \
  --model clip \
  --device auto \
  --split test \
  --prompts \
    "a histopathology image of hyperplastic polyp" \
    "a histopathology image of sessile serrated adenoma" \
  --label_prompts HP SSA \
  --recall_k 1 5 10 \
  --top_k 5 \
  --save_html_report \
  --save_report \
  --report_dir outputs/retrieval_demo

This generates:

• retrieval_results.csv
• retrieval_metrics.json
• optionally retrieval_report.html

retrieval_results.csv stores prompt-level top-k retrieval results. retrieval_metrics.json stores Recall@K metrics and experiment metadata when labels are available. Outputs are ignored by Git.

To generate a compact Markdown summary from a saved retrieval report:

pathvlm-litebench summarize-report \
  --task retrieval \
  --report_dir outputs/retrieval_demo

This writes experiment_summary.md in the report directory without rerunning model inference.

Save top-k visualization grids:

python examples/01_patch_text_retrieval_demo.py --model clip --device auto --save_visualization

Generate an HTML retrieval report:

python examples/01_patch_text_retrieval_demo.py --model clip --device auto --save_html_report

Use image embedding cache:

python examples/01_patch_text_retrieval_demo.py --model clip --device auto --use_cache

Combine cache, visualization, and HTML report:

python examples/01_patch_text_retrieval_demo.py --model clip --device auto --use_cache --save_visualization --save_html_report --top_k 3

Zero-shot classification with custom pathology-style class prompts:

python examples/02_zero_shot_classification_demo.py --model clip --device auto --class_names tumor normal necrosis --class_prompts "a histopathology image of tumor tissue" "a histopathology image of normal tissue" "a histopathology image showing necrosis" --top_k 2

Zero-shot classification with built-in class prompt generation:

python examples/02_zero_shot_classification_demo.py --model clip --device auto --class_names tumor normal necrosis --top_k 2

Prompt sensitivity with a different top-k value:

python examples/03_prompt_sensitivity_demo.py --model clip --device auto --top_k 2

Prompt sensitivity with pathology prompt templates:

python examples/03_prompt_sensitivity_demo.py --model clip --device auto --use_pathology_prompts --concepts tumor normal necrosis --top_k 2

Example Output

The retrieval demo prints top-k results for each text prompt:

========== Retrieval Results ==========

Prompt: a red image
  Top 1: index=0, score=0.2841, path=examples/demo_patches/patch_red.png
  Top 2: index=3, score=0.2317, path=examples/demo_patches/patch_green.png

The zero-shot demo prints predicted labels:

========== Zero-Shot Classification Results ==========

Image: examples/demo_patches/patch_red.png
Predicted: red (confidence=0.3012)
Top predictions:
  Top 1: class=red, probability=0.3012, logit=0.2841

The prompt sensitivity demo prints concept-level stability metrics:

========== Prompt Sensitivity Results ==========

Concept: red_like
Number of prompts: 3
Mean top-k overlap: 0.6667
Mean similarity std: 0.0312

Exact scores may vary depending on model version and runtime environment.

Results on real pathology data

Beyond the synthetic smoke test, here is what frozen vision-language models produce on a balanced 108-patch sample of the public NCT-CRC-HE colorectal histology dataset (9 tissue classes, no fine-tuning). Every model sees the same images and the same prompt template, so the numbers are directly comparable.

Domain pretraining is the whole story here: general-domain CLIP lands at 22% with an identical prompt, while the pathology vision-language models PLIP (60%) and CONCH (59%) roughly triple it — all well above the 1/9 = 11% random baseline, with no fine-tuning. Zero-shot accuracy is also prompt-sensitive, so a single shared template (an H&E image of {class}.) is used for every model to keep the comparison fair.

This chart is reproducible from the command line: compare-models runs the same frozen zero-shot benchmark and writes the bar chart, an overall-accuracy CSV, a per-class-accuracy CSV, and run metadata. Because a 108-patch sample is small, each accuracy now ships with a 95% bootstrap confidence interval (drawn as error bars on the chart and recorded in the CSV and metadata), and the metadata records the exact torch/transformers/numpy versions used so a run can be tied to its software stack. Tune the interval with --confidence, --bootstrap-resamples, and --seed.

pathvlm-litebench compare-models \
  --manifest dataset/CRC_VAL_HE_100_sample_manifest.csv \
  --models clip plip conch \
  --class-names "adipose tissue" background debris lymphocytes mucus \
    "smooth muscle" "normal colon mucosa" "cancer-associated stroma" \
    "colorectal adenocarcinoma epithelium" \
  --output-dir outputs/model_comparison

Looking at the strongest zero-shot model (PLIP) in more detail:

The per-class confusion matrix (image-text similarity, argmax over the 9 tissue prompts) reaches 60% overall accuracy with no training. The diagonal is strong for distinct tissues (adipose, background, lymphocytes, tumor); the off-diagonal mass honestly shows where a frozen encoder struggles, e.g. mucus confused with cancer-associated stroma.

A t-SNE projection of the same frozen embeddings shows that patches cluster by tissue type without any supervision — adipose, background, debris, tumor, and lymphocyte clusters are clearly separated, while the muscle/stroma classes overlap as expected for visually similar tissue.

_{Patches: NCT-CRC-HE-100K / CRC-VAL-HE-7K by Kather, J. N. et al., via Zenodo (DOI 10.5281/zenodo.1214456), licensed under CC BY 4.0. Figures are generated from unmodified patches.}

Linear Probe

Zero-shot accuracy depends on the prompt, and on a small sample it can understate how much usable signal a frozen encoder actually carries. The linear-probe command answers a complementary question: how linearly separable are the frozen features already? It trains only a logistic-regression classifier on top of the frozen image embeddings — the encoder itself is never touched — so it stays laptop-friendly while giving a fairer, prompt-free read on feature quality. This is the standard low-compute step beyond zero-shot in computational pathology.

The command reads a standard patch manifest with image_path, label, and split columns, encodes the train and test splits with the chosen frozen model, fits the probe on the train split, and evaluates on the test split. The bundled CRC_VAL_HE_100_sample_manifest.csv has only image_path and label, so point the probe at a manifest that carries a split column with train/test values:

pathvlm-litebench linear-probe \
  --manifest dataset/CRC_VAL_HE_100_split_manifest.csv \
  --model plip \
  --train-split train \
  --test-split test \
  --output-dir outputs/linear_probe

It writes predictions.csv, errors.csv, and metrics.json (the same report format as the zero-shot command). Accuracy ships with a 95% bootstrap confidence interval, and the metadata records the probe configuration (logistic_regression, C, max_iter, normalize, seed) plus the exact torch/transformers/scikit-learn versions used. Tune the classifier with --C, --max-iter, and --no-normalize; tune the interval with --confidence, --bootstrap-resamples, and --seed. Add --dry-run to validate the manifest and report split sizes without loading any model.

Outputs

Generated outputs are saved under outputs/ by default and are ignored by Git.

Typical outputs include:

outputs/
|-- cache/
|   |-- image_embeddings.pt
|   `-- image_paths.json
`-- retrieval_demo/
    |-- retrieval_report.html
    `-- topk_prompt_*.png

The automatically generated demo images are saved under:

examples/demo_patches/

This folder is also ignored by Git.

Low-Compute Design

This project is designed for laptop-friendly experimentation.

Current design choices:

• Use frozen CLIP-style models instead of training large models
• Start from patch-level images instead of full whole-slide image processing
• Avoid heavy dependencies such as OpenSlide and TIAToolbox in the early stage
• Cache image embeddings to reduce repeated computation
• Keep demos simple, reproducible, and easy to inspect
• Provide terminal outputs, image grids, and HTML reports
• Run smoke tests on CPU-only environments
• Accelerate model inference with CUDA on consumer-grade laptop GPUs when available
• Use default --device auto mode to choose CUDA when available and otherwise fall back to CPU

Current Limitations

• CLIP remains the default baseline model. PLIP is available as a pathology-specific CLIP-compatible model key.
• CONCH requires gated Hugging Face access and the optional mahmoodlab/CONCH package.
• The built-in demo images are not pathology images.
• Whole-slide image processing is not supported in the current version.
• No large-scale benchmark dataset is included.
• Prompt sensitivity analysis currently focuses on retrieval stability rather than clinical validity.

Future milestones may add more packaged patch-level benchmark examples, but the current release remains patch-level and local-first.

Roadmap

• Build basic project structure
• Implement a CLIP model wrapper
• Support patch image loading
• Implement image-text retrieval
• Add minimal command-line retrieval demo
• Add top-k image visualization
• Add embedding caching
• Add HTML retrieval report
• Add zero-shot classification utility
• Add zero-shot classification demo
• Add prompt sensitivity analysis utility
• Add prompt sensitivity demo
• Add lightweight model registry
• Add manifest conversion and balanced sampling utilities
• Add retrieval metrics such as Recall@K
• Add classification metrics beyond accuracy
• Add config-driven retrieval, zero-shot, and prompt sensitivity demos
• Add pathology-specific PLIP wrapper
• Add zero-shot, retrieval, and prompt sensitivity experiment summary Markdown utilities
• Add multi-run comparison summaries for saved report artifacts
• Implement optional CONCH wrapper
• Add zero-shot prompt-grid runner
• Document public patch benchmark workflow
• Add benchmark config validation CLI
• Add patch-coordinate heatmap rendering from existing score artifacts
• Add single-prompt patch-coordinate heatmap scoring and config workflow
• Add artifact-only score comparison for prompt-scored coordinate heatmaps
• Add prompt-set patch-coordinate heatmap scoring workflow
• Keep public documentation focused on user workflows

Related docs:

• Getting started: small dataset quickstart, ImageFolder quickstart, real patch workflow, data preparation
• Benchmark workflows: public patch benchmark workflow, prompt-grid workflow
• Coordinate heatmaps: artifact-only heatmap workflow, single-prompt scoring workflow, prompt-set workflow
• Project and release: project positioning, v0.11.0 release notes

Research Positioning

PathVLM-LiteBench is intended as a lightweight research engineering project for computational pathology and medical vision-language model evaluation.

The project does not aim to reproduce large-scale foundation model pretraining. Instead, it focuses on reproducible low-compute evaluation workflows that can help students and small research groups analyze pathology vision-language models.

For a more detailed discussion of the research motivation and scope, see docs/project_positioning.md.

Contributing

Contributions are welcome. Please see CONTRIBUTING.md for development principles, testing instructions, and contribution guidelines.

License

PathVLM-LiteBench is released under the MIT License. It is developed for academic and research use, but the MIT terms also permit commercial use, modification, and redistribution with attribution.

Note that model weights (CLIP, PLIP, CONCH) and any datasets you use are covered by their own separate licenses and access terms.