histotuner 0.2.8

Add your description here

histotuner

Supported token-extraction backends

histotuner can append multiple model-specific token tables into the same SpatialData Zarr while keeping shared geometry layers model-agnostic.

Currently supported token extractors:

• hf-hub:bioptimus/H-optimus-1
• hf-hub:MahmoodLab/UNI2-h
• hf-hub:paige-ai/Virchow2
• hf-hub:Wangyh/mSTAR
• hf-hub:prov-gigapath/prov-gigapath
• owkin/phikon-v2
• MahmoodLab/conchv1_5
• WenchuanZhang/Patho-CLIP-L
• majiabo/GPFM

Token-grid semantics

All currently supported models export a unified 14x14 token grid so token tables can be compared directly across models.

• phikon-v2 exports a native 14x14 patch-token grid.
• hf-hub:bioptimus/H-optimus-1, hf-hub:Wangyh/mSTAR, and hf-hub:prov-gigapath/prov-gigapath export native 14x14 grids.
• hf-hub:MahmoodLab/UNI2-h and hf-hub:paige-ai/Virchow2 have native 16x16 patch-token grids after special tokens are stripped, and histotuner adaptively average-pools them to 14x14.
• conchv1_5 is special:
  • the native vision encoder runs at 448x448 with patch16
  • that produces a native 28x28 patch-token grid
  • histotuner average-pools each non-overlapping 2x2 token neighborhood to export a compatibility 14x14 token grid
• Patho-CLIP-L is also special:
  • the native CLIP-L/14 vision encoder produces a 24x24 patch-token grid at 336x336 input resolution
  • histotuner adaptively average-pools that native 24x24 grid to export a compatibility 14x14 token grid
• GPFM is also special:
  • the native DINOv2 ViT-L/14 encoder produces a 16x16 patch-token grid at 224x224 input resolution
  • histotuner adaptively average-pools that native 16x16 grid to export a compatibility 14x14 token grid

That pooling choice is deliberate so downstream single-cell workflows can consume every supported model through the same 14x14 token layout. For the pooled models, this is a compatibility semantic rather than the model's native tokenization:

• UNI2-h and Virchow2: pooled from native 16x16
• conchv1_5: pooled from native 28x28
• Patho-CLIP-L: pooled from native 24x24
• GPFM: pooled from native 16x16

Not yet supported for token extraction

• none from the current requested set

O2 batch job generation

To generate one embedder.yaml and one embed_cluster.sh per sample folder on O2:

python generate_o2_jobs.py \
  --root-dir /n/scratch/users/a/ajn16/histotuner/full \
  --template-yaml embedder.yaml \
  --template-shell embed_cluster.sh \
  --output-dir /n/scratch/users/a/ajn16/histotuner/generated_jobs

  python generate_o2_jobs.py \
  --root-dir /n/scratch/users/a/ajn16/histotuner/heonly \
  --template-yaml embedder_HEonly.yaml \
  --template-shell embed_cluster_HEonly.sh \
  --output-dir /n/scratch/users/a/ajn16/histotuner/generated_jobs

To preview the sbatch submissions for the generated job scripts:

python submit_generated_jobs.py \
  --generated-dir /n/scratch/users/a/ajn16/histotuner/generated_jobs \
  --dry-run