wsiprocess 0.4

Whole Slide Image (WSI) Processing Library

Convert Helper for Histopathological / Cytopathological Machine Learning Tasks

Main Flow

1. Scan some WSIs.
2. Make some annotations with WSI annotation tools. (Recommended: WSIDissector(coming soon), ASAP is now available.)
3. Then wsiprocess helps converting WSI + Annotation data into patches and easy-to-use annotation data.

WSIPatcher will give you GUI.

Installation

pip User

1. Install libvips

   • Linux - apt install libvips
   • MacOS - brew install vips
   • Windows - Install tarball from here

2. Install wsiprocess

   pip install wsiprocess
   

Anaconda User

# Only for python 3.6 or higher
conda install -c tand826 wsiprocess

Documentation

• Documentation

Example

As a python module

Basic Usage

import wsiprocess as wp
slide = wp.slide("xxx.tiff")
annotation = wp.annotation("xxx.xml")
rule = wp.rule("xxx.json")

annotation.make_masks(slide, rule, foreground=True)

patcher = wp.patcher(slide, "classification", annotation)
patcher.get_patch_parallel(["benign", "malignant"])

Export annotaton xml of one class as mask image

import wsiprocess as wp
slide = wp.slide("xxx.tiff")
annotation = wp.annotation("xxx.xml")
annotation.make_masks(slide)
annotation.export_mask("xxx/masks", "benign")

Export annotation xml with inclusion definition as mask images, and save their thumbs

import wsiprocess as wp
slide = wp.slide("xxx.tiff")
annotation = wp.annotation("xxx.xml")
rule = wp.rule("xxx.json")
annotation.make_masks(slide, rule)
annotation.export_thumb_masks("xxx/masks")

Load mask data from image, and extract patches.

import wsiprocess as wp
import cv2
slide = wp.slide("xxx.tiff")
annotation = wp.annotation("", is_image=True)
target_classes = ["benign", "malignant"]
annotation.add_class(target_classes)
benign_mask = cv2.imread("benign_mask.png", 0) * 255
malignant_mask = cv2.imread("malignant_mask.png", 0) * 255
# Make sure your mask data includes only 0 as background or 255 as foreground
annotation.from_image(benign_mask, "benign")
annotation.from_image(malignant_mask, "malignant")
rule = wp.rule("xxx.json")
patcher = wp.patcher(slide, "classification", annotation=annotation)
patcher.get_patch_parallel(target_classes)

As a command line tool (recommended)

basic flow

wsiprocess [your method] xxx.tiff --annotation xxx.xml

Extract patches with mask of foreground area. The mask has pixels with 1 as foreground which are originally from 10 to 230 in the scale of 0-255, and pixels with 0 as background which are originally from 0 to 10 and from 230 to 255.

wsiprocess none xxx.tif -oa 0.01 -mm 10-230 -of 0.01 -ep

• Need recommendation for choice of arguments? Type wsiprocess -h or wsiprocess [your method] -h to see options.

As a docker command line tool (not working on)

# build the image
docker build . -t wsiprocess_image

# run the container
docker run --name wsiprocess_container -v [your files directory]:/data -it -d wsiprocess_image [commands] etc.

see Command Helper for commands

Convert to VOC / COCO / YOLO style format

# If already extracted patches...
# to COCO format
python wsiprocess/converters/wsiprocess_to_coco.py [root of wsiprocess] -s [directory to save to] -r [ratio of train, val and test]

# to VOC foramt
python wsiprocess/converters/wsiprocess_to_voc.py [root of wsiprocess] -s [directory to save to] -r [ratio of train, val and test]

# to YOLO format
python wsiprocess/converters/wsiprocess_to_yolo.py [root of wsiprocess] -s [directory to save to] -r [ratio of train, val and test]

# If not extracted patches yet...
# convert to VOC and COCO and YOLO
wsiprocess xxx.tiff method --annotation xxx.xml -vo -co -yo

Available WSIs

Test ongoing

• From below we tested wsi data.

  • :smile: => worked well.
  • :umbrella: => did not work well.
  • otherwise => did not check

Classification

• Aperio

  • CMU-1-JP2K-33005.svs
  • :smile: CMU-1-Small-Region.svs
  • :smile: CMU-1.svs
  • CMU-2.svs
  • CMU-3.svs
  • JP2K-33003-1.svs
  • JP2K-33003-2.svs

• Generic-TIFF

  • :umbrella:CMU-1.tiff
    • Can not set magnification.

• Hamamatsu-vms

  • :smile:CMU-1.zip
  • CMU-2.zip
  • CMU-3.zip
    • Could not DOWNLOAD from http://openslide.cs.cmu.edu/download/openslide-testdata/Hamamatsu-vms/

• Hamamatsu

  • :smile:CMU-1.ndpi
  • CMU-2.ndpi
  • CMU-3.ndpi
  • OS-1.ndpi
  • OS-2.ndpi
  • OS-3.ndpi

• Leica

  • :smile:Leica-1.scn
  • Leica-2.scn
  • Leica-3.scn
  • Leica-Fluorescence-1.scn

• Mirax

  • CMU-1-Exported.zip
  • CMU-1-Saved-1_16.zip
  • CMU-1-Saved-1_2.zip
  • :umbrella:CMU-1.zip
    • Can not make the foreground mask.
  • CMU-2.zip
  • CMU-3.zip
  • Mirax2-Fluorescence-1.zip
  • Mirax2-Fluorescence-2.zip
  • Mirax2.2-1.zip
  • Mirax2.2-2.zip
  • Mirax2.2-3.zip
  • Mirax2.2-4-BMP.zip
  • Mirax2.2-4-PNG.zip

• Olympus

  • OS-1.zip
  • OS-2.zip
  • OS-3.zip

• Trestle

  • :umbrella:CMU-1.zip
    • ASAP can not show the image properly, and it's hard to annotate.
  • CMU-2.zip
  • CMU-3.zip

• Ventana

  • OS-1.bif
  • OS-2.bif

• :umbrella:Zeiss : Can not load slide - :umbrella:Zeiss-1-Merged.zvi - :umbrella:Zeiss-1-Stacked.zvi - :umbrella:Zeiss-2-Merged.zvi - :umbrella:Zeiss-2-Stacked.zvi - :umbrella:Zeiss-3-Mosaic.zvi - :umbrella:Zeiss-4-Mosaic.zvi

Test

Download sample WSI

curl -O -C - http://openslide.cs.cmu.edu/download/openslide-testdata/CMU-1.ndpi

Make random annotation

• Install ASAP ( Linux / Windows ) - https://github.com/computationalpathologygroup/ASAP/releases
• Open CMU-1.ndpi and make some random annotation. - Save the annotation xml as "CMU-1.xml".

Run test.py

cd tests
pytest tests.py

Citation

@misc{https://doi.org/10.5281/zenodo.3887428,
  doi = {10.5281/ZENODO.3887428},
  url = {https://zenodo.org/record/3887428},
  author = {{Takumi Ando}},
  title = {tand826/wsiprocess: version 0.4},
  publisher = {Zenodo},
  year = {2020},
  copyright = {Open Access}
}