kaiko-eva 0.4.3

Evaluation Framework for oncology foundation models.

Oncology FM Evaluation Framework by kaiko.ai

Installation • How To Use • Quick Start • Documentation • Datasets • Benchmarks
Contribute • Acknowledgements

eva is an evaluation framework for oncology foundation models (FMs) by kaiko.ai. Check out the documentation for more information.

Highlights:

• Easy and reliable benchmark of Oncology FMs
• Supports patch-level classification, slide-level classification, semantic segmentation, and (visual) question answering tasks.
• Automatic embedding inference and evaluation of a downstream task
• Native support of popular medical datasets and models
• Produce statistics over multiple evaluation fits and multiple metrics

Installation

Simple installation from PyPI:

# to install the core version only
pip install kaiko-eva

# to install the expanded `vision` version
pip install 'kaiko-eva[vision]'

# to install the expanded `language` version
pip install 'kaiko-eva[language]'

# to install the expanded `multimodal` version
pip install 'kaiko-eva[multimodal]'

# to install everything
pip install 'kaiko-eva[all]'

To install the latest version of the main branch:

pip install "kaiko-eva[all] @ git+https://github.com/kaiko-ai/eva.git"

You can verify that the installation was successful by executing:

eva --version

How To Use

eva can be used directly from the terminal as a CLI tool as follows:

eva {fit,predict,predict_fit} --config url/or/path/to/the/config.yaml 

eva uses jsonargparse to make it easily configurable by automatically generating command line interfaces (CLIs), which allows to call any Python object from the command line. Moreover, the configuration structure is always in sync with the code. Thus, eva can be used either directly from Python or as a CLI tool (recommended).

For more information, please refer to the documentation.

Learn about Configs

The following interfaces are identical:

Python interface

Configuration file

# main.py # execute with: `python main.py` from torch import nn from eva import core from eva.vision import datasets, transforms # initialize trainer trainer = core.Trainer(max_steps=100) # initialize model model = core.HeadModule( backbone=nn.Flatten(), head=nn.Linear(150528, 4), criterion=nn.CrossEntropyLoss(), ) # initialize data data = core.DataModule( datasets=core.DatasetsSchema( train=datasets.BACH( root="data/bach", split="train", download=True, transforms=transforms.ResizeAndCrop(), ), ), dataloaders=core.DataloadersSchema( train=core.DataLoader(batch_size=32), ), ) # perform fit pipeline = core.Interface() pipeline.fit(trainer, model=model, data=data)

# main.yaml # execute with: `eva fit --config main.yaml` --- trainer: class_path: eva.Trainer init_args: max_steps: 100 model: class_path: eva.HeadModule init_args: backbone: torch.nn.Flatten head: class_path: torch.nn.Linear init_args: in_features: 150528 out_features: 4 criterion: torch.nn.CrossEntropyLoss data: class_path: eva.DataModule init_args: datasets: train: class_path: eva.vision.datasets.BACH init_args: root: ./data/bach split: train download: true transforms: eva.vision.transforms.ResizeAndCrop dataloaders: train: batch_size: 32

The .yaml file defines the functionality of eva by parsing and translating its content to Python objects directly. Native supported configs can be found at the configs directory of the repo, which can be both locally stored or remote.

Quick Start

We define two types of evaluations: online and offline. While online fit uses the backbone (FM) to perform forward passes during the fitting process, offline fit first generates embeddings with the backbone and then fits the model using these embeddings as input, resulting in a faster evaluation.

Here are some examples to get you started:

• Perform a downstream offline classification evaluation of DINO ViT-S/16 on the BACH dataset with linear probing by first pre-calculating the embeddings:

  DOWNLOAD_DATA=true \
  MODEL_NAME=universal/vit_small_patch16_224_dino \
  eva predict_fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/pathology/offline/classification/bach.yaml
  

• Perform a downstream online segmentation evaluation of DINO ViT-S/16 on the MoNuSAC dataset with the ConvDecoderWithImage decoder:

  DOWNLOAD_DATA=true \
  MODEL_NAME=universal/vit_small_patch16_224_dino \
  eva fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/pathology/online/segmentation/monusac.yaml
  

By default eva will perform 5 evaluation runs using different seeds, however, you can control the number of runs through the N_RUNS environment variable or in the configuration file. The results will be saved to ./logs by default, or to OUTPUT_ROOT if specified.

For more examples, take a look at the configs and tutorials.

[!NOTE] All the datasets that support automatic download in the repo have by default the option to automatically download set to false. For automatic download you have to manually set the environment variable DOWNLOAD_DATA=true or in the configuration file download=true.

Leaderboards

The following table shows the FMs we have evaluated with eva. For more detailed information about the evaluation process, please refer to our documentation.

Pathology

Radiology

Contributing

eva is an open source project and welcomes contributions of all kinds. Please checkout the developer and contributing guide for help on how to do so.

All contributors must follow the code of conduct.

Acknowledgements

Our codebase is built using multiple opensource contributions

Citation

If you find this repository useful, please consider giving a star ⭐ and adding the following citation:

@inproceedings{kaiko.ai2024eva,
    title={eva: Evaluation framework for pathology foundation models},
    author={kaiko.ai and Ioannis Gatopoulos and Nicolas K{\"a}nzig and Roman Moser and Sebastian Ot{\'a}lora},
    booktitle={Medical Imaging with Deep Learning},
    year={2024},
    url={https://openreview.net/forum?id=FNBQOPj18N}
}