kaiko-eva 0.0.2

Evaluation Framework for oncology foundation models.

Oncology FM Evaluation Framework by kaiko.ai

Installation • How To Use • Documentation • Datasets • Benchmarks
Contribute • Acknowledgements

eva is an evaluation framework for oncology foundation models (FMs) by kaiko.ai. Check out the documentation for more information.

Highlights:

• Easy and reliable benchmark of Oncology FMs
• Automatic embedding inference and evaluation of a downstream task
• Native support of popular medical datasets and models
• Produce statistics over multiple evaluation fits and multiple metrics

Installation

Simple installation from PyPI:

# to install the core version only
pip install kaiko-eva

# to install the expanded `vision` version
pip install 'kaiko-eva[vision]'

# to install everything
pip install 'kaiko-eva[all]'

To install the latest version of the main branch:

pip install "kaiko-eva[all] @ git+https://github.com/kaiko-ai/eva.git"

You can verify that the installation was successful by executing:

eva --version

How To Use

eva can be used directly from the terminal as a CLI tool as follows:

eva {fit,predict,predict_fit} --config url/or/path/to/the/config.yaml 

When used as a CLI tool, eva supports configuration files (.yaml) as an argument to define its functionality. Native supported configs can be found at the configs directory of the repo. Apart from cloning the repo, you can download the latest config folder as .zip from your browser from here. Alternatively, from a specific release the configs can be downloaded from the terminal as follows:

curl -LO https://github.com/kaiko-ai/eva/releases/download/0.0.1/configs.zip | unzip configs

For example, to perform a downstream evaluation of DINO ViT-S/16 on the BACH dataset with linear probing by first inferring the embeddings and performing 5 sequential fits, execute:

# from a locally stored config file
eva predict_fit --config ./configs/vision/dino_vit/offline/bach.yaml

# from a remote stored config file
eva predict_fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/offline/bach.yaml

[!NOTE] All the datasets that support automatic download in the repo have by default the option to automatically download set to false. For automatic download you have to manually set download=true.

To view all the possibles, execute:

eva --help

For more information, please refer to the documentation and tutorials.

Benchmarks

In this section you will find model benchmarks which were generated with eva.

Table I: WSI patch-level benchmark

Model	BACH	CRC	MHIST	PCam/val	PCam/test
ViT-S/16 (random) [1]	0.410	0.617	0.501	0.753	0.728
ViT-S/16 (ImageNet) [1]	0.695	0.935	0.831	0.864	0.849
ViT-B/8 (ImageNet) [1]	0.710	0.939	0.814	0.870	0.856
ViT-L/14 (ImageNet) [1]	0.707	0.916	0.832	0.873	0.888
DINO(p=16) [2]	0.801	0.934	0.768	0.889	0.895
Phikon [3]	0.725	0.935	0.777	0.912	0.915
UNI [4]	0.814	0.950	0.837	0.936	0.938
ViT-S/16 (kaiko.ai) [5]	0.797	0.943	0.828	0.903	0.893
ViT-S/8 (kaiko.ai) [5]	0.834	0.946	0.832	0.897	0.887
ViT-B/16 (kaiko.ai) [5]	0.810	0.960	0.826	0.900	0.898
ViT-B/8 (kaiko.ai) [5]	0.865	0.956	0.809	0.913	0.921
ViT-L/14 (kaiko.ai) [5]	0.870	0.930	0.809	0.908	0.898

Table I: Linear probing evaluation of FMs on patch-level downstream datasets.
We report averaged balanced accuracy over 5 runs, with an average standard deviation of ±0.003.

References:

1. "Emerging properties in self-supervised vision transformers”, arXiv
2. "Benchmarking self-supervised learning on diverse pathology datasets”, arXiv
3. "Scaling self-supervised learning for histopathology with masked image modeling”, medRxiv
4. "A General-Purpose Self-Supervised Model for Computational Pathology”, arXiv
5. "Towards Training Large-Scale Pathology Foundation Models: from TCGA to Hospital Scale”, arXiv

Contributing

eva is an open source project and welcomes contributions of all kinds. Please checkout the developer and contributing guide for help on how to do so.

All contributors must follow the code of conduct.

Acknowledgements

Our codebase is built using multiple opensource contributions

Citation

If you find this repository useful, please consider giving a star ⭐ and adding the following citation:

@inproceedings{
kaiko.ai2024eva,
title={eva: Evaluation framework for pathology foundation models},
author={kaiko.ai and Ioannis Gatopoulos and Nicolas K{\"a}nzig and Roman Moser and Sebastian Ot{\'a}lora},
booktitle={Medical Imaging with Deep Learning},
year={2024},
url={https://openreview.net/forum?id=FNBQOPj18N}
}