DrEvalPy: Python Cancer Cell Line Drug Response Prediction Suite

News: Our paper is out on Nature Communications!

Documentation at ReadTheDocs.

Focus on Innovating Your Models — DrEval Handles the Rest!

• DrEval is a toolkit that ensures drug response prediction evaluations are statistically sound, biologically meaningful, and reproducible.
• Focus on model innovation while using our automated standardized evaluation protocols and preprocessing workflows.
• A flexible model interface supports all model types (e.g. Machine Learning, Stats, Network-based analyses)

By contributing your model to the DrEval catalog, you can increase your work's exposure, reusability, and transferability.

Use DrEval to build drug response models that have an impact

1. Maintained, up-to-date baseline catalog, no need to re-implement literature models
2. Gold standard datasets for benchmarking
3. Consistent application-driven evaluation
4. Ablation studies with permutation tests
5. Cross-study evaluation for generalization analysis
6. Optimized nextflow pipeline for fast experiments
7. Easy-to-use hyperparameter tuning
8. Paper-ready visualizations to display performance

In the critical difference diagram, you can see which models outperform which other models significantly. The diagram is based on the average ranks of the models across all datasets and metrics, and the critical difference is calculated using the Nemenyi test. If model A is outside the bar corresponding to model B, it is significantly better (if it lies to the left) or worse (if it lies to the right) than model B.

This project is a collaboration of the Technical University of Munich (TUM, Germany) and the Freie Universität Berlin (FU, Germany).

Demo

Check out our demo notebook in Colab:

Expected runtime on a normal machine:

• Standalone demo: 38 minutes
• Nextflow demo: 5 minutes

Installation

Using pip:

pip install drevalpy

Optional Ray Tune support (for parallel hyperparameter tuning):

pip install "drevalpy[multiprocessing]"

On a regular machine, the installation should take about a minute.

Using docker:

docker pull ghcr.io/daisybio/drevalpy:latest

From source:

git clone https://github.com/daisybio/drevalpy.git
cd drevalpy
pip install poetry
pip install poetry-plugin-export
poetry install

Check your installation by running in your console:

drevalpy --help

Quickstart

To run models from the catalog, you can run:

drevalpy --run_id my_first_run --models NaiveTissueMeanPredictor NaiveDrugMeanPredictor --dataset_name TOYv1 --test_mode LCO

This will download a small toy drug response dataset, train our baseline models which just predict the drug or tissue means or the mean drug and cell line effects. It will evaluate in "LCO" which is the leave-cell-line-out splitting strategy using 7 fold cross validation. The results will be stored in

results/my_first_run/TOYv1/LCO

You can visualize them using

drevalpy report --run_id my_first_run --dataset_name TOYv1

This will create an index.html file in the results directory which you can open in your web browser.

You can also run a drug response experiment using Python:

from drevalpy.experiment import drug_response_experiment
from drevalpy.models import MODEL_FACTORY
from drevalpy.datasets import AVAILABLE_DATASETS

naive_mean = MODEL_FACTORY["NaivePredictor"] # a naive model that just predicts the training mean
enet = MODEL_FACTORY["ElasticNet"] # An Elastic Net based on drug fingerprints and gene expression of 1000 landmark genes
simple_nn = MODEL_FACTORY["SimpleNeuralNetwork"] # A neural network based on drug fingerprints and gene expression of 1000 landmark genes

toyv1 = AVAILABLE_DATASETS["TOYv1"](path_data="data")

drug_response_experiment(
            models=[enet, simple_nn],
            baselines=[naive_mean], # Ablation studies and robustness tests are not run for baselines.
            response_data=toyv1,
            n_cv_splits=2, # the number of cross validation splits. Should be higher in practice :)
            test_mode="LCO", # LCO means Leave-Cell-Line out. This means that the test and validation splits only contain unseen cell lines.
            run_id="my_first_run",
            path_data="data", # where the downloaded drug response and feature data is stored
            path_out="results", # results are stored here :)
            hyperparameter_tuning=False) # if True (default), hyperparameters of the models and baselines are tuned.

This will run the ElasticNet and SimpleNeuralNetwork models on the TOYv1 dataset, using the NaivePredictor as a baseline. The results will be stored in results/my_first_run/TOYv1/LCO. To obtain evaluation metrics, you can use:

from drevalpy.visualization.create_report import create_report

create_report(
    run_id="my_first_run",
    dataset=toyv1.dataset_name,
    path_data= "data",
    result_path="results",
)

We recommend the use of our Nextflow pipeline for computational demanding runs and for improved reproducibility. No knowledge of Nextflow is required to run it. The nextflow pipeline is available here: nf-core/drugresponseeval.

Example Report

Browse our benchmark results here. You can reproduce the whole analysis by running the following commands:

# Main run
nextflow run nf-core/drugresponseeval \
    -profile docker \
    --run_id main_results \
    --dataset_name CTRPv2 \
    --cross_study_datasets CTRPv1,CCLE,GDSC1,GDSC2 \
    --models DIPK,MultiViewRandomForest \
    --baselines SimpleNeuralNetwork,RandomForest,MultiViewNeuralNetwork,NaiveMeanEffectsPredictor,GradientBoosting,SRMF,ElasticNet,NaiveTissueMeanPredictor,NaivePredictor,SuperFELTR,NaiveCellLineMeanPredictor,NaiveDrugMeanPredictor \
    --test_mode LPO,LCO,LTO,LDO \
    --randomization_mode SVRC,SVRD \
    --randomization_type permutation \
    --measure LN_IC50

# EC50 run
nextflow run nf-core/drugresponseeval \
    -profile docker \
    --run_id ec50_run \
    --dataset_name CTRPv2 \
    --cross_study_datasets CTRPv1,CCLE,GDSC1,GDSC2,PDX_Bruna,BeatAML2 \
    --models RandomForest \
    --baselines NaiveMeanEffectsPredictor \
    --test_mode LCO \
    --measure pEC50

# AUC run
nextflow run nf-core/drugresponseeval \
    -profile docker \
    --run_id auc_run \
    --dataset_name CTRPv2 \
    --cross_study_datasets CTRPv1,CCLE,GDSC1,GDSC2,PDX_Bruna,BeatAML2 \
    --models RandomForest \
    --baselines NaiveMeanEffectsPredictor \
    --test_mode LCO \
    --measure AUC

# Invariant ablation run
# Run this on CPU
nextflow run nf-core/drugresponseeval \
    -profile docker \
    --run_id invariant-rf \
    --dataset_name CTRPv2 \
    --models MultiViewRandomForest \
    --baselines NaiveMeanEffectsPredictor \
    --test_mode LPO,LCO,LDO \
    --randomization_mode SVRC,SVRD \
    --randomization_type invariant \
    --measure LN_IC50

# modify the profile to run this on GPU, if possible
nextflow run nf-core/drugresponseeval \
    -profile docker \
    --run_id invariant-dipk \
    --dataset_name CTRPv2 \
    --models DIPK \
    --baselines NaiveMeanEffectsPredictor \
    --test_mode LPO,LCO,LDO \
    --randomization_mode SVRC,SVRD \
    --randomization_type invariant \
    --measure LN_IC50

## Inference on BeatAMl2, PDX_Bruna
# run this on CPU
nextflow run nf-core/drugresponseeval \
    -profile docker \
    --run_id infer_pdx_beat \
    --dataset_name CTRPv2 \
    --cross_study_datasets PDX_Bruna,BeatAML2 \
    --models RandomForest,SimpleNeuralNetwork,GradientBoosting,SRMF,ElasticNet,NaivePredictor,NaiveDrugMeanPredictor,NaiveCellLineMeanPredictor \
    --baselines NaiveMeanEffectsPredictor \
    --test_mode LPO,LCO,LDO \
    --measure LN_IC50

# modify profile to run this on GPU, if possible
nextflow run nf-core/drugresponseeval \
    -profile docker \
    --run_id dipk_pdx_beat \
    --dataset_name CTRPv2 \
    --cross_study_datasets PDX_Bruna,BeatAML2 \
    --models DIPK \
    --baselines NaiveMeanEffectsPredictor \
    --test_mode LPO,LCO,LDO \
    --measure LN_IC50

Contact

Main developers:

• Judith Bernett, Data Science in Systems Biology, TUM
• Pascal Iversen, Data Integration in the Life Sciences, FU Berlin, Hasso-Plattner-Institut