Two main structure types are supported, without or without following an index-based naming convention. Assuming in the following example that the data indexes are based on their origin, but anything should work. The folders named index0 and index1 could be renamed to any sets of strings.

The metrics and overall validation can be computed for multiple segmentation classes at the same time, granted that unique and name-matching sets of files (i.e., ground truth and prediction files) are provided.

1.1 Original data folder structure

The main data directory containing the original 3D volumes and corresponding manual annotations is expected to resemble the following structure using an index-based naming convention:

└── path/to/data/root/
    └── index0/
        ├── Pat001/
        │   ├── volumes/
        │   │   └── Pat001_MRI.nii.gz
        │   └── segmentations/
        │   │   ├── Pat001_MRI_label_tumor.nii.gz
        │   │   └── Pat001_MRI_label_other.nii.gz
        ├── Pat025/
        └── Pat050/
    └── index1/
        ├── Pat100/
        └── Pat150/

The main data directory containing the original 3D volumes and corresponding manual annotations is expected to resemble the following structure when not using an index-based naming convention:

└── path/to/data/root/
    └── Pat001/
    │   ├── Pat001_MRI.nii.gz
    │   ├── Pat001_MRI_label_tumor.nii.gz
    │   └── Pat001_MRI_label_other.nii.gz
    └── Pat010/
    │   ├── Pat010_MRI.nii.gz
    │   ├── Pat010_MRI_label_tumor.nii.gz
    │   └── Pat010_MRI_label_other.nii.gz
    [...]
    └── Pat100/
    │   ├── Pat100_MRI.nii.gz
    │   ├── Pat100_MRI_label_tumor.nii.gz
    │   └── Pat100_MRI_label_other.nii.gz

1.2 Inference results folder structure

Predictions results are expected to be stored inside a predictions/ sub-folder, the outer-most sub-folder naming convention inside the folder are the fold numbers. The inference results should be grouped inside what will become the validation folder, resembling the following structure when using an index-based naming convention.

└── path/to/validation/study/
    └── predictions/
        ├── 0/
        │   ├── index0_Pat001/
        │   │   ├── Pat001_MRI-pred_tumor.nii.gz
        │   │   └── Pat001_MRI-pred_other.nii.gz  
        │   ├── index0_Pat002/
        │   │   ├── Pat002_MRI-pred_tumor.nii.gz
        │   │   └── Pat002_MRI-pred_other.nii.gz  
        └── 1/
        │   ├── index1_Pat100/
        │   │   ├── Pat100_MRI-pred_tumor.nii.gz
        │   │   └── Pat100_MRI-pred_other.nii.gz  
        │   └── index1_Pat150/ 
        │   │   ├── Pat150_MRI-pred_tumor.nii.gz
        │   │   └── Pat150_MRI-pred_other.nii.gz  

The inference results should be grouped inside what will become the validation folder, resembling the following structure when not using an index-based naming convention.

└── path/to/validation/study/
    └── predictions/
        ├── 0/
        │   ├── Pat001/
        │   │   ├── Pat001_MRI-pred_tumor.nii.gz
        │   │   └── Pat001_MRI-pred_other.nii.gz  
        │   ├── Pat002/
        │   │   ├── Pat002_MRI-pred_tumor.nii.gz
        │   │   └── Pat002_MRI-pred_other.nii.gz  
        └── 1/
        │   ├── Pat100/
        │   │   ├── Pat100_MRI-pred_tumor.nii.gz
        │   │   └── Pat100_MRI-pred_other.nii.gz  
        │   └── Pat150/ 
        │   │   ├── Pat150_MRI-pred_tumor.nii.gz
        │   │   └── Pat150_MRI-pred_other.nii.gz  

1.3 Folds file

The file with patients' distribution within each fold used for training should list the content of the validation and test sets iteratively.
The file should be called cross_validation_folds.txt and placed in the validation study folder side-by-side with the predictions sub-folder.

An example of its content is given below when using an index-based naming convention:

  index0_Pat1000_MRI_sample index1_Pat1250_MRI_sample\n    
  index0_Pat001_MRI_sample index1_Pat025_MRI_sample\n  
  index0_Pat001_MRI_sample index1_Pat025_MRI_sample\n    
  index0_Pat100_MRI_sample index1_Pat150_MRI_sample\n  

An example of its content is given below when not using an index-based naming convention:

  Pat001_MRI Pat002_MRI\n    
  Pat100_MRI Pat150_MRI\n  
  Pat100_MRI Pat150_MRI\n    
  Pat200_MRI Pat250_MRI\n  

  cd /path/to/validation_metrics_computation  
  virtualenv -p python3 venv  
  source venv/bin/activate  
  TMPDIR=$PWD/venv pip install --cache-dir=$PWD/venv -e .

Then the final step is to do the following in a terminal.

  cd /path/to/validation_metrics_computation  
  cp blank_main_config.ini main_config.ini 

You can now edit your main_config.ini file for running the different processes.
An additional explanation of all parameters specified in the configuration file can be found in /Utils/resources.py.

After filling in the configuration file specifying all runtime parameters, according to the pattern from blank_main_config.ini, you should run first the validation task and then the study task.

CLI

raidionicsval -c CONFIG (-v debug)

CONFIG should point to a configuration file (*.ini).

Python module

from raidionicsval import compute
compute(config_filename="/path/to/main_config.ini")

"/path/to/main_config.ini" should point to a valid configuration file.

Docker

When calling Docker images, the --user flag must be properly used in order for the folders and files created inside the container to inherit the proper read/write permissions. The user ID is retrieved on-the-fly in the following examples, but it can be given in a more hard-coded fashion if known by the user.

docker pull dbouget/raidionics-val:v1.0-py38-cpu

For opening the Docker image and interacting with it, run:

docker run --entrypoint /bin/bash -v /home/<username>/<resources_path>:/workspace/resources -t -i --network=host --ipc=host --user $(id -u) dbouget/raidionics-val:v1.0-py38-cpu

The /home/<username>/<resources_path> before the column sign has to be changed to match a directory on your local machine containing the data to expose to the docker image. Namely, it must contain folder(s) with data to use as input for the validation studies, and it will contain the destination folder where the results will be saved.

For launching the Docker image as a CLI, run:

docker run -v /home/<username>/<resources_path>:/workspace/resources -t -i --network=host --ipc=host --user $(id -u) dbouget/raidionics-val:v1.0-py38-cpu -c /workspace/resources/<path>/<to>/main_config.ini -v <verbose>

The <path>/<to>/main_config.ini must point to a valid configuration file on your machine, as a relative path to the /home/<username>/<resources_path> described above. For example, if the file is located on my machine under /home/myuser/Data/Validation/main_config.ini, and that /home/myuser/Data is the mounted resources partition mounted on the Docker image, the new relative path will be Validation/main_config.ini.
The <verbose> level can be selected from [debug, info, warning, error].