Benchmarking tool for Counterfactual Explanations
Project description
Universal Counterfactual Benchmark Framework
A well-detailed tutorial on how to use this framework to test counterfactual generators can be found in: https://mazzine.medium.com/testing-counterfactual-generation-algorithms-905f5c45fc1c
Simple test tutorial
The example below shows how to run a simple test using all datasets (generating 10 counterfactuals per dataset, total 210).
STEP 0 - Computer Requirements
To run this benchmark to test your counterfactual generation algorithm you will need:
- Ubuntu 18.04
- Anaconda 2020 version or later
SETP 1 - Clone repository
Clone this repository using:
git clone
SETP 2 - Modify simple_test.py
SCRIPT
The script simple_test.py
has the instructions on how you should add your CF generator. Modify it to include your algorithm.
*You may want to copy and paste your code in the same folder *This script includes a dummy counterfactual generator (that only returns the factual instance), so you can run it first to understand more the framework and verify if the prerequisites are met.
There are 6 fields to be modified:
- Your framework name
- Dataset selection to be tested (categorical, numerical, mixed)
- Number of outputs from the neuronal network (1 or 2)
- Initial configuration of the counterfactual generator
- Generation of the counterfactual for the
factual
variable instance - Post-processing of the counterfactual generator result, output of counterfactual candidate
Run multiple shell
Example to run multiple without terminal message outputs
Arguments
- A - Starting Row
- B - Ending Row
- C - Name of the benchmark to be run
- D - Dataset to be run (0 to 21)
- E - Class (0 or 1)
cd ./benchmark
sh run_shell_multiple.sh A B C D E &> /dev/null
Example:
cd ./benchmark
sh run_shell_multiple.sh 0 10 benchmark_MLEXPLAIN.py 0 0 &> /dev/null
WARNING
ONLY USE THE SCRIPTS run_full_batch_DS0.sh
AND run_full_batch_DS1.sh
IN A POWERFUL COMPUTER, THESE SCRIPTS RUN ALL DATASETS IN ONE TIME FOR ONE CLASS
Example:
sh run_full_batch_DS0.sh benchmark_MLEXPLAIN.py &> /dev/null
Instructions using on Google Cloud Computing Engine
For this experiment, the Google Cloud Computing Engine was used with the following specification:
For datasets except InternetAdv
- Series: N1
- Machine Type: Custom
- Cores: 52
- CPU Platform: Intel Skylake or later
- Memory: 195 GB
- OS: Ubuntu 18.04
- Disk: SSD 400 GB
For InternetAdv dataset
- Series: N2D
- Machine Type: Custom
- Cores: 48
- CPU Platform: AMD Rome or later
- Memory: 384 GB
- OS: Ubuntu 18.04
- Disk: SSD 400 GB
Benchmark steps on GCCE
WARNING - THE FOLLOWING STEPS WILL RUN A SCRIPT THAT CONSUMES LOTS OF RESOURCES, SEE THE COMPUTING REQUIREMENTS BEFORE USING
Use as root
sudo su
Go to temporary folder
cd /tmp
Download Anaconda 2020.11
curl -O https://repo.anaconda.com/archive/Anaconda3-2021.11-Linux-x86_64.sh
Install Anaconda
bash Anaconda3-2021.11-Linux-x86_64.sh
Update source
source ~/.bashrc
Clone this repo
git clone ...
Enter repo benchmark folder
cd CounterfactualBenchmark/benchmark
Update Ubuntu Package Manager
apt-get update
Start run detached from current terminal (as it takes a long time to run)
nohup bash run_benchmark_full_0_50.sh &> /dev/null
The next steps are made to guarantee the job will not stop even if the terminal session closes
Press Ctrl+Z to make the process in background
Return process run
bg
Find process id (PROCESS_ID
)
jobs -l
Detach job from terminal session
disown PROCESS_ID
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.