CEval — Counterfactual Explanation Evaluator

CEval is a lightweight Python package for evaluating the quality of counterfactual explanations produced by post-hoc XAI (Explainable AI) methods. It computes 14 established metrics with a single call and works with diverse architectures.

Paper: Bayrak, B., & Bach, K. (2024). Evaluation of Instance-Based Explanations: An In-Depth Analysis of Counterfactual Evaluation Metrics, Challenges, and the CEval Toolkit. IEEE Access. doi:10.1109/ACCESS.2024.3410540

Why CEval?

When you build or compare counterfactual explainers, you need more than one number to judge quality. CEval lets you measure all key dimensions (validity, proximity, sparsity, diversity, feasibility, and more) in a single unified framework, across different explainers and datasets.

from ceval import CEval

evaluator = CEval(samples=test_df, label="income", data=train_df, model=clf)
evaluator.add_explainer("DiCE",  dice_cfs,  "generated-cf")
evaluator.add_explainer("DICE+", dicep_cfs, "generated-cf")

print(evaluator.comparison_table)

Installation

pip install CEval

Requirements: Python ≥ 3.9, pandas, numpy, scikit-learn, scipy, gower, category-encoders

Metrics

Not every metric applies to every explanation type, CEval handles this automatically and fills non-applicable cells with "-".

Quick Start

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from ceval import CEval

# 1. Prepare your data
train_df = ...   # pd.DataFrame with features + label column
test_df  = ...   # pd.DataFrame with features + label column
clf      = RandomForestClassifier().fit(train_df.drop("label", axis=1),
                                        train_df["label"])

# 2. Generate counterfactuals with your favourite explainer
#    (DiCE, PertCF, DICE, NICE, etc.)
counterfactuals = ...   # pd.DataFrame, same columns as test_df

# 3. Evaluate
evaluator = CEval(
    samples    = test_df,        # instances to explain
    label      = "label",        # target column name
    data       = train_df,       # background dataset (unlocks more metrics)
    model      = clf,            # fitted classifier (unlocks more metrics)
    k_nn       = 5,              # neighbours for kNN-based metrics
    constraints= ["age"],        # features that must not change (optional)
)

evaluator.add_explainer(
    name            = "MyExplainer",
    explanations    = counterfactuals,
    exp_type        = "generated-cf",   # "generated-cf" | "existed-cf" |
                                        # "generated-factual" | "existed-factual"
    mode            = "1to1",           # "1to1" | "1toN"
)

print(evaluator.comparison_table)

Explanation types

Explanation modes

Model Compatibility

CEval works with any classifier, not just scikit-learn.
Use the built-in wrappers from ceval.wrappers to adapt your model:

# PyTorch
from ceval.wrappers import TorchWrapper
model = TorchWrapper(my_net, num_classes=2, device="cuda")

# XGBoost  (works with XGBClassifier and native Booster)
from ceval.wrappers import XGBoostWrapper
model = XGBoostWrapper(xgb_clf)

# Keras / TensorFlow
from ceval.wrappers import KerasWrapper
model = KerasWrapper(keras_model, num_classes=3)

# Anything else — supply two callables
from ceval.wrappers import GenericWrapper
model = GenericWrapper(
    predict_fn       = lambda X: my_model.infer(X).argmax(axis=1),
    predict_proba_fn = lambda X: my_model.infer(X),
)

# Then use as normal
evaluator = CEval(samples=test_df, label="income", data=train_df, model=model)

If you pass an incompatible model without a wrapper, CEval raises a clear TypeError that tells you exactly which wrapper to use.

See examples/demo_adult_income.py for a complete working demo that:

• Loads the Adult Income dataset
• Trains a Random Forest classifier
• Generates counterfactuals with DiCE
• Evaluates them in both 1-to-1 and 1-to-N mode
• Prints a full comparison table

python examples/demo_adult_income.py

Expected output:

                     DiCE (1-to-1)  DiCE (1-to-N)
validity                      0.90          0.867
proximity_gower               0.11          0.152
sparsity                      0.32          0.347
yNN                           0.68          0.713
feasibility                  48.21         183.44
redundancy                    0.80          0.733
constraint_violation          0.50          0.233
...

Comparing Multiple Explainers

evaluator = CEval(samples=test_df, label="label", data=train_df, model=clf)

evaluator.add_explainer("DiCE",   dice_cfs,   "generated-cf", mode="1toN")
evaluator.add_explainer("PertCF", pertcf_cfs, "generated-cf", mode="1toN")
evaluator.add_explainer("NICE",   nice_cfs,   "existed-cf",   mode="1to1")

# Side-by-side comparison
print(evaluator.comparison_table.T)

API Reference

CEval:

CEval(samples, label, ...)

evaluator.add_explainer:

evaluator.add_explainer(name, explanations, exp_type, mode="1to1")

Registers an explainer and computes all applicable metrics. Results are appended to evaluator.comparison_table.

evaluator.comparison_table:

A pd.DataFrame with one row per explainer and one column per metric. Non-applicable metrics show "-".

Citation

If you use CEval in your research, please cite:

@article{bayrak2024ceval,
  title   = {Evaluation of Instance-Based Explanations: An In-Depth Analysis of Counterfactual Evaluation Metrics, Challenges, and the CEval Toolkit},
  author  = {Bayrak, Bet{\"u}l and Bach, Kerstin},
  journal = {IEEE Access},
  year    = {2024},
  doi     = {10.1109/ACCESS.2024.3410540}
}

Related Work

This package is part of a broader research effort on counterfactual explanation methods:

• PerCE — Personalised Counterfactual Explanations (IEEE)
• PertCF — Perturbation-based Counterfactual Explainer (Paper | Code)

License

MIT © Betül Bayrak