ferret-xai 0.4.1

Features

ferret offers a painless integration with Hugging Face models and naming conventions. If you are already using the transformers library, you immediately get access to our Explanation and Evaluation API.

Visualization

The Benchmark class exposes easy-to-use table visualization methods (e.g., within Jupyter Notebooks)

bench = Benchmark(model, tokenizer)

# Pretty-print feature attribution scores by all supported explainers
explanations = bench.explain("You look stunning!")
bench.show_table(explanations)

# Pretty-print all the supported evaluation metrics
evaluations = bench.evaluate_explanations(explanations)
bench.show_evaluation_table(evaluations)

Dataset Evaluations

The Benchmark class has a handy method to compute and average our evaluation metrics across multiple samples from a dataset.

import numpy as np
bench = Benchmark(model, tokenizer)

# Compute and average evaluation scores one of the supported dataset
samples = np.arange(20)
hatexdata = bench.load_dataset("hatexplain")
sample_evaluations =  bench.evaluate_samples(hatexdata, samples)

# Pretty-print the results
bench.show_samples_evaluation_table(sample_evaluations)

Planned Developement

See the changelog file for further details.

• ✅ GPU acceleartion support for inference (v0.4.0)

• ✅ Batched Inference for internal methods’s approximation steps (e.g., LIME or SHAP) (v0.4.0)

• ⚙️ Simplified Task API to support NLI, Zero-Shot Text Classification, Language Modeling (branch).

• ⚙️ Multi-sample explanation generation and evaluation

• ⚙️ Support to explainers for seq2seq and autoregressive generation through inseq.

• ⚙️ New evaluation measure: Sensitivity, Stability (Yin et al)

• ⚙️ New evaluation measure: Area Under the Threshold-Performance Curve (AUC-TP) (Atanasova et al.)

• ⚙️ New explainer: Sampling and Occlusion (SOC) (Jin et al., 2020)

• ⚙️ New explainer: Discretized Integrated Gradient (DIG) (Sanyal and Ren, 2021)

• ⚙️ Support additional form of aggregation over embeddings’ hidden dimension.

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

• Cookiecutter: https://github.com/audreyr/cookiecutter

• audreyr/cookiecutter-pypackage: https://github.com/audreyr/cookiecutter-pypackage

Logo and graphical assets made by Luca Attanasio.

If you are using ferret for your work, please consider citing us!

@article{attanasio2022ferret,
  title={ferret: a Framework for Benchmarking Explainers on Transformers},
  author={Attanasio, Giuseppe and Pastor, Eliana and Di Bonaventura, Chiara and Nozza, Debora},
  journal={arXiv preprint arXiv:2208.01575},
  year={2022}
}