interpreto 0.4.20

Interpretability toolbox for LLMs

📚 Explore Interpreto docs >>
🖼️ Checkout our explanation gallery >>

🚀 Quick Start

The library is available on PyPI, try pip install interpreto to install it.

Checkout the tutorials to get started:

• Attributions walkthrough (both classification and generation)
• Classification concept-based explanations
• Generation concept-based explanations

📦 What's Included

Interpreto 🪄 provides a modular framework encompassing Attribution Methods, Concept-Based Methods, and Evaluation Metrics.

🔥 Attribution Methods

Interpreto includes both inference-based and gradient-based attribution methods.

They all work seamlessly for both classification (...ForSequenceClassification) and generation (...ForCausalLM)

Inference-based Methods:

• KernelShap — Lundberg and Lee, 2017
• LIME — Ribeiro et al., 2013
• Occlusion — Zeiler and Fergus, 2014
• Sobol — Fel et al., 2021

Gradient-based methods:

• GradientShap — Lundberg and Lee, 2017
• InputxGradient — Simonyan et al., 2013
• Integrated Gradient — Sundararajan et al., 2017
• Saliency — Simonyan et al., 2013
• SmoothGrad — Smilkov et al., 2017
• SquareGrad — Hooker et al., 2019
• VarGrad — Richter et al., 2020

💡 Concept-Based Methods or Mechanistic Interpretability

Concept-based explanations aim to provide high-level interpretations of latent model representations.

Interpreto generalizes these methods through four core steps:

1. Split a model in two and obtain a dataset of activations
2. Concept Discovery (e.g., from latent embeddings)
3. Concept Interpretation (mapping discovered concepts to human-understandable elements)
4. Concept-to-Output Attribution (assessing concept relevance to model outputs)

1. Split a model in two and obtain a dataset of activations: (mainly via nnsight):

Choose any layer in any HuggingFace language model with our ModelWithSplitPoints based on nnsight. Then pass a dataset through it to obtain a dataset of activations.

2. Dictionary Learning for Concept Discovery (mainly via overcomplete):

• Interpret neurons directly via NeuronsAsConcepts
• NMF, Semi-NMF, ConvexNMF
• ICA, SVD, PCA, KMeans
• SAE variants: Vanilla SAE, TopK SAE, JumpReLU SAE, BatchTopK SAE

3. Available Concept Interpretation Techniques:

• Top-k tokens from tokenizer vocabulary via TopKInputs and use_vocab=True
• Top-k tokens/words/sentences/samples from specific datasets via TopKInputs
• Label concepts via LLMs with LLMLabels (Bills et al. 2023)

Concept Interpretation Techniques Added in the future:

• Input-to-concept attribution from dataset examples (Jourdan et al. 2023)
• Theme prediction via LLMs from top-k tokens/sentences
• Aligning concepts with human labels (Sajjad et al. 2022)
• Word cloud visualizations of concepts (Dalvi et al. 2022)
• VocabProj & TokenChange (Gur-Arieh et al. 2025)

4. Concept-to-Output Attribution:

Estimate the contribution of each concept to the model output.

Can be obtained with any concept-based explainer via MethodConcepts.concept_output_gradient().

Papers available in the future:

Thanks to this generalization encompassing all concept-based methods and our highly flexible architecture, we can easily obtain a large number of concept-based methods:

• CAV and TCAV: Kim et al. 2018, Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
• ConceptSHAP: Yeh et al. 2020, On Completeness-aware Concept-Based Explanations in Deep Neural Networks
• COCKATIEL: Jourdan et al. 2023, COCKATIEL: COntinuous Concept ranKed ATtribution with Interpretable ELements for explaining neural net classifiers on NLP
• Yun et al. 2021, Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors
• FFN values interpretation: Geva et al. 2022, Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space
• SparseCoding: Cunningham et al. 2023, Sparse Autoencoders Find Highly Interpretable Features in Language Models
• Parameter Interpretation: Dar et al. 2023, Analyzing Transformers in Embedding Space

📊 Evaluation Metrics

Evaluation Metrics for Attribution

To evaluate attribution methods faithfulness, there are the Insertion and Deletion metrics.

Evaluation Metrics for Concepts

Concept-based methods have several steps that can be evaluated together via ConSim.

Or independently:

• Concept-space (dictionary learning evaluation)
  • faithfulness: MSE, FID, and ReconstructionError
  • complexity: Sparsity, SparsityRatio, SparsityRatio
  • stability: Stability
• Concepts interpretations
  • No metric yet, will be included soon.
• Concept-to-Output attribution
  • No metric yet, will be included soon.

👍 Contributing

Feel free to propose your ideas or come and contribute with us on the Interpreto 🪄 toolbox! We have a specific document where we describe in a simple way how to make your first pull request.

👀 See Also

More from the DEEL project:

• Xplique a Python library dedicated to explaining neural networks (Images, Time Series, Tabular data) on TensorFlow.
• Puncc a Python library for predictive uncertainty quantification using conformal prediction.
• oodeel a Python library that performs post-hoc deep Out-of-Distribution (OOD) detection on already trained neural network image classifiers.
• deel-lip a Python library for training k-Lipschitz neural networks on TensorFlow.
• deel-torchlip a Python library for training k-Lipschitz neural networks on PyTorch.
• Influenciae a Python library dedicated to computing influence values for the discovery of potentially problematic samples in a dataset.
• DEEL White paper a summary of the DEEL team on the challenges of certifiable AI and the role of data quality, representativity and explainability for this purpose.

🙏 Acknowledgments

This project received funding from the French ”Investing for the Future – PIA3” program within the Artificial and Natural Intelligence Toulouse Institute (ANITI). The authors gratefully acknowledge the support of the DEEL and the FOR projects.

👨‍🎓 Creators

Interpreto 🪄 is a project of the FOR and the DEEL teams at the IRT Saint-Exupéry in Toulouse, France.

🗞️ Citation

If you use Interpreto 🪄 as part of your workflow in a scientific publication, please consider citing 🗞️ our paper:

@article{poche2025interpreto,
    title       = {Interpreto: An Explainability Library for Transformers},
    author      = {Poch{\'e}, Antonin and Mullor, Thomas and Sarti, Gabriele and Boisnard, Fr{\'e}d{\'e}ric and Friedrich, Corentin and Claye, Charlotte and Hoofd, Fran{\c{c}}ois and Bernas, Raphael and Hudelot, C{\'e}line and Jourdan, Fanny},
    journal     = {arXiv preprint arXiv:2512.09730},
    year        = {2025}
}

📝 License

The package is released under MIT license.