faithfulness 0.0.4

A library to evaluate faithfulness (factual correctness or consistency) of abstractive summaries.

Faithfulness 😇

An easy-to-use library to evaluate faithfulness (factual correctness) of abstractive summaries. Faithfulness is computed by comparing a summary with its original source document.

This library includes multiple faithfulness metrics based on:

• BERTScore
• Entailment
• Question Generation & Question Answering framework (QGQA)
• Named Entity Recognition (NER)
• Open Information Extraction (OpenIE)
• Semantic Role Labeling (SRL)
• Sentence Similarity (SentSim)

Installation ⚙️

1. $ conda create -n my_project python=3.8 This creates a new virtual environment for your project with conda. You can activate it with $ conda activate my_project.
2. $ conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.1 -c pytorch Please install PyTorch by following the instructions here. Make sure to install the CUDA variant that matches the CUDA version of your GPU.
3. $ pip install faithfulness This installs the faithfulness library and it's dependencies. Read more about the dependencies below.

All faithfulness metrics are model-based. Some models have to be installed manually:

• Download the SRL model here and save it in your project. e.g. /models/srl_model.tar.gz
• Download a spacy model: $ python -m spacy download en_core_web_sm
• Download CoreNLP: import stanza && stanza.install_corenlp()

Usage 🔥

from faithfulness.QGQA import QGQA

qgqa = QGQA()
summary = "Lorem ipsum dolor sit amet"
source = "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam ..."
faithfulness, info = qgqa.score(summary, source)

More examples can be found here 💯.

Evaluation 📊

We evaluated all faithfulness metrics by correlating them with human judgements on the XSUM dataset (link). You will soon be able to read more about the evaluation in our paper. (Master's thesis)

Method	Pearson (r)	Spearman (p)
🥇 BERTScore	0.501	0.486
🥈 Entailment	0.366	0.422
🥉 SentSim	0.392	0.389
SRL	0.393	0.377
NER	0.252	0.259
QGQA	0.228	0.258
OpenIE	0.169	0.185

Dependencies 🔗

By running $ pip install faithfulness you will install this library as well as the following dependencies:

• 🤗 transformers
• spaCy (used for Entailment, NER, QGQA, SentSim, SRL)
• Stanza (used for OpenIE)
• AllenNLP (used for SRL)
• SentenceTransformers (used for NER, OpenIE, QGQA, SentSim, SRL)

Troubleshooting 🛠

There are currently problems when installing allennlp and jsonnet. If you encounter "Building wheel for jsonnet (setup.py) ... error" during the installation please try:

apt-get install make 
apt-get install g++ 

or install jsonnet before installing this library

conda install -c conda-forge jsonnet
pip install faithfulness